Ask2Mask: Guided Data Selection for Masked Speech Modeling
Authors: Murali Karthick Baskar
Andrew Rosenberg
Bhuvana Ramabhadran
Yu Zhang
Pedro Moreno
Venue: ICLR 2022 Submitted (2022)
Keywords: Masked speech modeling (MSM)
Data selection
Self-supervision
ASR
Speech recognition

Abstract: Masked speech modeling (MSM) methods such as wav2vec2 or w2v-BERT learn representations over speech frames which are randomly masked within an utterance. While these methods improve performance of Automatic Speech Recognition (ASR) systems, they have one major limitation. They treat all unsupervised speech samples with equal weight, which hinders learning as not all samples have relevant information to learn meaningful representations. In this work,  we address this limitation. We propose ask2mask (ATM), a novel approach to focus on specific samples during MSM pre-training.  ATM employs an external ASR model or \textit{scorer} to weight unsupervised input samples in two different ways: 1) A fine-grained data selection is performed by masking over the highly confident input frames as chosen by the scorer. This allows the model to learn meaningful representations. 2) ATM is further extended to focus at utterance-level by weighting the final MSM loss with the utterance-level confidence score.  We conduct fine-tuning experiments on two well-benchmarked corpora: LibriSpeech (matching the pre-training data) and AMI (not matching the pre-training data). The results substantiate the efficacy of ATM on significantly improving the recognition performance under mismatched conditions (up to 11.6\% relative) while still yielding modest improvements under matched conditions.

Under review as a conference paper at ICLR 2022
AsK2MASK: GUIDED DATA SELECTION FOR MASKED SPEECH MODELING
Anonymous authors Paper under double-blind review
ABSTRACT
Masked speech modeling (MSM) methods such as wav2vec2 or w2v-BERT learn representations over speech frames which are randomly masked within an utter- ance. While these methods improve performance of Automatic Speech Recogni- tion (ASR) systems, they have one major limitation. They treat all unsupervised speech samples with equal weight, which hinders learning as not all samples have relevant information to learn meaningful representations. In this work, we address this limitation. We propose ask2mask (ATM), a novel approach to focus on spe- cific samples during MSM pre-training. ATM employs or scorer to weight unsupervised input samples in two different ways: 1) A fine- grained data selection is performed by masking over the highly confident input frames as chosen by the scorer. This allows the model to learn meaningful repre- sentations. 2) ATM further extended to focus at utterance-level by weighting the final MSM loss with the utterance-leve confidence score. We conduct fine-tuning experiments on two hell-benchmarked corpora: LibriSpeech (matching the pre- training data) and Commonvoice, TED-LIUM, AMI and CHiME-6 (not matching the pre-training data). The results substantiate the efficacy of ATM on signifi- cantly improving the recognition performance under mismatched conditions (up to 11.6% relative over published results and upto 4.46% relative over our internal baseline) while still yielding modest improvements under matched conditions.
external ASR model
INTRODUCTION
Self-training and self-supervised training techniques rely on huge amounts of unlabeled speech or text data for better generalization. The self-training techniques such as pseudo-labeling (Scudder) 1965 Kahn et al. 2020) and student-teacher training (Park et al. 2020) have shown promising improvements by incorporating the data selection process. This data selection step removes pseudo- labels with less confidence as denoted by the teacher model before feeding the input to student model. Xu et al. (202T) shows that self-training and self-supervised training are complementary to each other and also show that self-supervised models act as good initialization for self-traing self-training techniques. Self-super training (Hinton & Zemel 1994) is a representation learning approach which implicitly learns patterns in the data without relying explicit labels. Masked speech mod- eling (MSM) is the recent and successful self-supervised learning technique, thanks to the advent of BERT (Devlin al. 20188) in NLP which inspired learning speech representations from masked inputs. MSM techniques such as wav2vec2 (Baevski et al., (2020). HuBERT (Hsu et al.) (202 and w2v-BERT (Chung al. 202 have shown considerable gains across various down-stream speech tasks and have become the go-to models for ASR.
Unfortunately, MSM does not have a data selection scheme to discard the irrelevant input samples and instead imposes burden on the training criterion to learn the relevance of the input samples in learning meaningful representations, Hsu et al. (2021a) noticed the impact of not selecting relevant data from the huge amounts of unsupervised data during pre-training by showing degradation in ASR performance when fine-tuned to a target dataset with limited data. To mitigate this constraint, Chan et al. (2021) introduced substantially more fine-tuning data related to the target dataset but did not achieve satisfactory results. Hsu et al. (2021a) attempted to solve this issue by heuristically selecting the data from closed set of unsupervised speech databases or by pooling in data relevant to target dataset along with the existing pre-training dataset. However, this data selection approach is not done within the existing pre-training dataset and it is not completely empirically motivated.

---

Under review as a conference paper at ICLR 2022
In this study, in order to break the above limitation of the MSM techniques, propose simple strategy named ask2mask (ATM) to incorporate data selection within a chosen pretraining dataset.
In ATM, the masking is done over the input samples or speech frames with higher confi- dence as determined by the scorer. This is contrary to the random selection of frames to be masked in conventional MSM models. We hypothesize that this guided selection of frames to be masked allows the model to focus on the frames which can provide meaningful repre- sentations. The scoring model used in this work is necessarily a speech recognition model trained on small amount of data and provides frame-level confidence for each input.
The ATM technique further extended to exploit the confidence values provided by the scorer by directly using them to re-weight the MSM loss. We denote this approach ATM with loss scaling (ATM+S), It allows the model training to focus on certain utterances by down scaling the utterances with low-confidences.
Similar to our work based on masking with external guidance, there is work in NLP that also benefit by incorporating masking with knowledge. In SSnn et al.] (2019), masking is done at phrase-leve segments in BERT and has shown to learn semantic dependencies. In |Wang et al. (2019), phonetic knowledge is injected to mask over phonetic segments to perform spectral augmentation.
Our ATM approach is primarily motivated based the recent work by Vesely al. (2017) semi-supervised learning of conventional ASR systems which shows that performing data selection frame-level token-level pruning out the imput samples at frame-level has been studied in Ferreira et al. (202I) to improve both classification and regression tasks. Few works on unsupervised learning also highlight the importance of weighting the data based on its confidence (Wessel et al.] 20011: Ren et al., 2020) Coleman et al. 2019). We hypothesize that ATM can leverage the effect of data selection within
unsupervised data provides better performance. The importance of
particular training corpus to further enhance the recognition performance of MSM techniques.
To summarize, our contributions are listed as follows:
Novelty: To the extent of our knowledge, ATM is the first approach incorporate within- corpus data selection strategy in MSM. We also show that data selection can be simply performed inside MSM by guided selection of frames to be masked using a scorer model.
Technical contributions: We provide two simple strategies to incorporate data selection into MSM pretraining by applying the confidence of the scorer: 1) choosing the data at frame- level by applying guided masking 2) soft weighting the data at utterance level by scaling the MSM loss of each utterance with its corresponding confidence score. ATM is designed to be compatible all MSM based pre- pre-training techniques.
Empirical study: Analysis is done to find an optimal masking percentage for ATM and we highlight the effectiveness of ATM across varying masking percentages. The importance of masking frames with high confidence is substantiated by empirically comparing it with masking low confident frames and random frames respectively. Experiments are performed AMI data which is from distinct condition compared to Libri-light corpus used for MSM based pretraining. The results confirm the importance of ATM by improving the recognition performance evaluation sets of AMI by significant margin.
PRELIMINARIES ON MASKED SPEECH MODELING (MSM)
In this section, we formally define the masked speech modeling (MSM) technique and brief primary instantiations including wav2vec2 and w2y-BERT. The technique can be formulated by defining input speech sequence X = [T1 I2, ..., ], where He is the log Mel-filterbank feature vector at time t. X is sent to the feature encoder & to obtain the encoded representations E = ((X). The feature encoder contains convolutional layers performing subsampling at a factor of 4 and reducing the total number of frames of an utterance from T' to T to get E [e,, C2 ..., er]. ]. E is then sent to two parallel modules: 1) masking component, and 2) quantizer.
2

---

Under review as a conference paper at ICLR 2022
2.1 MASKING
The idea behind masking input samples and predicting them was initially proposed in BERT (De- vlin al., 2018) and later adopted speech (Baevski et al. 2020) with modifications suit the characteristics of speech input. The masking is done over sets of frames or blocks 61 bg.... ..., bx and accommodates overlap between blocks. Here K is the number of masked blocks in randomly masked encoded sequence E. The importance of block masking motivated by the improvements observed in Span- Dpan-BERT'S by Joshi et al. (2020) and ERNIE (Sun et al. 2019). The block 6₁ = [ix c), where ig is the starting index the masked block and is the corresponding right context size de- noting the number of consecutive speech frames. Here are randomly sampled from uniform distribution. has been empirically observed by Baevski et al. (2020) that 49% the frames are masked and 10 is chosen as the golden ratio to attain best representation during pre-training.
2.2 QUANTIZER
Gumbel-softmax quantizer component V/ is used to get quantized representations which act targets for wav2vec2 and w2v-BERT models. These quantized representations align to phonetic units as described in Baevski al. (2020)- Each quantized vector is of L. dimensions which denote the number of targets or codes used in a codebook. Each incoming input E is projected to L dimensions within the quantizer before applying the Gumbel-softmax.
2.3 CONTEXT NETWORK AND MSM Loss
War2vec2-conformer In this model type, the unmasked sequence E is sent V to get T/ (E), where Q (9) 92....7777 described in Baevski al. 2020). The masked sequence É is fed to the context network S2 which contains conformer blocks to learn contextual representations from the input.
(?(E) denotes the output of the context network. The contrastive loss Letr(C; qj) objective is computed between the quantized representation ęj and context network output oj C for all masked time instances j E J. Diversity loss Ldiv is computed as an auxiliary objective in wav2vec2 to force the model to choose diverse codes in the quantization codebook. Detailed description of Lavo is in Baevski et al. (2020). The final training objective denoted as:
frve fat + 0.1 Latior
where Lar = Eÿ== Latr(Cj, 4j).
HuBERT-conformer: This is another variant of waw2vec2-conformer model with two major dif- ferences: 1) Targets are k-means cluster ids which are computed over a small portion of input 2) Cross-entropy loss (ūj. yj) is computed between the prediction of the context network ú, and the k-means cluster target V/j-
W2V-BERT: This model marries the concept of wav2vec2 and BERT model by including ad- ditional context network A containing conformer blocks in addition Ω as in wav2vec2. The A receives the output of the Ω and strives to further learn refined contextual information to get H = ACC). The targets of w2v-BERT yj computed by taking an argmax over the codebook dimensions L of quantized representations qj as:
Vj arg max LLL
(2)
Finally, the cross-entropy loss Lee(ūj; Yj) is computed between the prediction ú; softmax(hy) and the target y₁ over the masked time instances J. The final training objective Lub = Loe + Lwo combination of cross-entropy loss and wav2vec2 loss. A block diagrammatic overview of the above MSM architectures are available in appendix A.7
3 ASK2MASK (ATM)
The primary reason to employ pre-training models is to exploit the abundantly available unsuper- vised data for improving ASR under limited availability of supervised data. While the MSM models such way2vec2 and w2y-BERS described in Section exploit the unsupervised data, they treat
3

---

Under review as a conference paper at ICLR 2022
each data with equal weight for computing the final Instee,, generate 8, for each encoded frame et. This is used to select relevant data in a fine-grained manner during masking for computing the loss objective.
3.1 METHODOLOGY
For each encoded feature frame E E, the scorer emits probabilities p(v; =1|E); L of the frame belonging to a particular label. The scorer model is a CTC based synneeeycchoooous ASR model separately trained with limited amount of data. Our initial intuition was to chose the scorer's training data to match the target data condition, however our empirical analysis (cf. Section 5.3) shows that the performance is agnostic to the scorer model's training data. Finally, the confidence score St of the frame is defined as the maximum probability across all labels:
maxp(u, E
(3)
We sample K masking start indices (i1, .. } with probabilities:
pli
deg
(4)
Σ 80
og ($1.---52-) 1}
That is, we sample beginning frames with probability proportional to the scores of each frame. The indicator function ōig(in.. 1) ensures that we sample without replacement. This is the key difference between ATM and the random masking in prior works described in Section 2.1 Prior works uniformly sample the start indices of each masking block bj.k, while the ATM uses the probability distribution induced by the scorer. K is determined by the percentage of frames to be masked.
We hypothesize that frames with maximum confidence from external scoring model will be 1) easiest to learn using an MSM training criteria and 2) most informative in for pretraining to facilitate fine-tuning. Conversely, the lowest confidence frames, those more confusable to external scoring model, will be the least reliably learned by MSM and least informative for pretraining.
The resulting frames are sent as input the MSM architecture and the final loss objective L is determined by either of the MSM objectives Lwv or Lub described in Sections 2.3 This modified training objective allows the model to focus on learning from gradients calculated from the frames with high confidences.
3.2
ATM WITH MSM LOSS SCALING (ATM+S)
The ATM loss is computed over the frames with high confidence performing fine ane-grained data selection within th speech sequence X,, Utterances with higher average frame confidence as measured by the scorer are accorded higher value than those with more confused frames. To perform data selection at coarser utterance level, confidence scores s. are computed as:
81
St.a
(5)
For simplicity, we denote = 8.. and the MSM loss computed over each masked frame is scaled by impose the importance of a particular utterance The final training objective farm of particular speech sequence is denoted as:
Larm 8 Latm
(6)
3.3 PROBABILITY AS CONFIDENCE MEASURE IN ATM
The ATM uses probability as simple form of confidence measure to each frame. The analysis of confidence measures for semi-supervision ASR has been done in Vesely et al. (2017) and they show that posterior probability acts reliable confidence measure for frame, word and sentence
4

---

Under review as a conference paper at ICLR 2022
based data selection. They also perform extensive analysis using the posteriors for hybrid ASR systems. Based on the motivation from this work we chose to use softmax probability directly confidence score. A similar observation has been noted in Ferreira et al. (202T). where the usage of softmax probability directly as a confidence measure has been applied to select relevant data samples during training. We also experimented with Entropy and exponential scaling log scaling on softmax probabilities as confidence measure, but it did not fetch advantage over simple usage of probability.
4 EXPERIMENTAL SETUP
All experiments including pre-training and fine-tuning are performed using 80 dimensional log Mel- filterbank features computed over the sampled 16kHz audio. Datasets (such AMI) contains wide- band audio and are downsampled to 16kHz. We evaluate with the test-other (LibriSpeech partition) to show the importance of ATM matched data conditions, while IHM-eval and SDM-eval (AMI partitions) used to validate the model under mismatched conditions.
4.1 DATASETS USED
Pretraining (PT): Libri-light (LL,-60k) dataset contains 60k hours of unlabeled speech and used
to pre-train all MSM models. LL-60k is the most widely used large unsupervised speech corpus for various techniques. Each input speech sequence constructed by first randomly selecting 32-64 seconds segments from the original utterance. From these segments, a contiguous 32 second region is extracted from random starting point on-the-fly thy-flyy during MSM PT as described in Zhang et al. 2020b).
Finetuming (FT): Different target datasets including 1) 100 & 960 hours of Librispeech (LS-100 & LS-960) (Panayotov al. 2015). 2) 100 hours of AMI and 3) speechstew (approx. 5k hours) (Chan et al. 2021) are used to perform our FT experiments. Each dataset used is specific to a certain target data condition, for instance LS-960 is closely matches the LL-60k, AMI dataset is distinct from the LL-60k condition and it contains speech from two kinds of microphones (i) Independent head microphone (IHM). (ii) single distant microphone (SDM). SpeechStew is composed of datasets chosen from multiple conditions to create mixed domain aggregate corpus. Details of its processing are described Chan al. (2021).
Evaluation: We hypothesize that evaluation over AMI using IHM-eval and SDM-eval reveals the effectiveness of ATM providing informative samples for better representation learning. We also evaluate using evaluation sets from Tedlium and Common voice their training counter parts are used in SteechStew based FT. Finally, we also evaluate using CHiME-6 Watanabe et al. (2020) without using any FT data from CHiME-6 training set to compare the performance of ATM on completely unseen target dataset.
Scorer training data: A CTC (Graves et al. 2006) based conformer model with 100M parameters trained on LS-100 ("LS-scorer"), A similar model is also trained on AMI ("AMI-scorer"). Word- piece model (WPM) with 1024 tokens used as labels for training the scorer models. All the results in this paper use "LS-scorer" besides the comparison Section 5.3
4.2 MSM ARCHITECTURE
Mav2vec2-conforeee This is wav2vec2 with conformer based context network which first encodes the filterbank features using two 2D convolutional layers with strides (2,2). Model has 100M/600M parameters and is denoted as "w2v2-conformer-LXXX 12v2-conformer--/X
similar to it differs in using the k-means based quantizer with 1024 targets and com- putes the cross-entropy loss as described in Hsu al. (2021b). The "L/XL" size models contains context network Ω 12/24 conformer layers with 8 attention heads and 1024 hidden dimensions.
HuBBERTT-confrmme-LLXXX
W2y-BERT: W2y-BERT is explored using two model sizes: one with 100M parameters denoted as "w2v-BERT-L" and containing 2 conformer layers in context net Ω and 4 conformer layers in A. 600M parameter model is denoted as "w2v-BERT-XLL contains conformer layers in and 24 conformer layers in A. Each conformer block contains 1024 hidden dimensions with 8 attention
5

---

Under review as a conference paper at ICLR 2022
heads, kernel size of 5 with local context of 128. The remaining architecture is identical to the configuration defined in Chhung et al. (2021).
4.3 PT AND FT CONFIGURATION
The models L/XI. are trained with global batch size of 512/2048 on 64/256 Google TPU V3 cores for 2-4 days respectively. Adam optimizer used with learning rate schedule (Section 5.3 of Vaswani et al. (2017)) with 2c-3 peak learning rate and 25k warmup steps. The model training configuration follows similar procedure as described in Zhang et al. (2020b).
The FT is done by employing the context network from the PT model by adding a final layer with 1024 WPM units learnt using the RNN-T objective function (Zhang et al., 2020a). The FT is done on w2v-BERT-XX
w2V2-conformer-XX XL and HuBBERT-coformer-- XL after 400k PT model updates. The W2v-BERTHH model is FT after 900k PT model updates. w2v-BERT-- is used to initially perform wide range of analysis and hypereparameter optimization ATM. w2v-BERT-XL. is finally used to compare the results of ATM across existing works in literature. w2v2-conformer-XX and HuBERT-con corT--ooofrme---L are also used in our experiments. All these models trained with the same configuration as in Zhang et al. (20205).
ATM ANALYSIS
The empirical study on ATM is done primarily using w2v-BERT-L since this generates the best WER performance across similarly sized models (cf. Figure 2 The pre-trained models fine- tuned with either LS-100 or AMI. The resulting finetuned models are evaluated on IHM and SDM evaluation sets understand the domain generalization aspect of ATM. Librispeech evaluation sets are used in unison to study how ATM behaves under matching domain condition. These experiments performed with using the loss scaling (it will be discussed in Section 5.5)
5.1 MASKING PERCENTAGES
The number masked frames within an utter- ance plays key role in masked input learning and in this study, we vary the masking percent- ages from 30% 50% to determine the best percentage for ATM approach. Previous works on wav2vec2 (Baevski et al. (0200) showed that masking 49% of the frames is ideal for 30 second utterance and this has been followed subsequent works such as HuBERT and w2v-
Random-IHM
ATM-HHM
Random-test-oe teabb-hhh-tttttehee
ATM-ttest-ther
20
12
on IHM--val
18
11
% WER on lest other
WER on
18
10
BERT. case of ATM, this can differ as the frames selected are of higher confidences. Fig- usooss that ATM achieves its "sweet spot" with 40% masking for both IHM-eval and test- other set. Interestingly ATM's performance is stable across large variations in masking rates with relatively good performance with mask- ing rate low as 30%. This is a significant difference from the uniform sampling of prior work which suffers significant drop in perfor- mance as the masking rate goes below 40%. The result indicates that masking the right set of frames, which ATM aims to do, is able to pro- mote more stable performance. For instance, ATM achieves %WER of 12.65 with 33% masking and 12.52 with 40% masking on IHM- eval respectively as shown in Figure [ The recognition performance on test-other and IHM-eval improves over baseline from 8.86% to 8.79% and 13.38% to 12.52% respectively by using ATM.
14
30
35
45
50
Masking
Figure 1: Recognition performance of w2v-BERT with ATM and random masking IHM-eval and test-other sets by varying the masking percent- age during pre-training. The FT is performed on LS-100 for evaluating test-other, while IHM- eval evaluated with model FT with AMI. Ran- dom masking shows a substantial shift in perfor- mance when varying the masking from 30% to 40% while the ATM remains robust to changes in masking percentage.

---

Under review as a conference paper at ICLR 2022
5.2 ATM MASKING STRATEGIES
The default setting of ATM is chosen based on hypothesis that those frames that are scored with high-confidence from an external scoring model will be most useful as candidates for MSM pretraining. This hypothesis is interro- gated in this section by analysing the impact of choosing the frames with low confidences or equal mix of both high and low confidence frames (Mixed). For masking low confident frames, we modify the score in Equation 3
Table I: Performance comparison (in %WER) AMI evaluation sets using w2-BERT with random masking (baseline) and with ATM using high, low and mixed confidence scores from the scorer. The FT is done with AMI.
Model
Confidence Level
PT-LL. FI-AMI
IHM-eval
SDM-eval
as:
Baseline Random
13.38
31.63
High
12.52
27.34
8t = max p(ee = 1 E)
(7)
ATM
Low
14.14
37.77
1
Mixed
13.96
30.51
We evaluated these three masking strategies of ATM on both IHM and SDM evaluation sets. Table T shows the comparison between these sampling strategies. We observe that masking high confident frames are consistently better than masking the low confidence counterparts. In fact, "Low" confident frames perform worse than the baseline with random masking. Finally, we observe that performance of "Mixed" falls between that of "High" and "Low". The "Mixed" strategy is similar to random masking, as both high and low confidence frames are selected. This similarity is also reflected in comparable performance between "Mixed" and random masking. These results provide support for our initial hypothesis that masking frames with high confidence leads to better pre-training.
5.3 How TO CHOOSE THE SCORING MODEL
The scorer used in this work a speech recognition model (100M parameters) trained in supervised fash- ion. The scorer is chosen based on the target down- stream task and in addition this, the scorer needs to be frame-synchronous to provide confidence for every frame in a speech sequence. In this work, we use frame- synchronous ASR system as the scorer by employing the connectionist temporal classification (CTC) objective. The CTC is preferred over the RNN-T by analysing the reliability of the frame-level predictions. To analyse the importance of the supervised data used to train the scorer, we trained two scorer models: I.S-scorer and AMI-scorer are CTC models trained with LS-100 and AMI dataset respectively. The AMI-scorer outperforms on SDM-eval by improving the &WR from 27.34 to 27.00. Surprisingly, our results on Table 2 shows that the results on IHM-eval using an LS-scorer are comparable to the AMI-scorer. Evaluation on test and test-other shows that LS- is better than A.I--corrr on both sets. Based on these observa- tions, we choose the LS-scorer as the universal scoring model for all ATM based pre-trained models regardless of the target domain (eg: AMI) used in our experiments. Table 2 shows that although matching the scorer to the target domain improves the performance, the difference is not significant.
Table 2: Cross analysis of ATM perfor- mance (in % &WER) using AMI and LS scorers. The FT is done on LS-100 to evaluate the test and test-other, while the FT is done on AMI to evaluate us- ing IHM-eval and SDM-eval
Evalset
LS-scorer
AMI-scorer
test
3.89
3.93
test-other
8.92
9.68
IHM-eval
12.52
12.3
SDM-eval
27.34
27.00
5.4 CONSISTENCY ACROSS DIFFERENT ARCHITECTURES
Figure 2 shows that ATM consistently outperforms on both IHM-eval and SDM-eval across multiple MSM architectures including wav2vec2 and HuBERT. In the case of IHM-eval, ATM attains a relative improvement of 9% over w222-conformer-L, 4% relative improvement over HuBEERT-confommer-L and 5% relative gain over w2v-BERT-L baseline models respectively. W2v2- conformer-L using ATM obtained 6.2% relative improvement over its baseline counterpart and HuBEERT-coffrmerr_] with ATM attained 7.9% rel. improvement over HuBERT-L baseline on SDM-eval respectively. On the other hand w2y-BERT-L baseline is better compared to w2v2- conformer-L. and HuBERR--confrmerrL on both IHM-eval and SDM-eval by achieving 12.52% and 27.34% WER respectively.
7

---

Under review as a conference paper at ICLR 2022
Baseline
15
14.63
Baseline
ATM
34
% WER on SDM-eval
% WER on IHM-eval
14
32
13.31
13.38
31.63
30
30.25
13
29.47
28
27.34
12
26
w2v2-dfr-L HUBERT-cfi-L W2v-BERT-L
w2v2-di-L
HuBERT-dfr-L W2V-BERT-L
Figure 2: Performance comparison of different MSM architectures with and without applying ATM on IHM-eval and SDM-eval in AMI. All these models are FT using AMI. Here "cfr" refers to conformer.
5.5 ATM WITH LOSS SCALING (ATM+S) ANALYSIS
ATM training incorporate utterance-level weighting by scaling the MSM loss obtained using w2y-BERT-L. models with the utterance-level confidence score according to Equation 6 We evalu-
Table 3: WER ATM+S by fine-tuning on AMI using w2v-BERT-L. model
MSM arch.
Type
IHM-eval
SDM-eval
Baseline
13.38
31.63
w2v-BERT-L
Baseline+S
13.14
28.16
ATM
12.52
27.34
ATM+S 13.05
27.19
ate the value of utterance-leve loss scaling by re- re-weighting utterances in the context of both base- line MSM (i.e., without ATM frame selection) and ATM (ATM+S). These results are in Table 3 Re-weighting utterances by scaling the MSM loss with the confidence score on baseline model is denoted as "Baseline+S" and ATM is labeled "ATM+S". MSM loss scaling is effective even without ATM; baseline+S improves over baseline on both IHM-eval and SDM-eval. Moreover, ATM+S improves
ATM SDM-eval by attaining 27.19% WER while showing degradation on IHM-eval. This shows that ATM+S is effective on very hard evaluation task such as SDM-eval compared IHM-eval. On the IHM-eval test set, the impact of MSM loss scaling is observed the Baseline MSM without ATM. We hypothesize that ATM+S may not able to provide improve- ment IHM-eval as ATM already incorporates optimally incorporates scorer information on this task.
RESULTS
In this Section, XL models are used to compare the importance of ATM on LS-960, AMI and SpeechStew. These three datasets show the effect of ATM diverse conditions with much larger model. Results are compared with appropriate prior work. Table shows that ATM and ATM+S improves over dev-other while on dev there was improvement only using ATM and ATM+S. Although the ATM and ATM+S does not show improvement on test and test-other, matches the very strong baseline. Considering the similarity between I.S-960 and PT data, ATM manages to provide gains without hurting the performance across all Librispeech evaluation sets. This validates argument that MSM models are better without any data selection when trained under matched data condition but can benefit under mismatched conditions.
Table Jjolsentsesents the results of ATM on AMI by comparing it with 22v2-conformer-y XL baseline and w2v-BERT- baselines. We include w2y2-conformer-XL to further test the consistency of ATM on XL models when evaluated on harder tasks. ATM+S and ATM observes consistent gains over baseline both IHM--val and SDM-eval when trained with XI. models. However, ATM+S did not
8

---

Under review as a conference paper at ICLR 2022
Table 4: %WER obtained by FT with LS-960 using w2v-BERT-XL model using baseline, ATM
and ATM+S. The results show the impact of our proposed approach on matched condition since Librispeech evaluation sets are treated as closer Libri-light PT domain.
MSM arch.
Type
dev
dev-other
test
test-other
w2y2-conformer-XL
Baseline (Chung et al. 2021
1.7
3.5
1.7
3.5
Baseline (Chung et al. 2021)
1,5
2.9
1.5
2.9
w2v-BERT-XL
ATM
1.4
2.8
1.5
2.9
ATM+S
1.5
2.8
1.5
2.9
Table 5: %WER obtained by FT with AMI using w2v-BERT-XL model using baseline, ATM and
ATM+S. Evaluation is done on AMI test sets to highlight the effect on mismatched condition.
MSM arch.
Type
IHM-eval
SDM-eval
Baseline
10.4
25.7
w w2V2-conformer-XX XL
ATM
10.0
24.5
ATM+S
9.8
23.9
Baseline
10.1
25.1
w2v-BERT-XL
ATM
9.5
23.7
ATM+S
9.5
23.5
demonstrate improvement on IHM-eval using w2V-BERT-XI. Table 0 analyses the effect of ATM
Table 6: Comparison with state-of-the-art results on SpeechStew. The FT is done on SpeechStew and the results are evaluated using Kaldi scoring to match published results. Note that the model has never seen any CHiME-6 data, and we use it as an example for zero-shot learning mode.
AMI
Model
Type
Commonvoice
Tedlium
CHiME-6
IHM
SDM
Speechstew
12.1
5.3
9.0
21.7
57.2
Chan et al.
2021
w2v2-conformer.
11.5
5.6
9.6
23.8
56.4
Chan al.
2021
Baseline ATM
11.2
5.3
9.2
21.5
55.5
w2v-BERT-XI.
10.8
5.3
9.0
21.0
54.3
ATM+S
10.7
5.2
8.9
20.7
53.9
and ATM+S on multiple evaluation sets such as Commonvoice, Tedlium, AMI and CHiME-6. These four sets are chosen based on the mismatch range from minimum to maximum and for instance, Commonvoice has the minimum mismatch with Libri-light data, while CHiME-6 has the maximum mismatch. The state-of-the-arr results published in Chan al. 2021 obtained by choosing the best Conformer model supervisedly trained with multiple datasets such as AMI, Common Voice, Broadcast News, Librispeech, SwitchboardFischhrr. TED-LIUM and Wall Street Journal. Note that the training data did not include the CHiME-6 data. The authors in Chan et al.) 2222 show that simply training ASR with lots of data leads to best results compared the wav2vec2 finetuned model. Their best results are denoted in table and will be used to compare with our best ATM results.
Our baseline 22y-BERT-XL attained better results over the published w222-conformer-Xl and Speechstew results. Commonvoice and CHiME-6, the baseline attained 7.4% and 2.9% rela- tive improvement over Speechstew respectively. However, by including our ATM and ATM+S with w2v-BERT-XL, there was consistent improvement across all range mismatched domains. For in- stance, ATM+S attains 5.76% relative improvement on CHiME-6 over the Speechstew. This result clearly justifies that selection of reasonable input samples during pre-training reduces the necessity of having finetuning data from the same domain to improve performance. To further substantiate this, the results on AMI show 4.6% relative improvement on AMI-SDM over Speechstew which is of different domain compared to pre-training domain. In case of minimal mismatch domain such as
9

---

Under review as a conference paper at ICLR 2022
Commonvoice, the ATM attained 11.6% relative improvement over Speechstew. These observations show that ATM and ATM+S demonstrate their effectiveness to generalize to unseen and challenging speech recognition conditions.
CONCLUSION
In this work, we introduce ask2mask (ATM) to perform data selection over unsupervised samples for MSM based pre-training to focus relevant information and learn meaningful representations. ATM achieves 21.0% WER on mismatched AMI SDM set with guided masking and 20.7% WER is obtained by including loss scaling (ATM+S). We empirically show that ATM is more robust to changes in masking percentage compared to random masking. as typically used in MSM. Our re- sults substantiate the importance of learning from high confident frames by attaining improvements across multiple evaluation sets. An important aspect of AT' approach is its flexibility to incorporate into any MSM pretraining techniques and ATM+S pre-training methods. In our future work, we wish to apply ATM over pretraining data contain- ing data from multiple domains (Hsu al., 2021a Likhomanenko improvements. We also consider two future enhancements to ATM: (1) Joint training of the scorer model with MSM model by simultaneous training on supervised and unsupervised data. (2) Perform active learning by sharing the parameters of MSM with the scorer once the MSM is well trained.
also be easily adopted into self-supervised
2020) achieve further
ETHICS STATEMENT
While are unaware of any specifie bias in Ask2Mask, is possible that bias within the Speech- Stew and LibriLight training corpora themselves may introduce bias to the resulting ASR model. Ask2Mask uses an ASR model trained on labeled data as scoring model identify regions for mask- ing; any bias in this scoring model (trained on LibriSpeech-100 100 or AMI) could also impact the fine-tuning process, leading bias in the final model. Without being aware of specific biasing effects, it is unclear how, if at all, biases in the scoring model and downstream ASR model may interact with each other.
Automatic Speech Recognition (ASR) can exacerbate security and privacy issues by facilitating the search and analysis of speech, though in and of itself ASR does not pose a security nor privacy issue.
REPRODUCIBILITY STATEMENT
All of the corpora used for I) training the scorer model, and 2) pretraining and finetune the ASR models, as well as 3) all evaluations are publicly available and use standard published partitions. We aim to describe Ask2Mask and Ask2Mask with scaling comprehensively in the text to facilitate reproducibility. Implementation
major components, Conformer RNN-T models, wav2vee 2.0 and CTC training are available within in public, open-source libraries including Nvidia-nemo (link) and ESPNet (link). This supports the reproducibility Ask2Mask and the results reported this paper within multiple frameworks. Additional details to promote reproducibility are provided in the Appendix.
REFERENCES
Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A frame- work for self-supervised learning of speech representations. arXiv preprint arXiv:2006. 11477, 2020.
William Chan, Daniel Park, Chris Lee, Yu Zhang, Quoe Le, and Mohammad Norouzi. Speechstew: Simply mix all available speech recognition data to train one large neural network. arXiv preprint arXiv:2104.02133 2021.
Yu-An Chung, Yu Zhang, Wei Han, Chung-Cheer Chiu, James Qin, Ruoming Pang, and Yonghui Wu. W2v-bert: Combining contrastive learning and masked language modeling for self- supervised speech pre-training. arXiv preprint arXiv: 108.0200000 2021.
10

---

Under review as a conference paper at ICLR 2022
Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning. arXiv preprint arXiv: 18829, 2019.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: /eXXiv:88/0.0 1810004805 2018.
Jonathas Ferreira, Marcele Mendonca, and Paulo SR Diniz. Data selection in neural networks. IEEE
Open Journal of Signal Processing, 2021.
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist tem- poral classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pp. 369-376, 2006.
Geoffrey Hinton and Richard S Zemel. Autoencoders, minimum description length, and helmholtz free energy. Advances in neural information processing systems, 6:3-10, 1994.
Wei-Ning Hsu, Anuroop Sriram, Alexei Baevski, Tatiana Likhomanenko, Qiantong Xu, Vineel Pratap, Jacob Kahn, Ann Lee, Ronan Collobert, Gabriel Synnaeve, et al. Robust wav2vec 2.0: Analyzing domain shift in sule-super pre-training. arXiv preprint arXiv:2104.01027, 2021a.
Wei-Ning Hsu, Yao-Hung Hubert Tsai, Benjamin Bolte, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: How much can a bad teacher benefit asr pre-training? In ICASSP 2021- 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6533 6537. IEEE, 2021b.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel Weld, Luke Zettlemoyer and Omer Levy. Span- bert: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77, 2020.
Jacob Kahn, Ann Lec, and Awni Hannun. Self-training for end-to-end speech recognition. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7084-7088. IEEE, 2020.
Tatiana Likhomanenko. Qiantong Xu, Vineel Pratap. Paden Tomasello, Jacob Kahn, Gilad Avidov, Ronan Collobert, and Gabriel Synnaeve. Rethinking evaluation in asr: Are our models robust enough? arXiv preprint arXiv:2010.11745. 2020.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 5206-5210. IEEE, 2015.
Daniel Park, Yu Zhang, Ye Jia, Wei Han, Chung-Cheng Chiu, Bo Li, Yonghui Wu, and Quoc V Le. Improved noisy student training for automatic speech recognition. arXiv preprint arXiv:2005.09629 2020.
Zhongzheng Ren, Raymond Yeh, and Alexander Schwing. Not all unlabeled data are equal: Learn- ing to weight data in semi-supervised learning. Advances in Neural Information Processing Sys- tems, 33, 2020.
Henry Scudder. Probability of error of some adaptive pattern-recognition machines. IEEE Transac- tions on Information Theory, 11(3):333-37,, 1965.
Yu Sun, Shuohuan Wang. Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. Emnee: Enhanced representation through knowledge integration. arXiv preprint ar arXiv:1904.0000
2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Karel Vesely, Lukáš Burget, and Jan Cernocky. Semi-supervised dnn training with word selection for asr. In Interspeech, pp. 3687-3691, 2017.
11

---

Under review as a conference paper at ICLR 2022
Chengyi Wang, Yu Wu, Yujiao Du, Jinyu Li, Shujie Liu, Liang Lu, Shuo Ren, Guoli Ye, Sheng Zhao, and Ming Zhou. Semantic mask for transformer based end-to-end speech recognition. arXiv preprint arXiv: arXhv:191..03010 2019.
Shinji Watanabe, Michael Mandel, Jon Barker, Emmanuel Vincent, Ashish Arora, Xuankai Chang, Sanjeev Khudanpur, Vimal Manohar, Daniel Povey, Desh Raj. David Snyder, Aswin Shanmugam Subramamian, Jan Trmal, Bar Ben Yair, Christoph Boeddeker, Zhaoheng Ni, Yusuke Fujita, Shota Horiguchi, Naoyuki Kanda, Takuya Yoshioka, and Neville Ryant. Chime-6 challenge:tackling multispeaker speech recognition for unsegmented recordings, 2020.
Frank Wessel, Ralf Schluter, Klaus Macherey, and Hermann Ney. Confidence measures for large vocabulary continuous speech recognition. IEEE Transactions speech and audio processing, 9(3):288-22) 298, 2001.
Qiantong Xu, Alexei Baevski, Tatiana Likhomanenko, Paden Tomasello, Alexis Conneau, Ronan Collobert, Gabriel Synnaeve, and Michael Auli. Self- Self-training and pre-training are complemen- tary for speech recognition. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3030-3034. IEEE, 2021.
Qian Zhang, Han Lu, Hasim Sak, Anshuman Tripathi, Erik McDermott, Stephen Koo, and Shankar Kumar. Transformer transducer: A streamable speech recognition model with transformer coders and rn-tt loss. In /CASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7829-7832 IEEE, 2020a.
Yu Zhang, James Qin, Daniel Park, Wei Han, Chung-Cheng Chiu, Ruoming Pang, Quoc V Le, and Yonghui Wu. Pushing the limits of semi-supervised learning for automatic speech recognition. arXiv preprint arXiv:2010.10500 2020b.
A APPENDIX
A.1 DETERMINE THE BEST PT AND FT CHECKPOINT (CKPT.)
Selecting the best PT ckpt. to perform the fine-tuning is cumbersome as it involves finding the best PT ckpt. based on the validation accuracy quantized targets and also run fine-tuning experiments on multiple PT ckpts. Figure 3 shows that ATM starts to show improved performance after 250k PT ckpt. On contrary our belief that ATM might require less FT, both baseline and ATM requires same number of FT iterations to achieve the best performance.
100000
80000
FT
12
S0000
40000
30.0
10
0
45
10000
90
0.6
0.4
0.
166
le6
Figure 3: 2D plot of the &WER IHM-eval across PT and FT ckpts. using baseline (left) and
ATM (right) models. The FT done using AMI.
A.2 ANALYSTS OF SCORER BASED ON & WER ON PT DATA (1 HOUR OF LIBRI-LIGHT
We further analyse the impact of the recognition performance of the scorer on ATM, by choosing three categories of scorers such bad, better and best. The 10 hours of supervised libri-light (LL- 10h) data used evaluate the performance these scorers and the results are as follows: Initial model checkpoint (ckpt.). LS-scorer attains 51.8% WER and AMI-scorer achieves 72.7% this
12

---

Under review as a conference paper at ICLR 2022
ckpt respectively. Intermediate model ckpt. Evaluating LS-scorer at this ckpt., achieves 45.8% and AMI-scorer achieves 57.1% WER. Final model ckpt. chosen after convergence. The LS-scorer gains 35.5% WER and AMI-scorer attains 46.8% by evaluating at this ckpt. Figure 4 shows more
LS-scorer performance
AMI--corer performance
S03
23
27:34
27
27.12
27.4
12:12
12:02
12.3
12:15
Figure 4: Comparing the behavior of ATM using LS-scorer and AMI-scorer having different % WER on hour of Libri-light (LL-Ihr) with AMI evaluation sets)
challenging scenario by testing the ability ATM using scorers from different domains with varying range of performances. A LS- with 51.8% WER LL-1hr performs well both IHM-eval and SDM-eval, while the AMI-scorer with 46.8% WER slightly improves over the above mentioned LS-scorer. The key aspect in selecting better scorer is choose ckpt. either from the category which achieves % WER within the range of 50-60 on the small subset of pre-training data.
A.3 ANALYSIS OF ATM ON LIBRISPEECH
Our ATM analysis is also done on w2v- ----ETT-L to validate the ATM on LS-100 and is present in table [ ATM attained 4.3% and rel. imp. on test-other over the baseline model. Apart from that there is slight change in test set which can be treated as noise. On both dev-other and test we did not find any improvement and instead there was slight degradation in performance.
Table 7: %WER of ATM by varying the frames selection based on high, low and mixed confidence (conf.) scores using w2v-BERT-L all evaluation on Librispeech.
PT-LL, FT-LS100
Model
Type
dev
dev-other
test
test-other
w2v-BERT-L
Baseline
3.78
8,86
3.85
9.32
High-conf. 3.71
8.97
3.89
8.92
ATM
Low-conf. 3.98
9.59
4.15
9.76
Mixed-conf. 3.87
9.25
3.98
9.42
To further evaluate the impact of increasing the model parameters from "L" size to "XL" size, we FT LS-100 using MSM models with XL size and the results are table 8 We did not find any consistency in the performance across the evaluation sets using any of the MSM architectures. Slight gains are observed on test or dev or dev-other using w2v2-conformer-XL Once the baseline in w2v- BERT-XL gets better, ATM did not achieve gains on test-other. This scenario can be explained due to effectiveness of MSM pre-training under matched condition and can perform well without any ncessary data selection approach.
A.4 ATM+S ANALYSIS ON VALIDATION DATA DURING PT
The effect of ATM and ATM+S is analysed by plotting the validation scores dev-other during pre- training.. The first plot in figuresJshows that the contrastive loss improves over the baseline with the aid of ATM and is further enhanced with ATM+S. The second plot shows the number of unique
13

---

Under review as a conference paper at ICLR 2022
Table 8: Performance comparison of different MSM architectures with and without applying ATM on all evaluation sets on Librispeech.
PT-LL, FI-LSI00
Model
Type
dev
dev-other
test
test-other
w2v2-Conformer-2 XL
Baseline
2.5
4.7
2.6
4.9
ATM
2.4
4.6
2.5
5.0
HuBERT-Confome--XX
Baseline
2.5
4.7
2.6
5,0
ATM
2.5
4.6
2.5
5.0
W2V-BERT-XL
Baseline
2.4
4.4
2.5
4.6
ATM
2.3
4.4
2.4
4.7
Baseline
0.96
29
ATM ATM+S
0.18
Conrastive cos
2.05
MSM Acturacy
0.94
28
Uniqus
0.9
0.92
Baseline
0.14
Baseline
0.91
ATM
ATM
2.65
ATM+S
0.12
ATM-5
0.1
0.2
0.3
0.4
0.5
0.1
0.2
0.3
0.4
0.5
0.1
0.2
0.3
0.4
0.5
Thaining Beradions (n million)
Training herations millio]]
Training Berations (In million)
Figure 5: Validating baseline, ATM and ATM+S using contrastive loss, number of unique codes used from the codebook and the MSM accuracy on dev-other during pre-training. A small bump is observed around 0.1 million training iteration due the change in learning rate and ignored during analysis.
codes used from the quantizer codebook. Analysing this plot helps us to understand if the validation loss accuracy is improved by just using less % of unique codes which will affect the performance at FT. Among the 1024 codes, 94%-95% are used by both ATM and ATM+S. This is similar to the % unique codes used by baseline model and confirms that improvement of ATM and ATM+S is not by choosing smaller set of unique codes. The third plot shows that the MSM accuracy of ATM and ATM+S improves over bascline model. ATM+S shows that re-weighting each utterance complementary the ATM.
A.5 COMPARISON BETWEEN FRAME-LEVEL. AND UTTERANCE-LEVEL LOSS SCALING
ATM+S performs MSM loss scaling using the utterance-level confidences which performs focus each utterance at coarse level. We also experimented with scaling with frame-level confidence scores. Our experiments showed that scaling all utterances with frame-level confidence hurts the model performance. To solve this issue, we randomly selected utterances which participate in frame scaling. Scaling 10% utterances resulted better performance and the results are shown in Table
Table 9: % &WER on Librispeech evaluation sets using ATM with utterance scaling and frame scaling. The frame scaling analysed with choosing the best percentage of utterances that participate in scaling.
PT-LL, FT-LS100
Model
Type
dev
dev-other
test
test-other
w2v-BERT-L
Baseline
3.78
8.86
3.85
9.32
None
3.71
8.97
3.89
8.92
Utterance-level
3.64
8.79
3.95
8.95
ATM
Frame-level-10s;
3.73
8.97
3.84
9.06
Frame-level-50e
4.01
9.35
4.14
9.54
Trame-level-1009
4.23
9.77
4.89
9.97
14

---

Under review as a conference paper at ICLR 2022
A.6 RESULTS ON COMMONVOICE WITH PUNCTUATION NORMALIZATION
Some previous work, including Likhomanenko et al. (2020), report Commonvoice results by scoring with normalization process that removes punctuation. For comparison, do the same in Table
Table 10: WWR on Commonvoice using models FT with speechstew.
Model
Without norm.
With norm.
Speechstew (Chan et al. 2021))
12.1
9.7
W2222-oofformer-X (Chan et (2021)
11.5
9.1
W2v-BERT-XL
11.2
9.3
w2v-BERT-XL ATM
10.8
9.2
w2v-BERT-XX + ATM+S
10.7
9.1
A.7 BLOCK DIAGRAMMATIC VIEW OF MSM ARCHITECTURES
C1
Masking
Net
82
1/2
C2
Bottleneck Features
63
K-means Cluster Computation
1/1
Cross-Entropy
65
Loss
Figure 6: Working procedure of HoBBore-oofoomer model described in section 2.3 means cluster ids labels and they are refined using the bottleneck features extracted from the context network itself.
The k-
C1
C1
Masking
Net
6
63
Q1
Contrastive Loss
Gumbel- Softmax
Q2
64
Quantizer
65
Figure 7: Working procedure of Waw2vec2-conformee model described in section 2.3 The coded representations are masked and passed to context network Ω and the resulting output is learnt to be closer quantized output
A.8 STATISTICAL SIGNIFICANCE ANALYSIS ON % WER PERFORMANCE FOR MULTIPLE EVALUATION SETS
The table results are on Librispeech and obtaining 0.1% improvement in Librispeech testsets is statistically signficant. For instance, the dev-clean test set contains 54402 words and 0.1% gains denotes a recovery
words. Also, the recent works on self-supervised training such as HuBERT shows improvement between wav2vec2-Large and HuBERT-Large only on dev-clean with 0.1 % gain in Table
15

---

Under review as a conference paper at ICLR 2022
C1
?
C1
Masking
62
Net
e3
Cross-Entropy Loss
64
Gumbel- Softmax
M/
Quantizer
65
Q1
Contrastive Loss
2/
Figure 8: Working procedure of W2V-BERT model described in section 2.3 Cross-entropy loss is computed between predictions of context network and the quantized labels Vj- Contrastive loss computed in parallel as way2vec2.
Table 11: %WER on Commonvoice using models FT with speechstew.
Dataset
Abs. &WER Imp.
# Total words
Words recovered (~)
dev-clean
0.1
54402
54
test-clean
0.1
52576
52
AMI-IHM
0.1
89635
16


Reviews:
[{"review_id": "vMcHdOsyF7", "reviewer": "Reviewer_YoKo", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n\n- Experiments are extensive to evaluate different aspects of the proposal (amount of masking vs. WER, type of scorer vs WER, effect of ATM approach vs. ATM+scaling approach)\n- The method is relatively easy to implement with existing open source tools. \n\nWeaknesses:\n\nNotation of the equations needs some updates, for example:\n- Eq. 2. $L$ is used both as an index and as a set of codebook indices\n- Eq. 5. The denominator should use a summation index different than $t$\n\nThe novelty is somewhat limited. Having a weighting mechanism instead of uniform distribution for masking is relatively straightforward extension of the existing masked based modeling approaches. \n\nSome details are missing. Please refer to the questions below:\n1) Introduction: “The scoring model is necessarily a speech recognition model trained on small amount of data” -> it could have been a large model on a known larger dataset. Would a better scorer help with ATM?\n2) “ignoring utterances with low-confidence” -> Not exactly true, to the reviewer's understanding, confidence level is just a scaling factor. Or do the authors put a threshold on scores?\n3) In the introduction of Section 5, which part of AMI is used as the training subset? Without mentioning this detail, it is hard to see why it can show the domain generalization aspect. Since the experiments usually mention LS fine-tune -> test on LS test, and AMI fine-tune and test on AMI test set, the generalization claim is not very clear.\n4) Details of fine-tuning is missing. Do the authors use, for instance, w2v-BERT-L + ATM based embeddings and train a new ASR model? Or do the authors add a new output layer to map input speech to word-piece units? \n5) In Table 4 and its explanation, do the authors refer to match and mismatch based on the dataset used in pre-training and fine-tuning?\n\nMinor typos: Introduction, first paragraph: pesudo -> pseudo;\nConfidences cores -> confidence scores\n\nPlease add reference number to CHiME-6 dataset in Section 4.1.", "summary_of_the_paper": "This study investigates the use of an external frame-synchronous CTC-based ASR system to get confidence scores for frames in masked speech modeling which is called AskToMask (ATM). These confidence scores are used in two ways: \n1) as a distribution to determine the segment to mask (higher confidence means higher probability), \n2) the utterance level average of the per-frame confidence scores is used as a scaling factor of the per-utterance loss. \n\nThe experiments perform unsupervised pre-training using these two proposed methods and then performs supervised fine-tuning for the final ASR task. Experiments on Librispeech and AMI show that confidence score-based selection of masking improves the final WER as compared to using uniform distribution in the mask selection (the first approach versus baseline). The second approach provides marginal gains on top of the first approach. ", "summary_of_the_review": "The paper uses ASR based scores to mask high-confident regions in masked speech modeling. Using this weighting idea is not very novel even though it may be the first implementation in the speech domain. The main strength of the paper is that it provides extensive comparison for different aspects of the approach. The writing may benefit from some notational fixes and addition of some implementation details as discussed in the detailed review. From the discussion above, it can be seen that there are more weaknesses as compared to the strengths.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Hu5-fzWQXb2", "reviewer": "Reviewer_fRrU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I personally think that authors have a good point here, how to do better mask selection in self-supervised learning. Obviously, not all speech frames are\nof equal importance. Studies, such as in automatic speaker recognition have been made, where it was clearly found out that different phones carried \ndifferent amount of speaker recognition cues than others. In ASR, you can take non-speech region vs vowel and you get a big difference in importance. \n\nHowever, the way it was done here, I believe is not correct. First of all, the idea of self-supervised learning is to learn task independent features that\ncan be then used for any task. So, then in principle, the learning signal should be something that does not have any one task in mind. But okay, lets say\nyour intention is to do only ASR and your chosen model, of course then you can proceed as described in the present paper. But then your need to compare \nwith other ASR models that use _similar_ amount (and the same) labeled training data. Which in your case is the data used in the scoring model. So SoTA \nend-to-end mehtods need to be used (and ESPnet is a good way to start in looking for implementations https://github.com/espnet/espnet). \n\nAnd finally, if you look at the Table 6, the difference in WER betwene proposed, that uses extra data, vs baseline is not too great. Only in CHIME we see\nsome real improvement. \n\nMinor comments: \n\n- L_div is not explained in more detail. \n- Equation (3) is not needed. \n- This sentence is very suspicious: \"The scorer’s training data is chosen such that it is matches the target data condition\". Wouldn't it be better\nthat the whole learned model is as general as possible? Could you elaborate on this?\n- In Table 2, could you add the metric (WER) in the caption. \n- Please replace string \"%WER\" with \"WER (%)\". Also in the same vein it would be important to make all captions self-contained, so that \nreader should be able to understand the Table and Figure without referencing to the the main text. \n\nI have read the authors rebuttal and discussion and due to sufficient answer to my concerns I have raised the score. ", "summary_of_the_paper": "In this paper, authors proposed to use an external scorer to weight the frames to be masked for the MSM loss. Idea is interesting as not all speech frames are of equal importance. ", "summary_of_the_review": "Authors add more labeled training data, as an external scorer, and still results are significantly better. Authors need to compare against SoTA models where all models see exactly the same data. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5WJe8ii3oV", "reviewer": "Reviewer_UPo9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Improving training effectiveness through better masking is an important research direction for speech representation learning. \n\nStrengths:\nThe paper presents a robust experimental setup with three SOTA models (W2V, HuBERT, W2V-BERT) evaluated in two sizes L/XL on multiple datasets (LS, AMI, CHIME-6, Tedlium, and Commonvoice.\n\nWeaknesses:\nThere are three major points here:\n1) The paper presents an incremental improvement over the baseline systems with minor novelty. \n2) The proposed approach relies on access to labeled data during the pre-training stage, which departs from the standard setup for self-supervised learning. The pre-training stage in self-supervised learning is task agnostic to open the door for utilizing the learned representations in a wide array of downstream tasks.\n3) The observed gains are modest using the proposed approach. With 100h of labeled data, semi-supervised methods (either from random weights or initialized from self-supervised models) would bring much lower WERs.\n\nGeneral comments:\nEquation 5 encourage sampling from places with high confidence, which may lead to oversampling silence and vowel segments. How would you encourage the selection of diverse positive samples? \nThe primary motivation for filtering data in semi-supervised learning is to remove noisy inputs and ones with bad teacher labels/confidence. A similar goal can be achieved in self-supervised learning using voice activity detection (VAD) to remove long silences and other signal processing methods. Dealing with data problems in self-supervised learning has also been studied in (https://hal.archives-ouvertes.fr/hal-03070411/document)\nOne way to benefit from a supervised teacher is to benefit from the teacher segmentations and labels, not only confidence scores, for masking and negative sampling as done here (https://arxiv.org/pdf/2103.05149.pdf)\n", "summary_of_the_paper": "Many self-supervised speech representation learning methods use masked prediction at their core. This paper proposes Ask2Mask (ATM) approach for informed masking during learning through a supervised teacher model that provides frame-level posteriors probabilities of linguistic output units. The frame-posterior probabilities extracted from the supervised teacher model are also used as utterance weights to favor ones with high confidence.", "summary_of_the_review": "The paper presents an approach for improving self-supervised learning approaches using a supervised teacher model. The problem is important, but the proposed method and the results aren't convincing.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4wa5TY3iHtr", "reviewer": "Reviewer_EBBL", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- paper introduces confidence scores into masked modelling\n- paper shows that even simplest of schemes can lead to small-to-medium gains\n- comprehensive performance comparison across many speech data sets\n- large-scale, non-trivial, models and data sizes\n\nWeaknesses\n- limited technical novelty\n- technical quality of paper (numerous typos (eq 2, ), undefined terms ({\\bm E}^{rm}), sloppy notation (1,...,k-1 does not equal K), confusing notation (eq 5)); section 3.1 is meant to be the key section but you are making a very limited attempt to be clear not just for speech but also more general audiences \n- limited discussion about the use of confidence scores in ASR\n- lack of any discussion or investigation regarding suitability of softmax probabilities as surrogates of confidence scores\n- your claim about gains up to 11.6% seems to apply to CV task if you compare 12.1% of SpeechStew and your ATM+S approach. If this is true your statement in the abstract is misleading at best. Please focus on comparing unweighted and weighted schemes. Please make it clear when you are comparing your best number and someone else's best numbers. ", "summary_of_the_paper": "This paper describes a weighted masking scheme for learning speech representations. Weights in this scheme are surrogates of confidence scores obtained from an external ASR system. The paper also describes how utterance-level confidence scores can be incorporate to down-weight contribution from utterance that are likely to be problematic for an ASR system. \n\nThe paper makes use of Librispeech dataset for learning representations and Librispeech and AMI datasets for training external ASR systems. The paper evaluates both schemes on Librispeech, AMI, and some other datasets. Experimental results show that the proposed schemes offer small-to-medium gains over the unweighted reference masking scheme. \n\n", "summary_of_the_review": "This paper explores the use of weighted masking for learning speech representation. The proposed scheme relies on confidence scores obtained from external ASR systems to decide which speech segments should and should not be used for masked modelling. An extensive empirical investigation shows that weighted masking is a promising direction for learning accurate speech representations. \n\nAlthough the technical novelty is limited, this paper introduces weighted masking by means of confidence scores (very popular subject) into speech representation learning area. Unfortunately, this paper currently contains a number of technical issues (see above). Furthermore it offers a limited insight about the quality of confidence scores used and sensitivity of the proposed scheme to this important in practical applications factor (e.g. limited resource languages where WERs of typical external ASR systems range wildly).\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an approach that uses ASR-based scores to guide masking high-confident blocks for speech representation learning. As most of the reviewers mentioned, it is an incremental improvement over baseline systems with limited novelty. About the use of confidence scores which is a key factor of the method, it lacks enough discussion on its quality and sensitivity.
Decision: Reject