Year: 2021

Abstract:
Masked speech modeling (MSM) methods such as wav2vec2 or w2v-BERT learn representations over speech frames which are randomly masked within an utterance. While these methods improve performance of Automatic Speech Recognition (ASR) systems, they have one major limitation. They treat all unsupervised speech samples with equal weight, which hinders learning as not all samples have relevant information to learn meaningful representations. In this work, we address this limitation. We propose ask2mask (ATM), a novel approach to focus on specific samples during MSM pre-training. ATM employs an external ASR model or scorer to weight unsupervised input samples in two different ways: 1) A finegrained data selection is performed by masking over the highly confident input frames as chosen by the scorer. This allows the model to learn meaningful representations. 2) ATM is further extended to focus at utterance-level by weighting the final MSM loss with the utterance-level confidence score. We conduct fine-tuning experiments on two well-benchmarked corpora: LibriSpeech (matching the pretraining data) and Commonvoice, TED-LIUM, AMI and CHiME-6 (not matching the pre-training data). The results substantiate the efficacy of ATM on significantly improving the recognition performance under mismatched conditions (up to 11.6% relative over published results and upto 4.46% relative over our internal baseline) while still yielding modest improvements under matched conditions.

heading: 1 INTRODUCTION
text: Self-training and self-supervised training techniques rely on huge amounts of unlabeled speech or text data for better generalization. The self-training techniques such as pseudo-labeling (Scudder, 1965; Kahn et al., 2020) and student-teacher training (Park et al., 2020) have shown promising improvements by incorporating the data selection process. This data selection step removes pseudolabels with less confidence as denoted by the teacher model before feeding the input to a student model. Xu et al. (2021) shows that self-training and self-supervised training are complementary to each other and also show that self-supervised models act as good initialization for self-training techniques. Self-supervised training (Hinton & Zemel, 1994) is a representation learning approach which implicitly learns patterns in the data without relying on explicit labels. Masked speech modeling (MSM) is the recent and successful self-supervised learning technique, thanks to the advent of BERT (Devlin et al., 2018) in NLP which inspired learning speech representations from masked inputs. MSM techniques such as wav2vec2 (Baevski et al., 2020), HuBERT (Hsu et al., 2021b) and w2v-BERT (Chung et al., 2021) have shown considerable gains across various down-stream speech tasks and have become the go-to models for ASR.
Unfortunately, MSM does not have a data selection scheme to discard the irrelevant input samples and instead imposes burden on the training criterion to learn the relevance of the input samples in learning meaningful representations. Hsu et al. (2021a) noticed the impact of not selecting relevant data from the huge amounts of unsupervised data during pre-training by showing degradation in ASR performance when fine-tuned to a target dataset with limited data. To mitigate this constraint, Chan et al. (2021) introduced substantially more fine-tuning data related to the target dataset but did not achieve satisfactory results. Hsu et al. (2021a) attempted to solve this issue by heuristically selecting the data from a closed set of unsupervised speech databases or by pooling in data relevant to target dataset along with the existing pre-training dataset. However, this data selection approach is not done within the existing pre-training dataset and it is not completely empirically motivated.
In this study, in order to break the above limitation of the MSM techniques, we propose a simple strategy named ask2mask (ATM) to incorporate data selection within a chosen pretraining dataset.
• In ATM, the masking is done over the input samples or speech frames with higher confidence as determined by the scorer. This is contrary to the random selection of frames to be masked in conventional MSM models. We hypothesize that this guided selection of frames to be masked allows the model to focus on the frames which can provide meaningful representations. The scoring model used in this work is necessarily a speech recognition model trained on small amount of data and provides frame-level confidence for each input.
• The ATM technique is further extended to exploit the confidence values provided by the scorer by directly using them to re-weight the MSM loss. We denote this approach as ATM with loss scaling (ATM+S). It allows the model training to focus on certain utterances by down scaling the utterances with low-confidences.
Similar to our work based on masking with external guidance, there is work in NLP that also benefit by incorporating masking with knowledge. In Sun et al. (2019), masking is done at phrase-level segments in BERT and has shown to learn semantic dependencies. In Wang et al. (2019), phonetic knowledge is injected to mask over phonetic segments to perform spectral augmentation.
Our ATM approach is primarily motivated based on the recent work by Veselý et al. (2017) on semi-supervised learning of conventional ASR systems which shows that performing data selection at frame-level or token-level on unsupervised data provides better performance. The importance of pruning out the input samples at frame-level has been studied in Ferreira et al. (2021) to improve both classification and regression tasks. Few works on unsupervised learning also highlight the importance of weighting the data based on its confidence (Wessel et al., 2001; Ren et al., 2020; Coleman et al., 2019). We hypothesize that ATM can leverage the effect of data selection within a particular training corpus to further enhance the recognition performance of MSM techniques.
To summarize, our contributions are listed as follows:
• Novelty: To the extent of our knowledge, ATM is the first approach to incorporate a withincorpus data selection strategy in MSM. We also show that data selection can be simply performed inside MSM by guided selection of frames to be masked using a scorer model.
• Technical contributions: We provide two simple strategies to incorporate data selection into MSM pretraining by applying the confidence of the scorer: 1) choosing the data at framelevel by applying guided masking 2) soft weighting the data at utterance level by scaling the MSM loss of each utterance with its corresponding confidence score. ATM is designed to be compatible to all MSM based pre-training techniques.
• Empirical study: Analysis is done to find an optimal masking percentage for ATM and we highlight the effectiveness of ATM across varying masking percentages. The importance of masking frames with high confidence is substantiated by empirically comparing it with masking low confident frames and random frames respectively. Experiments are performed on AMI data which is from a distinct condition compared to Libri-light corpus used for MSM based pretraining. The results confirm the importance of ATM by improving the recognition performance on evaluation sets of AMI by a significant margin.

heading: 2 PRELIMINARIES ON MASKED SPEECH MODELING (MSM)
text: In this section, we formally define the masked speech modeling (MSM) technique and brief primary instantiations including wav2vec2 and w2v-BERT. The technique can be formulated by defining input speech sequence X = [x1, x2, ..., xT ′ ], where xt is the log Mel-filterbank feature vector at time t. X is sent to the feature encoder Φ to obtain the encoded representations E = Φ(X). The feature encoder contains convolutional layers performing subsampling at a factor of 4 and reducing the total number of frames of an utterance from T ′ to T to get E = [e1, e2, ..., eT ]. E is then sent to two parallel modules: 1) masking component, and 2) quantizer.

heading: 2.1 MASKING
text: The idea behind masking input samples and predicting them was initially proposed in BERT (Devlin et al., 2018) and later adopted to speech (Baevski et al., 2020) with modifications to suit the characteristics of speech input. The masking is done over sets of frames or blocks b1, b2, ..., bK and accommodates overlap between blocks. Here K is the number of masked blocks in a randomly masked encoded sequence Ẽ. The importance of block masking is motivated by the improvements observed in Span-BERT by Joshi et al. (2020) and ERNIE (Sun et al., 2019). The block bk = [ik, c], where ik is the starting index of the masked block and c is the corresponding right context size denoting the number of consecutive speech frames. Here ik are randomly sampled from a uniform distribution. It has been empirically observed by Baevski et al. (2020) that 49% of the frames are masked and c = 10 is chosen as the golden ratio to attain best representation during pre-training.

heading: 2.2 QUANTIZER
text: Gumbel-softmax quantizer component Ψ is used to get quantized representations which act as targets for wav2vec2 and w2v-BERT models. These quantized representations align to phonetic units as described in Baevski et al. (2020). Each quantized vector is of L dimensions which denote the number of targets or codes used in a codebook. Each incoming input E is projected to L dimensions within the quantizer before applying the Gumbel-softmax.

heading: 2.3 CONTEXT NETWORK AND MSM LOSS
text: Wav2vec2-conformer: In this model type, the unmasked sequence E is sent to Ψ to get Q = Ψ(E), where Q = [q1, q2, .., qT ] as described in Baevski et al. (2020). The masked sequence Ẽ is fed to the context network Ω which contains conformer blocks to learn contextual representations from the input. C = Ω(E) denotes the output of the context network. The contrastive loss Lctr(cj , qj) objective is computed between the quantized representation qj and context network output cj ∈ C for all masked time instances j ∈ J . Diversity loss Ldiv is computed as an auxiliary objective in wav2vec2 to force the model to choose diverse codes in the quantization codebook. Detailed description of Ldiv is in Baevski et al. (2020). The final training objective is denoted as:
Lwv = Lctr + 0.1 · Ldiv, (1) where Lctr = ∑J j=1 Lctr(cj , qj).
HuBERT-conformer: This is another variant of wav2vec2-conformer model with two major differences: 1) Targets are k-means cluster ids which are computed over a small portion of input 2) Cross-entropy loss Lce(ŷj , yj) is computed between the prediction of the context network ŷj and the k-means cluster id target yj .
W2V-BERT: This model marries the concept of wav2vec2 and BERT model by including an additional context network Λ containing conformer blocks in addition to Ω as in wav2vec2. The Λ receives the output of the Ω and strives to further learn refined contextual information to get H = Λ(C). The targets of w2v-BERT yj is computed by taking an argmax over the codebook dimensions L of quantized representations qj,l as:
yj = arg max l
qj, l, l ∈ L (2)
Finally, the cross-entropy loss Lce(ŷj , yj) is computed between the prediction ŷj = softmax(hj) and the target yj over the masked time instances J . The final training objective Lwb = Lce + Lwv is a combination of cross-entropy loss and wav2vec2 loss. A block diagrammatic overview of the above MSM architectures are available in appendix A.7.

heading: 3 ASK2MASK (ATM)
text: The primary reason to employ pre-training models is to exploit the abundantly available unsupervised data for improving ASR under limited availability of supervised data. While the MSM models such as wav2vec2 and w2v-BERT described in Section 2 exploit the unsupervised data, they treat
each data with equal weight for computing the final objective.Instead, we generate a score st for each encoded frame et. This is used to select relevant data in a fine-grained manner during masking for computing the loss objective.

heading: 3.1 METHODOLOGY
text: For each encoded feature frame et ∈ E, the scorer emits probabilities p(vt = l |E); l ∈ L of the frame belonging to a particular label. The scorer model is a CTC based frame-synchronous ASR model separately trained with a limited amount of data. Our initial intuition was to chose the scorer’s training data to match the target data condition, however our empirical analysis in (cf. Section 5.3) shows that the performance is agnostic to the scorer model’s training data. Finally, the confidence score st of the frame is defined as the maximum probability across all labels:
st = max l p(vt = l |E) (3)
We sample K masking start indices {i1, .., ik} with probabilities:
p(ik = t) = st∑ sv
v/∈{i1,..,ik−1}
· δt/∈{i1.,,ik−1}, (4)
That is, we sample beginning frames with probability proportional to the scores of each frame. The indicator function δt/∈{i1.,,ik−1} ensures that we sample without replacement. This is the key difference between ATM and the random masking in prior works as described in Section 2.1. Prior works uniformly sample the start indices of each masking block b1:K , while the ATM uses the probability distribution induced by the scorer. K is determined by the percentage of frames to be masked.
We hypothesize that frames with maximum confidence from an external scoring model will be 1) easiest to learn using an MSM training criteria and 2) most informative in for pretraining to facilitate fine-tuning. Conversely, the lowest confidence frames, those more confusable to an external scoring model, will be the least reliably learned by MSM and least informative for pretraining.
The resulting frames are sent as input to the MSM architecture and the final loss objective L is determined by either of the MSM objectives Lwv or Lwb described in Sections 2.3. This modified training objective allows the model to focus on learning from gradients calculated from the frames with high confidences.

heading: 3.2 ATM WITH MSM LOSS SCALING (ATM+S)
text: The ATM loss is computed over the frames with high confidence performing a fine-grained data selection within a uth speech sequence Xu. Utterances with higher average frame confidence as measured by the scorer are accorded higher value than those with more confused frames. To perform data selection at a coarser utterance level, confidence scores su are computed as:
su = 1
T T∑ t=1 st,u (5)
For simplicity, we denote s = su and the MSM loss computed over each masked frame is scaled by s to impose the importance of a particular utterance u. The final training objective L′atm of a particular speech sequence is denoted as:
L ′
atm = s · Latm (6)

heading: 3.3 PROBABILITY AS CONFIDENCE MEASURE IN ATM
text: The ATM uses probability as a simple form of confidence measure to each frame. The analysis of confidence measures for semi-supervision in ASR has been done in Veselý et al. (2017) and they show that posterior probability acts as a reliable confidence measure for frame, word and sentence
based data selection. They also perform an extensive analysis on using the posteriors for hybrid ASR systems. Based on the motivation from this work we chose to use softmax probability directly as our confidence score. A similar observation has been noted in Ferreira et al. (2021), where the usage of softmax probability directly as a confidence measure has been applied to select relevant data samples during training. We also experimented with Entropy and exponential scaling or log scaling on softmax probabilities as confidence measure, but it did not fetch advantage over simple usage of probability.

heading: 4 EXPERIMENTAL SETUP
text: All experiments including pre-training and fine-tuning are performed using 80 dimensional log Melfilterbank features computed over the sampled 16kHz audio. Datasets (such as AMI) contains wideband audio and are downsampled to 16kHz. We evaluate with the test-other (LibriSpeech partition) to show the importance of ATM on matched data conditions, while IHM-eval and SDM-eval (AMI partitions) is used to validate the model under mismatched conditions.

heading: 4.1 DATASETS USED
text: Pretraining (PT): Libri-light (LL-60k) dataset contains 60k hours of unlabeled speech and is used to pre-train all MSM models. LL-60k is the most widely used large unsupervised speech corpus for various PT techniques. Each input speech sequence is constructed by first randomly selecting 32-64 seconds segments from the original utterance. From these segments, a contiguous 32 second region is extracted from a random starting point on-the-fly during MSM PT as described in Zhang et al. (2020b).
Finetuning (FT): Different target datasets including 1) 100 & 960 hours of Librispeech (LS-100 & LS-960) (Panayotov et al., 2015). 2) 100 hours of AMI and 3) speechstew (approx. 5k hours) (Chan et al., 2021) are used to perform our FT experiments. Each dataset used is specific to a certain target data condition, for instance LS-960 is closely matches the LL-60k, AMI dataset is distinct from the LL-60k condition and it contains speech from two kinds of microphones (i) Independent head microphone (IHM). (ii) single distant microphone (SDM). SpeechStew is composed of datasets chosen from multiple conditions to create a mixed domain aggregate corpus. Details of its processing are described in Chan et al. (2021).
Evaluation: We hypothesize that evaluation over AMI using IHM-eval and SDM-eval reveals the effectiveness of ATM in providing informative samples for better representation learning. We also evaluate using evaluation sets from Tedlium and Common voice as their training counter parts are used in SpeechStew based FT. Finally, we also evaluate using CHiME-6 Watanabe et al. (2020) without using any FT data from CHiME-6 training set to compare the performance of ATM on completely unseen target dataset.
Scorer training data: A CTC (Graves et al., 2006) based conformer model with 100M parameters is trained on LS-100 (“LS-scorer”). A similar model is also trained on AMI (“AMI-scorer”). Wordpiece model (WPM) with 1024 tokens are used as labels for training the scorer models. All the results in this paper use “LS-scorer” besides the comparison Section 5.3.

heading: 4.2 MSM ARCHITECTURE
text: Wav2vec2-conformer: This is a wav2vec2 with conformer based context network which first encodes the filterbank features using two 2D convolutional layers with strides (2,2). Model has 100M/600M parameters and is denoted as “w2v2-conformer-L/XL”. HuBERT-conformer-L/XL is similar to w2v2-conformer-L/XL - it differs in using the k-means based quantizer with 1024 targets and computes the cross-entropy loss as described in Hsu et al. (2021b). The “L/XL” size models contains context network Ω 12/24 conformer layers with 8 attention heads and 1024 hidden dimensions.
W2v-BERT: W2v-BERT is explored using two model sizes: one with 100M parameters denoted as “w2v-BERT-L” and containing 2 conformer layers in context net Ω and 4 conformer layers in Λ. A 600M parameter model is denoted as “w2v-BERT-XL” contains 8 conformer layers in Ω and 24 conformer layers in Λ. Each conformer block contains 1024 hidden dimensions with 8 attention
heads, kernel size of 5 with local context of 128. The remaining architecture is identical to the configuration defined in Chung et al. (2021).

heading: 4.3 PT AND FT CONFIGURATION
text: The models L/XL are trained with a global batch size of 512/2048 on 64/256 Google TPU V3 cores for 2-4 days respectively. Adam optimizer is used with a learning rate schedule (Section 5.3 of Vaswani et al. (2017)) with 2e-3 as peak learning rate and 25k warmup steps. The model training configuration follows similar procedure as described in Zhang et al. (2020b).
The FT is done by employing the context network from the PT model by adding a final layer with 1024 WPM units learnt using the RNN-T objective function (Zhang et al., 2020a). The FT is done on w2v-BERT-XL, w2v2-conformer-XL and HuBERT-conformer-XL after 400k PT model updates. The w2v-BERT-L model is FT after 900k PT model updates. w2v-BERT-L is used to initially perform wide range of analysis and hyper-parameter optimization on ATM. w2v-BERT-XL is finally used to compare the results of ATM across existing works in literature. w2v2-conformer-XL and HuBERT-conformer-XL are also used in our experiments. All these models are trained with the same configuration as in Zhang et al. (2020b).

heading: 5 ATM ANALYSIS
text: The empirical study on ATM is done primarily using w2v-BERT-L since this generates the best WER performance across similarly sized models (cf. Figure 2). The pre-trained models are finetuned with either LS-100 or AMI. The resulting finetuned models are evaluated on IHM and SDM evaluation sets to understand the domain generalization aspect of ATM. Librispeech evaluation sets are used in unison to study how ATM behaves under matching domain condition. These experiments are performed with using the loss scaling (it will be discussed in Section 5.5).
5.1 MASKING PERCENTAGES
The number of masked frames within an utterance plays a key role in masked input learning and in this study, we vary the masking percentages from 30% to 50% to determine the best percentage for ATM approach. Previous works on wav2vec2 (Baevski et al. (2020)) showed that masking 49% of the frames is ideal for 30 second utterance and this has been followed subsequent works such as HuBERT and w2vBERT. In case of ATM, this can differ as the frames selected are of higher confidences. Figure 1 shows that ATM achieves its “sweet spot” with 40% masking for both IHM-eval and testother set. Interestingly ATM’s performance is stable across large variations in masking rates with relatively good performance with masking rate as low as 30%. This is a significant difference from the uniform sampling of prior work which suffers significant drop in performance as the masking rate goes below 40%. The result indicates that masking the right set of frames, which ATM aims to do, is able to promote more stable performance. For instance, ATM achieves a %WER of 12.65 with 33% masking and 12.52 with 40% masking on IHMeval respectively as shown in Figure 1. The
recognition performance on test-other and IHM-eval improves over baseline from 8.86% to 8.79% and 13.38% to 12.52% respectively by using ATM.
5.2 ATM MASKING STRATEGIES
The default setting of ATM is chosen based on a hypothesis that those frames that are scored with high-confidence from an external scoring model will be most useful as candidates for MSM pretraining. This hypothesis is interrogated in this section by analysing the impact of choosing the frames with low confidences or equal mix of both high and low confidence frames (Mixed). For masking low confident frames, we modify the score in Equation 3 as:
st = 1−max l p(vt = l |E) (7)
We evaluated these three masking strategies of ATM on both IHM and SDM evaluation sets. Table 1 shows the comparison between these sampling strategies. We observe that masking high confident frames are consistently better than masking the low confidence counterparts. In fact, “Low” confident frames perform worse than the baseline with random masking. Finally, we observe that performance of “Mixed” falls between that of “High” and “Low”. The “Mixed” strategy is similar to random masking, as both high and low confidence frames are selected. This similarity is also reflected in comparable performance between “Mixed” and random masking. These results provide support for our initial hypothesis that masking frames with high confidence leads to better pre-training.

heading: 5.3 HOW TO CHOOSE THE SCORING MODEL
text: The scorer used in this work is a speech recognition model (100M parameters) trained in a supervised fashion. The scorer is chosen based on the target downstream task and in addition to this, the scorer needs to be frame-synchronous to provide confidence for every frame in a speech sequence. In this work, we use a framesynchronous ASR system as the scorer by employing the connectionist temporal classification (CTC) objective. The CTC is preferred over the RNN-T by analysing the reliability of the frame-level predictions. To analyse the importance of the supervised data used to train the scorer, we trained two scorer models: LS-scorer and AMI-scorer are CTC models trained with LS-100 and AMI dataset respectively. The AMI-scorer outperforms on SDM-eval
by improving the %WER from 27.34 to 27.00. Surprisingly, our results on Table 2, shows that the results on IHM-eval using an LS-scorer are comparable to the AMI-scorer. Evaluation on test and test-other shows that LS-scorer is better than AMI-scorer on both sets. Based on these observations, we choose the LS-scorer as the universal scoring model for all ATM based pre-trained models regardless of the target domain (eg: AMI) used in our experiments. Table 2 shows that although matching the scorer to the target domain improves the performance, the difference is not significant.

heading: 5.4 CONSISTENCY ACROSS DIFFERENT ARCHITECTURES
text: Figure 2 shows that ATM consistently outperforms on both IHM-eval and SDM-eval across multiple MSM architectures including wav2vec2 and HuBERT. In the case of IHM-eval, ATM attains a relative improvement of 9% over w2v2-conformer-L, 4% relative improvement over HuBERT-conformer-L and 5% relative gain over w2v-BERT-L baseline models respectively. W2v2conformer-L using ATM obtained 6.2% relative improvement over its baseline counterpart and HuBERT-conformer-L with ATM attained 7.9% rel. improvement over HuBERT-L baseline on SDM-eval respectively. On the other hand w2v-BERT-L baseline is better compared to w2v2conformer-L and HuBERT-conformer-L on both IHM-eval and SDM-eval by achieving 12.52% and 27.34% WER respectively.

heading: 5.5 ATM WITH LOSS SCALING (ATM+S) ANALYSIS
text: ATM training can incorporate utterance-level weighting by scaling the MSM loss obtained using w2v-BERT-L models with the utterance-level confidence score according to Equation 6. We evalu-
ate the value of utterance-level loss scaling by re-weighting utterances in the context of both baseline MSM (i.e., without ATM frame selection) and ATM (ATM+S). These results are in Table 3. Re-weighting utterances by scaling the MSM loss with the confidence score on baseline model is denoted as “Baseline+S” and on ATM is labeled as “ATM+S”. MSM loss scaling is effective even without ATM; baseline+S improves over baseline on both IHM-eval and SDM-eval. Moreover, ATM+S improves over ATM on SDM-eval by attaining 27.19% WER while showing degradation on IHM-eval. This shows that ATM+S is effective on very hard evaluation task such as SDM-eval compared to IHM-eval. On the IHM-eval test set, the impact of MSM loss scaling is observed over the Baseline MSM without ATM. We hypothesize that ATM+S may not able to provide improvement on IHM-eval as ATM already incorporates optimally incorporates scorer information on this task.

heading: 6 RESULTS
text: In this Section, XL models are used to compare the importance of ATM on LS-960, AMI and SpeechStew. These three datasets show the effect of ATM on diverse conditions with a much larger model. Results are compared with appropriate prior work. Table 4 shows that ATM and ATM+S improves over dev-other while on dev set there was improvement only using ATM and not ATM+S. Although the ATM and ATM+S does not show improvement on test and test-other, matches the very strong baseline. Considering the similarity between LS-960 and PT data, ATM manages to provide gains without hurting the performance across all Librispeech evaluation sets. This validates our argument that MSM models are better without any data selection when trained under matched data condition but can benefit under mismatched conditions.
Table 5 presents the results of ATM on AMI by comparing it with w2v2-conformer-XL baseline and w2v-BERT-XL baselines. We include w2v2-conformer-XL to further test the consistency of ATM on XL models when evaluated on harder tasks. ATM+S and ATM observes consistent gains over baseline on both IHM-eval and SDM-eval when trained with XL models. However, ATM+S did not
demonstrate improvement on IHM-eval using w2v-BERT-XL. Table 6 analyses the effect of ATM
and ATM+S on multiple evaluation sets such as Commonvoice, Tedlium, AMI and CHiME-6. These four sets are chosen based on the mismatch range from minimum to maximum and for instance, Commonvoice has the minimum mismatch with Libri-light data, while CHiME-6 has the maximum mismatch. The state-of-the-art results published in Chan et al. (2021) are obtained by choosing the best Conformer model supervisedly trained with multiple datasets such as AMI, CommonVoice, Broadcast News, Librispeech, Switchboard/Fischer, TED-LIUM and Wall Street Journal. Note that the training data did not include the CHiME-6 data. The authors in Chan et al. (2021) show that simply training an ASR with lots of data leads to best results compared to the wav2vec2 finetuned model. Their best results are denoted in table 6 and will be used to compare with our best ATM results.
Our baseline w2v-BERT-XL attained better results over the published w2v2-conformer-XL and Speechstew results. In Commonvoice and CHiME-6, the baseline attained 7.4% and 2.9% relative improvement over Speechstew respectively. However, by including our ATM and ATM+S with w2v-BERT-XL, there was consistent improvement across all range of mismatched domains. For instance, ATM+S attains 5.76% relative improvement on CHiME-6 over the Speechstew. This result clearly justifies that selection of reasonable input samples during pre-training reduces the necessity of having finetuning data from the same domain to improve performance. To further substantiate this, the results on AMI show a 4.6% relative improvement on AMI-SDM over Speechstew which is of different domain compared to pre-training domain. In case of minimal mismatch domain such as
Commonvoice, the ATM attained 11.6% relative improvement over Speechstew. These observations show that ATM and ATM+S demonstrate their effectiveness to generalize to unseen and challenging speech recognition conditions.

heading: 7 CONCLUSION
text: In this work, we introduce ask2mask (ATM) to perform data selection over unsupervised samples for MSM based pre-training to focus on relevant information and learn meaningful representations. ATM achieves 21.0% WER on mismatched AMI SDM set with guided masking and a 20.7% WER is obtained by including loss scaling (ATM+S). We empirically show that ATM is more robust to changes in masking percentage compared to random masking. as typically used in MSM. Our results substantiate the importance of learning from high confident frames by attaining improvements across multiple evaluation sets. An important aspect of ATM approach is its flexibility to incorporate into any MSM pretraining techniques and ATM+S can also be easily adopted into self-supervised pre-training methods. In our future work, we wish to apply ATM over pretraining data containing data from multiple domains (Hsu et al., 2021a; Likhomanenko et al., 2020) to achieve further improvements. We also consider two future enhancements to ATM: (1) Joint training of the scorer model with MSM model by simultaneous training on supervised and unsupervised data. (2) Perform active learning by sharing the parameters of MSM with the scorer once the MSM is well trained.

heading: ETHICS STATEMENT
text: While we are unaware of any specific bias in Ask2Mask, it is possible that bias within the SpeechStew and LibriLight training corpora themselves may introduce bias to the resulting ASR model. Ask2Mask uses an ASR model trained on labeled data as a scoring model identify regions for masking; any bias in this scoring model (trained on LibriSpeech-100 or AMI) could also impact the fine-tuning process, leading to bias in the final model. Without being aware of specific biasing effects, it is unclear how, if at all, biases in the scoring model and downstream ASR model may interact with each other.
Automatic Speech Recognition (ASR) can exacerbate security and privacy issues by facilitating the search and analysis of speech, though in and of itself ASR does not pose a security nor privacy issue.

heading: REPRODUCIBILITY STATEMENT
text: All of the corpora used for 1) training the scorer model, and 2) pretraining and finetune the ASR models, as well as 3) all evaluations are publicly available and use standard published partitions. We aim to describe Ask2Mask and Ask2Mask with scaling comprehensively in the text to facilitate reproducibility. Implementation of major components, Conformer RNN-T models, wav2vec 2.0 and CTC training are available within in public, open-source libraries including Nvidia-nemo (link) and ESPNet (link). This supports the reproducibility of Ask2Mask and the results reported in this paper within multiple frameworks. Additional details to promote reproducibility are provided in the Appendix.

heading: A APPENDIX

heading: A.1 DETERMINE THE BEST PT AND FT CHECKPOINT (CKPT.)
text: Selecting the best PT ckpt. to perform the fine-tuning is cumbersome as it involves finding the best PT ckpt. based on the validation accuracy on quantized targets and also run fine-tuning experiments on multiple PT ckpts. Figure 3 shows that ATM starts to show improved performance after 250k PT ckpt. On contrary to our belief that ATM might require less FT, both baseline and ATM requires same number of FT iterations to achieve the best performance.

heading: A.2 ANALYSIS OF SCORER BASED ON %WER ON PT DATA (1 HOUR OF LIBRI-LIGHT )
text: We further analyse the impact of the recognition performance of the scorer on ATM, by choosing three categories of scorers such as bad, better and best. The 10 hours of supervised libri-light (LL10h) data is used to evaluate the performance of these scorers and the results are as follows: Initial model checkpoint (ckpt.). LS-scorer attains 51.8% WER and AMI-scorer achieves 72.7% at this
ckpt respectively. Intermediate model ckpt. Evaluating LS-scorer at this ckpt., achieves 45.8% and AMI-scorer achieves 57.1% WER. Final model ckpt. chosen after convergence. The LS-scorer gains 35.5% WER and AMI-scorer attains 46.8% by evaluating at this ckpt. Figure 4 shows a more
challenging scenario by testing the ability of ATM using scorers from different domains with varying range of performances. A LS-scorer with 51.8%WER on LL-1hr performs well on both IHM-eval and SDM-eval, while the AMI-scorer with 46.8% WER slightly improves over the above mentioned LS-scorer. The key aspect in selecting better scorer is to choose ckpt. either from the category which achieves %WER within the range of 50-60 on the small subset of pre-training data.

heading: A.3 ANALYSIS OF ATM ON LIBRISPEECH
text: Our ATM analysis is also done on w2v-BERT-L to validate the ATM on LS-100 and is present in table 7. ATM attained 4.3% and rel. imp. on test-other over the baseline model. Apart from that there is a slight change in test set which can be treated as noise. On both dev-other and test we did not find any improvement and instead there was slight degradation in performance.
To further evaluate the impact of increasing the model parameters from “L” size to “XL” size, we FT on LS-100 using MSM models with XL size and the results are in table 8. We did not find any consistency in the performance across the evaluation sets using any of the MSM architectures. Slight gains are observed on test or dev or dev-other using w2v2-conformer-XL. Once the baseline in w2vBERT-XL gets better, ATM did not achieve gains on test-other. This scenario can be explained due to effectiveness of MSM pre-training under matched condition and can perform well without any ncessary data selection approach.

heading: A.4 ATM+S ANALYSIS ON VALIDATION DATA DURING PT
text: The effect of ATM and ATM+S is analysed by plotting the validation scores on dev-other during pre-training. The first plot in figure 5 shows that the contrastive loss improves over the baseline with the aid of ATM and is further enhanced with ATM+S. The second plot shows the number of unique
codes used from the quantizer codebook. Analysing this plot helps us to understand if the validation loss or accuracy is improved by just using less % of unique codes which will affect the performance at FT. Among the 1024 codes, 94%-95% are used by both ATM and ATM+S. This is similar to the % unique codes used by the baseline model and confirms that improvement of ATM and ATM+S is not by choosing smaller set of unique codes. The third plot shows that the MSM accuracy of ATM and ATM+S improves over baseline model. ATM+S shows that re-weighting each utterance is complementary to the ATM.

heading: A.5 COMPARISON BETWEEN FRAME-LEVEL AND UTTERANCE-LEVEL LOSS SCALING
text: ATM+S performs MSM loss scaling using the utterance-level confidences which performs focus on each utterance at a coarse level. We also experimented with scaling with frame-level confidence scores. Our experiments showed that scaling all utterances with frame-level confidence hurts the model performance. To solve this issue, we randomly selected utterances which participate in frame scaling. Scaling 10% utterances resulted in better performance and the results are shown in Table 9.

heading: A.6 RESULTS ON COMMONVOICE WITH PUNCTUATION NORMALIZATION
text: Some previous work, including Likhomanenko et al. (2020), report Commonvoice results by scoring with a normalization process that removes punctuation. For comparison, we do the same in Table 11.

heading: A.7 BLOCK DIAGRAMMATIC VIEW OF MSM ARCHITECTURES

heading: A.8 STATISTICAL SIGNIFICANCE ANALYSIS ON %WER PERFORMANCE FOR MULTIPLE EVALUATION SETS
text: The table 4 results are on Librispeech and obtaining 0.1% improvement in Librispeech testsets is statistically signficant. For instance, the dev-clean test set contains 54402 words and 0.1% gains denotes a recovery of 54 words. Also, the recent works on self-supervised training such as HuBERT shows improvement between wav2vec2-Large and HuBERT-Large only on dev-clean with 0.1% gain in Table 3.

Review summaries:
Many self-supervised speech representation learning methods use masked prediction at their core. This paper proposes Ask2Mask (ATM) approach for informed masking during learning through a supervised teacher model that provides frame-level posteriors probabilities of linguistic output units. The frame-posterior probabilities extracted from the supervised teacher model are also used as utterance weights to favor ones with high confidence.

This study investigates the use of an external frame-synchronous CTC-based ASR system to get confidence scores for frames in masked speech modeling which is called AskToMask (ATM). These confidence scores are used in two ways:  1) as a distribution to determine the segment to mask (higher confidence means higher probability),  2) the utterance level average of the per-frame confidence scores is used as a scaling factor of the per-utterance loss.   The experiments perform unsupervised pre-training using these two proposed methods and then performs supervised fine-tuning for the final ASR task. Experiments on Librispeech and AMI show that confidence score-based selection of masking improves the final WER as compared to using uniform distribution in the mask selection (the first approach versus baseline). The second approach provides marginal gains on top of the first approach. 

This paper describes a weighted masking scheme for learning speech representations. Weights in this scheme are surrogates of confidence scores obtained from an external ASR system. The paper also describes how utterance-level confidence scores can be incorporate to down-weight contribution from utterance that are likely to be problematic for an ASR system.   The paper makes use of Librispeech dataset for learning representations and Librispeech and AMI datasets for training external ASR systems. The paper evaluates both schemes on Librispeech, AMI, and some other datasets. Experimental results show that the proposed schemes offer small-to-medium gains over the unweighted reference masking scheme.