Title: SELF-SUPERVISED SPEECH RECOGNITION VIA LOCAL PRIOR MATCHING

Abstract: We propose local prior matching (LPM), a self-supervised objective for speech recognition. The LPM objective leverages a strong language model to provide learning signal given unlabeled speech. Since LPM uses a language model, it can take advantage of vast quantities of both unpaired text and speech. The loss is theoretically well-motivated and simple to implement. More importantly, LPM is effective. Starting from a model trained on 100 hours of labeled speech, with an additional 360 hours of unlabeled data LPM reduces the WER by 26% and 31% relative on a clean and noisy test set, respectively. This bridges the gap by 54% and 73% WER on the two test sets relative to a fully supervised model on the same 360 hours with labels. By augmenting LPM with an additional 500 hours of noisy data, we further improve the WER on the noisy test set by 15% relative. Furthermore, we perform extensive ablative studies to show the importance of various configurations of our self-supervised approach.

1. INTRODUCTION:

Fully supervised learning remains the mainstream paradigm for state-of-the-art automatic speech recognition (ASR). These systems require huge annotated data sets (Li et al., 2017; Chiu et al., 2018; Hannun et al., 2014; Amodei et al., 2016) , which are time-consuming and expensive to collect. This hinders the development of accurate ASR systems for low resource languages (Precoda, 2013) . In fact, out of over 6,000 spoken languages, fewer than 150 are supported by commercial ASR service providers. In sharp contrast to how we teach machines to recognize speech, humans do not learn by listening to thousand hours of speech and simultaneously reading the corresponding transcriptions. Instead, as noted in (Chomsky, 1986; Kuhl, 2004; Glass, 2012; Dupoux, 2018) , humans possess an inherent ability to learn from vast quantities of unlabeled speech. Consider the case of conversing with someone with a strong accent. Even when the speaker pronounces several words in an unusual way, one can often correctly understand the sentence. We argue that the source of indirect supervision in processing unlabeled speech comes from prior knowledge about the world and the context of the speech.

Inspired by this, we devise a self-supervised learning framework termed local prior matching (LPM). We apply this framework to speech recognition allowing an ASR model to learn from unlabeled speech by leveraging a strong language model, which serves as the prior for self-supervision. Given an unlabeled utterance, the ASR model proposes multiple hypotheses and the language model provides a learning signal by evaluating the plausibility of each one.

We evaluate the LPM method on the LibriSpeech corpus (Panayotov et al., 2015) , using 100 hours of labeled speech to seed the proposal model. Using 360 hours of additional labeled data reduces the word error rate (WER) by 3.8% and 13.1% absolute on an easier and a more-challenging test set, respectively. Using the same 360 hours but without labels, LPM reduces the WER by 2.1% and 9.5% absolute on the same two test sets, effectively bridging the gap to fully-supervised learning by 54% and 73%. In addition, by augmenting LPM with another 500 hours, for a total of 860 hours of unlabeled speech, LPM reduces WER on the noisier test set by 14.2% in total. Hence, LPM is able to surpass the performance of using 360 hours of labeled data by taking advantage of about twice the amount of unlabeled data. We also conduct extensive ablation studies and analyses in order to demonstrate the significance of each proposed component. For reproducibility, software will be open source and made publicly available in the camera-ready version.

2. METHOD:

2.1 PRELIMINARIES Let x denote an utterance of speech and y be a transcription. We assume speech is generated following a two-step process:

EQUATION

where the text y is first generated from the language model (LM) p y , and the speech x is then generated from a text-to-speech (TTS) model p x|y conditioned on y. The posterior p y|x is then the ASR model of interest. For convenience, we refer to p xy (x, y) = p y (y) p x|y (x | y) as the joint distribution, and p x (x) = y p xy (x, y) as the marginal distribution. In a typical supervised learning setting, one has access to a labeled dataset D l , which contains paired samples {(x (i) , y (i) )} N i=1 drawn from the joint distribution. An ASR model q y|x can be trained by minimizing the marginal-weighted KL-divergence: E x∼px [D KL (p y|x || q y|x )], which can be estimated with (x,y)∈D l -log q y|x (y | x) + const from samples (i.e. the commonly used cross-entropy loss). In the self-supervised setting we only have access to unlabeled speech dataset D s u , yet we still wish to optimize the same marginal weighted KL-divergence. Since we do not have samples from p y|x=x directly, we can, under mild assumptions, estimate it.

Assumption 1 Different y can lead to the same pronunciation. One reason is the tokenization of words may not be unique. For example y can be the sequence of word-pieces "c at " or "ca t ", both of which result in the same spoken word. Homophones (e.g. "right" and "write") also have the same pronunciation. To formalize this property, let f : Y → Z be a function which maps a text sequence to its pronunciation z ∈ Z (a broader definition of "pronunciation" here, where z = f (y) is a distribution over phoneme sequences for a given text y), and let

p x|y (x | y) = p x|z (x | f (y)).

We let the set Y z = {y : f (y) = z} ⊂ Y contain all text samples with the same pronunciation and hence the same conditional distribution, p x|z=z .

Assumption 2 We assume that two pairs (y, y ) that have different pronunciations (i.e., f (y) = f (y )) either sound similar or are very different. For instance, "lose" and "loose" result in almost the same pronunciation whereas an English speaker would rarely mistake "winter is coming" for "you know nothing". In the former case, the conditional distributions p x|y (x | y) and p x|y (x | y ) may overlap substantially whereas in the latter case they barely intersect. More formally, for each z, we assume the pronunciation space Z can be partitioned by acoustic similarity as Z + z ∪Z - z . For all z + ∈ Z + z , the conditional distributions are similar to that of z, such that D KL (p x|z=z || p x|z=z + ) ≤ for small . On the other hand, we assume the set of dissimilar pronunciations Z - z contributes very little to the marginal probability of non-outlier speech samples. In other words, for any z and x where The following two figures depict the joint and the posterior distributions of four texts, respectively, where it is assumed f (y 1 ) = f (y 2 ) = z a , f (y 3 ) = z b , f (y 4 ) = z c , and p y (y 1 ) > p y (y 3 ) > p y (y 2 ) > p y (y 4 ). The rightmost figure presents local prior, our proposed approximation to the p y|x , computed from the priors of texts whose p x|y are above the threshold δ at a given x.

p x|z (x | z) ≥ δ, we assume z -∈Z - z y -∈Y z -p x|y (x | y -) p y (y -) p x (x).

2.2. LOCAL PRIOR MATCHING:

Equipped with the above assumptions, we next present a tractable approximation of the posterior p y|x using unlabeled speech data. Figure 1 shows a toy example of the proposed approach. Let x be a non-outlier speech sample that is generated conditioned on a text sample y * , whose pronunciation is f (y * ) = z * . The posterior can be derived with Bayes' rule as follows:

EQUATION

The set Y can be partitioned according to pronunciation to arrive at equation 2, which is then approximated using by remark 2 to get equation 3, where 1 is the indicator function. In equation 4,

p x|y (x | ŷ) is simply replaced with p x|z (x | f ( ŷ))

as stated in remark 1. Despite the simplification, estimating equation 4 requires a language model, a TTS model, and the enumeration of text sequences with similar pronunciations to that of z * . To avoid the latter two requirements, we make the following approximations:

1. Replace p x|z (x | ẑ) with p x|z (x | z *

) for all ẑ ∈ Z + z * by exploiting the assumption that these distributions are very similar to each other (i.e., from remark 2 we have

D KL (p x|z=z * || p x|z= ẑ ) ≤ ). 2. Replace ∪ ẑ∈Z +

z * Y ẑ with hypotheses proposed from a beam search using a bootstrapped ASR proposal model. We refer to the proposal model as r y|x and the set of hypotheses for x generated from the beam search as B(r y|x , x, k), where k is the beam size.

The approximated posterior py|x (y | x) can then be written as:

EQUATION

= p y (y) ŷ∈B(r y|x ,x,k) p y ( ŷ)

EQUATION

The approximated posterior only requires computing language model probabilities of the beam search hypotheses, which is tractable. We refer to equation 6 as the local prior, since it is a renormalized distribution in the neighborhood of y * . We also propose local prior matching (LPM) as an unsupervised objective for training an ASR model q y|x :

L lpm (q y|x ; x, p y , r y|x , k) = D KL (p y|x=x ||q y|x=x ) = -H(p y|x=x ) - y∈B(r y|x ,x,k) p y (y) ŷ∈B(r y|x ,x,k) p y ( ŷ) log q y|x (y | x),

which minimizes the KL-divergence between the local prior and the model distribution. For clarity, we term q y|x the online model, and let q y|x (y | x; θ q ) and r y|x (y | x; θ r ) denote the models that are parameterized by and θ q and θ r , respectively. Since the LPM objective simply re-weights hypotheses by the local prior, we do not need a continuous relaxation of y to make p y differentiable with respect to θ q as in Yang et al. (2018); Liu et al. (2019) . The gradient of the proposed LPM objective is given by: ∂L lpm (q y|x ; x, p y , r y|x , k)

∂θ q = y∈B(r y|x ,x,k) p y (y) ŷ∈B(r y|x ,x,k) p y ( ŷ) ∂ log q y|x (y | x) ∂θ q .

We use paired audio and text to estimate r y|x and unpaired speech to compute the LPM objective.

The LPM objective can also take advantage of large unpaired text corpora in order to obtain better language models which directly improve the quality of the approximate posterior.

2.3. PROPOSAL MODEL:

The quality of the posterior approximation py|x depends on the proposal model r y|x . Instead of using a fixed proposal model throughout the entire training process, we update r y|x with q y|x if the latter is better.

On-policy beam search The first approach always uses the online model q y|x as the proposal model. This means θ r = θ q and is effectively a form of on-policy beam search, since the model used to generate hypotheses is also the model we update.

Off-policy beam search While the on-policy method benefits from the immediate improvement of the online model, it also suffers immediately if the gradient update from a mini-batch deteriorates performance. This can result in instability during optimization. We consider a second option which does not tie θ r and θ q but instead updates θ r with θ q every T steps only when the performance of the online model q y|x is better than that of the proposal model r y|x by some metric. We refer to the second option as off-policy beam search. To avoid overfitting to the training set, we use the character error rate (CER) on the validation set as the metric for the proposal model update. We set T = 1000 for all experiments with the off-policy beam search.

2.4. FILTERING HYPOTHESES USING ESTIMATED LENGTHS:

As noted in Chorowski & Jaitly (2017) , sequence-to-sequence ASR models sometimes predict endof-sentence (EOS) tokens too early or generate looping n-grams, resulting in hypotheses that are significantly shorter or longer than the set of acoustically matched texts for a given utterance. Of the two failure modes, the former is more harmful to training with LPM. The reason is that the LPM objective assumes all hypotheses obtained from beam search are acoustically reasonable, and weights each of them by linguistic plausibility given by an LM. While LMs are effective in discriminating plausibility between sentences of similar lengths, we find empirically they tend to assign higher probabilities to shorter sentences than to longer sentences, even when the longer ones are more plausible and grammatically correct than the shorter ones. As a result, truncated hypotheses are assigned higher weights than those that are acoustically matched but longer, which in turn encourages earlier prediction of EOS tokens and forms a catastrophic feedback loop particularly with the on-policy beam search.

To address this issue, we propose a simple filtering heuristic based on the text length. Before training the model, a text length L is estimated for each unlabeled speech sample x. During training, only hypotheses with length close to L are retained for the LPM objective computation. Let len(y) denote the length of y. We keep a hypothesis y only if r lb • L ≤ len(y) ≤ r ub • L , where r lb and r ub are the text length lower and upper bound ratios, respectively. Several methods can be used to estimate the text length on an unlabeled utterance, including using the average speaking rate (Peng et al., 2019) or generating a phoneme/syllable segmentation (Adell & Bonafonte, 2004; Scharenborg et al., 2010; Wang et al., 2017) . In this work, we estimate the length by using using that of the best hypothesis generated from the initial proposal model, generated with either ASR-only greedy decoding or ASR+LM beam search decoding.

3. EXPERIMENTAL SETUP:

Dataset We evaluate our approach on LibriSpeech (Panayotov et al., 2015) , a crowd-sourced audio book corpus derived from the LibriVox Project. The training set contains 960 hours of speech, officially split into three sets: train-clean-100, train-clean-360, and train-other-500, where the first two sets are easier and the third set is noisier and more accented. Similarly, the development and test sets are also split according to difficulty, resulting in four partitions: {dev, test}×{clean, other}, each of which contains roughly five hours of speech. In this work we use train-clean-100 as the paired speech data, and the other two training splits as the unpaired data.

We train the language model on the unpaired text data provided with LibriSpeech, which includes approximately 14,500 books collected from Project Gutenberg. Some of the books in the text corpus overlap with those in the LibriSpeech training set. To avoid training the LM on the ground truth text of the unlabeled speech, we exclude the 997 overlapping books from the text data. We follow the same recipe as Kahn et al. (2019) to pre-process the remaining text.

Neural Network Architecture The proposal model r y|x and the online model q y|x are sequenceto-sequence neural networks (Bahdanau et al., 2016; Chorowski & Jaitly, 2017) with the same timedepth separable (TDS) architecture proposed in Hannun et al. (2019) . The encoder is fully convolutional, composed of TDS blocks which reduce the number of parameters while keeping the receptive field large. The decoder is a single layer recurrent neural network (RNN) with gated recurrent units (GRUs), equipped with a single-headed inner-product key-value attention (Vaswani et al., 2017) for querying information from the encoder outputs. We follow the recipe of Kahn et al. (2019) which uses fewer TDS blocks in the encoder compared to Hannun et al. (2019) in order to generalize better when trained on the smaller LibriSpeech train-clean-100. The output target of the decoder at each step is a posterior distribution over 5,000 word pieces, generated with the SentencePiece toolkit (Kudo & Richardson, 2018) using transcripts from train-clean-100.

To enable efficient evaluation of the language model probabilities, which is required at each training step, we use the gated convolutional language model architecture (ConvLM) proposed in Dauphin et al. ( 2017), which achieves competitive performances compared to recurrent models while significantly reducing the latency. We use the same 5,000 word-piece vocabulary for the LM which is trained with the same model configuration and recipe as Zeghidour et al. (2018) . The trained ConvLM achieves a token perplexity of 34.24 on the development set.

Optimization We use both paired and unpaired data to optimize q y|x . To simplify the optimization procedure, the model is provided with either a paired or an unpaired batch at each step, alternated with a fixed ratio m l : m u . When given a paired batch of n samples, {(x (i) , y (i) )} n i=1 , the model minimizes the standard cross-entropy loss, 1 n i -log q y|x (y (i) | x (i) ). When provided with an unpaired batch {x (i) } n i=1 , the model minimizes a weighted LPM loss, α n • i L lpm (q y|x ; x (i) , p y , r y|x , k). The weight α and the mixing ratio m l /m u are used to balance the supervised and self-training objectives. For regularization we use 20% dropout (Srivastava et al., 2014) , 10% label smoothing, 1% decoder input sampling, and 1% word piece sampling (Kudo, 2018) following Kahn et al. (2019) . We use SGD without momentum to train the online model with an initial learning rate of 5e-2. To achieve a good CER on the development sets, the model is trained for at least 1.6M steps (paired and unpaired ones combined) with a batch size of 16 (8 GPUs × 2 per GPU). The learning rate is annealed by a factor of two every 0.64M steps. All experiments in this paper are implemented in the wav2letter++ framework (Pratap et al., 2018) .

Initialization To initialize the proposal model and the online model, we consider three checkpoints from a baseline model trained on train-clean-100 for a varying number of steps using only the supervised objective. The three checkpoints, whose parameters are denoted as θ A , θ B , and θ C , are trained for about 300k / 40k / 16k steps, achieving average development set CERs of 13% / 20% / 38%, respectively. To obtain a better approximation py|x for the posterior, it is desirable to use the best-performed checkpoint to initialize the proposal model. By contrast, as observed in Kahn et al. (2019) that training from scratch achieves consistently better performances than starting from a well-trained model, we hypothesize that initializing the online model with an earlier checkpoint may also have a similar effect. Hence, we initialize θ r = θ A and θ q = θ C if not otherwise specified.

4. RESULTS:

The best supervised model trained only on train-clean-100 (θ A ) achieves a WER of 14.00%/37.02% on dev-clean/dev-other, respectively. Unless otherwise stated, we use train-clean-360 as the unpaired speech dataset, (r lb , r ub ) = (0.95, 1.05) for length filtering with reference lengths obtained from ASR-only greedy decoding.

4.1. BEAM SIZE, MIXING RATIO, AND LPM WEIGHTS:

Table 1 shows how the WER varies with the beam size and the mixing ratio. for all mixing ratios, the model improves the most from beam size of k = 1 to k = 2, showing the benefit of considering multiple hypotheses. The improvement is greater when a higher mixing ratio of unpaired-to-paired speech is used. In addition, we note that the LM is effectively unused when k = 1 because the LM probability assigned to each hypothesis is normalized within the beam. If there is only one hypothesis, it will be assigned an approximate posterior probability of one. The amount of improvement diminishes with larger beam sizes, and the performance even starts to degrade beyond k = 4 when using a higher mixing ratio. This may result from the inclusion of worse hypotheses which have a better scored under the LM. We use the best setting for the following experiments with a mixing ratio r l : r u = 1 : 4, a beam size k = 4, and an LPM weight α = 0.2. We present detailed results varying the LPM weight in the Appendix.

4.2. PROPOSAL MODEL UPDATE AND MODEL INITIALIZATION:

In addition to the two proposal model update strategies proposed in Section 2.3, termed on-policy and off-policy (better), we experiment with two additional strategies. The first, off-policy (never), uses a fixed proposal model throughout training without ever updating. The second, off-policy (always), updates the proposal model with the online model every T steps (i.e., set θ r ← θ q ) regardless of the performance.

The full results are shown in Table 2 . Four key takeaways are as follows.

(1) For all combinations of (r y|x , q y|x ) initialization, off-policy (never) is the worst. This demonstrates the importance of updating the proposal model to generate better hypotheses during training.

(2) Off-policy (always) consistently outperforms on-policy. We observe that training is significantly stabilized by reducing the proposal model update frequency from every step to every 1,000 steps. The effect is particularly prominent when initializing r y|x and q y|x from an earlier checkpoint (9.62% vs 20.02% on dev-clean, and 27.51% vs 45.62% on dev-other).

(3) Off-policy (better) achieves the best WER in all settings and outperforms off-policy (always) by a larger margin when initializing from an earlier checkpoint. (4) Unlike the other strategies, off-policy (better) demonstrates consistent improvement when using a less-trained initial online model. In the following experiments, we initialize models with θ r = θ A and θ q = θ C unless otherwise specified. 

4.3. LENGTH FILTERING:

Table 3 shows the impact of length filtering with different proposal model update strategies, where both the proposal model and the online model are initialized with θ A . As discussed in Section 2.4, on-policy suffers more than off-policy without length filtering. If the proposal model is never updated, then length filtering does not affect the final WER. We hypothesize that length filtering keeps the proposal model stable during training. Table 4 compares three different reference lengths: the oracle length (Oracle), the predicted length from ASR + LM beam search decoding (ASR + LM Dec), and the predicted length from ASR-only greedy decoding (ASR-only Dec). We observe that the WER does not differ much when using different reference lengths for filtering. We study how the LM affects the performance by experimenting with three alternative hypothesis weighting methods. The first method uses the LM score normalized by token length, p y (y) 1/len(y) , which avoids favoring overly short hypotheses. The second method computes the LM probability of each hypothesis and shuffles them, which preserves the entropy of the original approximate posterior but randomizes the weights. The third assigns an equal weight for each hypothesis. Table 5 presents the results with three beam sizes: k = 2, 4, 8. Using the LM weights directly performs the best for all beam sizes. Length normalization (LenNorm) is better than shuffled and uniform, which have similar performance. This suggests that both ranking hypotheses correctly and scaling the weights proportional to the LM probability are important.

We also observe how the quality of the LM affects the results. We test LMs of various quality by using the same ConvLM but trained for a different number of steps. We quantify LM quality by the token perplexity (PPL) on the development set. Table 6 shows a clear positive correlation between the LM quality and the final WER. This is expected given that the better LM results in a more accurate posterior approximation py|x . The best performing model is trained for 3.2M steps, with a learning rate annealed by a factor of two every 1.28M steps when using 360 hours of unpaired speech, and every 0.64M steps when using 860 hours of unpaired speech. Reference lengths for filtering are obtained from ASR+LM beam search decoding. We compare LPM to fully supervised models and a pseudo labeling (PL) method in Table 7 . Compared to LPM, PL demands extensive hyper-parameter tuning for the ASR+LM beam search as well as heuristic data filtering methods. to obtain high quality pseudo transcripts. We chose PL as a baseline because we follow the same experimental setup, and, more importantly, it achieves the state-of-the-art results on LibriSpeech when using train-clean-100 as paired data and train-other-360 as unpaired speech. A more complete table with results from the literature is in the Appendix.

The upper half of Table 7 shows greedy decoding results without an LM. For the fully supervised model, when removing train-clean-360 the WER increases by 6.86% on test-clean and 13.36% on test-other. Using train-clean-360 speech without transcripts, LPM reduces the absolute WER by 5.64% and 12.21% on the two test sets, which recovers 82% and 91%, respectively, of the WER drop from removing the labels. Adding noisier train-other-500 to the unpaired set (total 860hr D s u ) further reduces the WER, and LPM achieves a better WER on the noisy sets (dev-other and testother) compared to the supervised model trained on 460 hours of clean paired data. In addition, LPM outperforms PL in all settings. This trend is consistent even when decoding with a strong ConvLM. 8 shows WERs on the unlabeled data of an LPM model at the beginning (step=0) and at the end (step=3.2M) of training. Note that this is a proxy of quality for LPM, since multiple hypotheses are used when setting k ≥ 2. We compare to the WER of the pseudo transcripts used in PL. Although generating hypotheses without an LM leads to worse WER than that of PL, toward the end of training, the proposal model of LPM produces better predictions on both train-clean-360 and train-other-500 than the fixed ones used in PL (ASR+LM stable). Furthermore, the WER on train-other-500 much higher for PL (21.51%) than for LPM at the end of training (13.00%), which explains why LPM achieves much better performances than PL when using the full 860hr of unpaired data.

5.2. LINGUISTIC PLAUSIBILITY:

We expect models trained with LPM to generate more semantically and grammatically correct text since the ASR model receives direct supervision from the LM. Table 9 shows the proposed hypotheses for two utterances using a supervised baseline model and a model trained with LPM. The baseline model proposes erroneous hypotheses which are easy to discard even without the audio or the ground truth transcript. On the other hand, LPM generates hypotheses that are both grammatically and semantically plausible, with acceptable substitution errors in some cases (e.g., might/would). We also notice in Table 9 that the LM probabilities correlate well with linguistic plausibility for texts of similar lengths (e.g., hypotheses of the same utterance), but not for texts with large differences in length. Motivated by this observation, we propose to quantify linguistic plausibility of an ASR model by measuring the LM token perplexity exp( y log p y (y)/ y len(y)) of the hypotheses on the development set obtained using ASR-only greedy decoding. Results are shown in Table 10 . The ground truth text has the lowest perplexity on both sets as expected. We also observe that while all models are worse on dev-other than on dev-clean, LPM exhibits the smallest perplexity difference between the two sets, demonstrating how it successfully distills knowledge from the LM. In Drexler & Glass (2018) and Karita et al. (2018) , bi-encoder network architectures are used, which map text and speech to representations in a shared space with their corresponding encoder, and then apply a shared decoder to map from the shared space to the text space. Another line of work adds a TTS model (Tjandra et al., 2017; 2019; Baskar et al., 2019) or a text-to-encoding (TTE) model (Hayashi et al., 2018; Hori et al., 2019) in the loop of ASR training, which can be utilized for back-translation style data augmentation (Sennrich et al., 2016) or cycle-consistency training (Zhu et al., 2017) . Liu et al. ( 2019) treats ASR as a generative model that conditions on speech instead of random noise vectors, and adopts the generative adversarial network (GAN) (Goodfellow et al., 2014) framework in order to improve the fidelity of ASR-generated texts (Liu et al., 2019) . All the aforementioned methods involve additional modules that must be jointly optimized with the ASR model and require finding a careful balance between multiple training objectives. Moreover, some methods require gradient to flow through samples drawn from the ASR outputs, and hence rely on techniques like Gumbel-softmax (Jang et al., 2017) , straight-through estimator (Bengio et al., 2013 ), or REINFORCE (Williams, 1992) . The LPM objective can evaluate output quality yet is fully differentiable.

Knowledge distillation is also easy to used for semi-supervised learning (Cui et al., 2017; Li et al., 2019; Parthasarathi & Strom, 2019 ) because the teacher model can provide a training target for the unlabeled speech. Rather than learning from a teacher ASR model, LPM learns from a teacher LM.

Pseudo labelling (Veselỳ et al., 2017; Manohar et al., 2018; Kahn et al., 2019) is most similar to our approach. Pseudo labelling uses a seed ASR model to transcribe unlabeled speech. The ASR model is then trained on the predicted labels along with the original labeled data used to train the seed model. In addition to being well motivated theoretically, we highlight the key methodological advantages of LPM below. (1) While previous work uses fixed pseudo transcripts throughout training and focuses on data filtering, we demonstrate the benefit of generating hypotheses on-the-fly as the proposal model improves. This is efficient since we only use an end-to-end ASR model for beam search decoding. (2) We motivate and show empirically the benefit from considering multiple hypotheses rather than just the top hypothesis (Veselỳ et al., 2017; Kahn et al., 2019) . (3) Manohar et al. ( 2018) consider multiple hypotheses in a numerator lattice and maximize the aggregated probability of the lattice regardless as to how the probability is allocated amongst the word sequences.

In contrast, we verify in the LM ablation study that considering the allocation of the posterior probability amongst the hypotheses is important.

Aside from semi-supervised learning, this work is also related to unsupervised domain adaptation for ASR, where unlabeled speech of the target domain is provided. Unlike the proposed method, previous studies mainly focus on learning domain invariant features (Sun et al., 2017; Hsu & Glass, 2018; Meng et al., 2017; 2018; 2019) or data augmentation through learned speech transformation (Hsu et al., 2017; 2018) .

7. CONCLUSION AND FUTURE WORK:

We introduce local prior matching, a self-supervised learning objective for speech recognition, and demonstrate note-able reductions in WER when with the addition of unpaired audio and text. We also perform an extensive empirical study to demonstrate the importance of various configurations of LPM. While LPM is motivated by how humans learn to recognize speech, the proposed method can be applied to other sequence transduction tasks including machine translation (Sennrich et al., 2016 ) and text summarization (Nallapati et al., 2016) , provided a good prior for the domain.

We consider two promising directions for future in self-supervised speech recognition with LPM and in general: (1) Empower the model with more capabilities, such as learning to select whether or not to use a sample for self-supervision.

(2) Using more context signals for prior estimation which can consider previous sentences as well as inputs from other modalities.

:



Figures and Tables:

Figure 1: Illustration of the assumed generative process and the proposed posterior approximation. The leftmost figure shows p x|z of three pronunciations: z a , z b and z c , where z a , z b ∈ Z + za and z c ∈ Z - za .The following two figures depict the joint and the posterior distributions of four texts, respectively, where it is assumed f (y 1 ) = f (y 2 ) = z a , f (y 3 ) = z b , f (y 4 ) = z c , and p y (y 1 ) > p y (y 3 ) > p y (y 2 ) > p y (y 4 ). The rightmost figure presents local prior, our proposed approximation to the p y|x , computed from the priors of texts whose p x|y are above the threshold δ at a given x.

Vary mixing ratio and beam size. An LPM weight α = 0.2 is used.

Table content: <table><tr><td>D s u</td><td>r l : ru</td><td>k = 1</td><td colspan="3">dev-clean / dev-other WER (%) k = 2 k = 4 k = 8</td><td>k = 16</td></tr><tr><td/><td>4 : 1</td><td colspan="5">10.75 / 31.62 10.60 / 30.96 10.25 / 30.67 10.14 / 30.17 10.09 / 29.99</td></tr><tr><td>360hr</td><td>1 : 1 1 : 4</td><td>10.43 / 29.76 11.09 / 29.89</td><td>9.56 / 28.83 9.34 / 27.45</td><td>9.37 / 28.10 9.00 / 26.47</td><td>9.06 / 27.35 9.15 / 26.52</td><td>8.88 / 27.25 9.36 / 27.00</td></tr><tr><td/><td>1 : 9</td><td colspan="2">12.11 / 30.89 10.11 / 27.71</td><td>9.76 / 27.08</td><td colspan="2">10.17 / 27.41 10.30 / 27.62</td></tr><tr><td colspan="2">860hr 1 : 4</td><td>10.59 / 26.05</td><td>9.37 / 23.85</td><td>8.68 / 22.53</td><td>8.37 / 21.56</td><td>8.37 / 21.33</td></tr></table>

Vary proposal model update strategies and online model initialization.

Table content: <table><tr><td colspan="2">Init r y|x r y|x update</td><td colspan="3">dev-clean / dev-other WERs (%) Init q y|x = A Init q y|x = B Init q y|x = C</td></tr><tr><td/><td>On-Policy</td><td>9.50 / 28.29</td><td>N/A</td><td>N/A</td></tr><tr><td>A</td><td>Off-Policy (never) Off-Policy (always)</td><td colspan="3">11.19 / 31.74 11.14 / 31.69 11.24 / 31.53 9.40 / 27.79 9.27 / 27.33 9.52 / 27.34</td></tr><tr><td/><td>Off-Policy (better)</td><td>9.20 / 27.42</td><td>9.14 / 26.80</td><td>9.00 / 26.47</td></tr><tr><td/><td>On-Policy</td><td>N/A</td><td>10.17 / 28.35</td><td>N/A</td></tr><tr><td>B</td><td>Off-Policy (never) Off-Policy (always)</td><td colspan="3">13.61 / 35.39 13.95 / 35.43 13.56 / 35.62 9.50 / 27.81 9.56 / 27.58 9.79 / 27.44</td></tr><tr><td/><td>Off-Policy (better)</td><td>9.30 / 27.34</td><td>9.26 / 27.01</td><td>9.15 / 26.63</td></tr><tr><td/><td>On-Policy</td><td>N/A</td><td>N/A</td><td>20.20 / 45.62</td></tr><tr><td>C</td><td>Off-Policy (never) Off-Policy (always)</td><td colspan="3">20.59 / 44.09 22.95 / 46.43 23.42 / 46.89 9.52 / 27.89 9.46 / 27.35 9.62 / 27.51</td></tr><tr><td/><td>Off-Policy (better)</td><td>9.44 / 27.34</td><td>9.31 / 27.26</td><td>9.43 / 27.19</td></tr></table>

Ablation study for length filtering.

Table content: <table><tr><td/><td/><td/><td colspan="2">Table 4: Length filtering using different ref-</td></tr><tr><td/><td/><td/><td>erence lengths.</td></tr><tr><td>r y|x update</td><td colspan="2">dev-clean / dev-other WER (%) No filtering With filtering</td><td colspan="2">Reference Length L dev-{clean / other}</td></tr><tr><td colspan="2">On-Policy Off-Policy (never) Off-Policy (always) 13.99 / 35.52 26.65 / 59.07 11.18 / 31.83 Off-Policy (better) 11.42 / 31.56</td><td>9.50 / 28.29 11.19 / 31.74 9.40 / 27.79 9.20 / 27.42</td><td>Oracle ASR + LM Dec ASR-only Dec</td><td>8.85 / 26.39 8.99 / 26.36 9.00 / 26.47</td></tr><tr><td colspan="3">4.4 CHOICE OF LANGUAGE MODELS</td><td/></tr></table>

Comparison of LPM and alternative hypothesis weighting methods.

Table content: <table><tr><td>py|x</td><td colspan="3">dev-clean / dev-other WER (%) k = 2 k = 4 k = 8</td></tr><tr><td>Proposed</td><td>9.34 / 27.45</td><td>9.00 / 26.47</td><td>9.15 / 26.52</td></tr><tr><td colspan="4">LenNorm 10.52 / 29.10 10.32 / 28.69 10.20 / 28.39</td></tr><tr><td>Shuffled</td><td colspan="3">10.76 / 29.32 10.94 / 29.17 10.56 / 29.20</td></tr><tr><td>Uniform</td><td colspan="3">10.96 / 29.46 10.74 / 29.09 10.84 / 29.21</td></tr></table>

Using p y of different development token perplexity.

Table content: <table><tr><td colspan="2">py PPL dev-{clean / other}</td></tr><tr><td>34.24</td><td>9.00 / 26.47</td></tr><tr><td>64.22</td><td>10.08 / 26.92</td></tr><tr><td>97.87</td><td>10.90 / 27.97</td></tr><tr><td>142.12</td><td>11.53 / 28.74</td></tr><tr><td>180.71</td><td>13.18 / 30.74</td></tr></table>

Results of baselines and the proposed methods.

Table content: <table><tr><td/><td>D l</td><td>D s u</td><td>LM</td><td colspan="2">dev WER (%) clean other</td><td colspan="2">test WER (%) clean other</td></tr><tr><td>Supervised</td><td colspan="2">100hr N/A</td><td>None</td><td colspan="4">14.00 37.02 14.85 39.95</td></tr><tr><td>Supervised</td><td colspan="2">460hr N/A</td><td>None</td><td>7.20</td><td>25.32</td><td>7.99</td><td>26.59</td></tr><tr><td>PL (ASR greedy) (Kahn et al., 2019)</td><td colspan="3">100hr 360hr None</td><td colspan="4">12.27 33.42 12.57 35.36</td></tr><tr><td colspan="4">PL (ASR+LM stable) (Kahn et al., 2019) 100hr 360hr None</td><td>9.30</td><td>28.79</td><td>9.84</td><td>30.15</td></tr><tr><td colspan="4">PL (ASR+LM stable) (Kahn et al., 2019) 100hr 860hr None</td><td>9.03</td><td>26.03</td><td>9.44</td><td>27.25</td></tr><tr><td>Local Prior Matching</td><td colspan="3">100hr 360hr None</td><td>8.85</td><td>26.33</td><td>9.21</td><td>27.74</td></tr><tr><td>Local Prior Matching</td><td colspan="3">100hr 860hr None</td><td>8.08</td><td>21.52</td><td>8.37</td><td>22.89</td></tr><tr><td>Supervised</td><td colspan="2">100hr N/A</td><td>ConvLM</td><td>7.78</td><td>28.15</td><td>8.06</td><td>30.44</td></tr><tr><td>Supervised</td><td colspan="2">460hr N/A</td><td>ConvLM</td><td>3.98</td><td>17.00</td><td>4.23</td><td>17.36</td></tr><tr><td>PL (ASR greedy) (Kahn et al., 2019)</td><td colspan="3">100hr 360hr ConvLM</td><td>6.19</td><td>23.53</td><td>6.81</td><td>24.99</td></tr><tr><td colspan="4">PL (ASR+LM stable) (Kahn et al., 2019) 100hr 360hr ConvLM</td><td>5.73</td><td>22.54</td><td>6.35</td><td>24.13</td></tr><tr><td colspan="4">PL (ASR+LM stable) (Kahn et al., 2019) 100hr 860hr ConvLM</td><td>6.31</td><td>21.87</td><td>6.84</td><td>23.29</td></tr><tr><td>Local Prior Matching</td><td colspan="3">100hr 360hr ConvLM</td><td>5.69</td><td>20.22</td><td>5.99</td><td>20.93</td></tr><tr><td>Local Prior Matching</td><td colspan="3">100hr 860hr ConvLM</td><td>5.39</td><td>14.89</td><td>5.78</td><td>16.27</td></tr><tr><td>5 ANALYSIS</td><td/><td/><td/><td/><td/><td/></tr><tr><td colspan="4">5.1 HYPOTHESIS QUALITY OF UNLABELED TRAINING SPEECH</td><td/><td/><td/></tr><tr><td colspan="8">As discussed in Section 4.2, updating the proposal model is crucial to improve the hypotheses used</td></tr><tr><td colspan="3">during training. To quantify the improvement, Table</td><td/><td/><td/><td/></tr></table>

Comparison of the label quality on the unlabeled speech data between the two methods when trained on 100hr D l and 860hr D s u .

Table content: <table><tr><td>Labelling Method</td><td/><td>Step</td><td colspan="2">train WER (%) clean-360 other-500</td></tr><tr><td>Pseudo Label</td><td>ASR+LM stable</td><td>All</td><td>8.25</td><td>21.51</td></tr><tr><td colspan="2">Local Prior Match Proposal greedy</td><td>0</td><td>14.81</td><td>29.03</td></tr><tr><td colspan="3">Local Prior Match Proposal greedy 3.2M</td><td>7.37</td><td>13.00</td></tr></table>

Comparison of beam search hypotheses from the supervised model (100hr D l ) and the model trained with LPM (100hr D l + 360hr D s u ). Hypotheses are ranked by ASR score q y|x .

Table content: <table><tr><td colspan="3">Model Rank log py</td><td>Beam Search Hypotheses (k = 4)</td></tr><tr><td>Ref.</td><td>-</td><td>-34.14</td><td>she walk ed very fast after she left the house $</td></tr><tr><td/><td>1</td><td>-47.58</td><td>she looked very thought after she left the house $</td></tr><tr><td>Sup.</td><td>2 3</td><td>-38.54 -53.47</td><td>she looked very fat after she left the house $ she what very thought after she left the house $</td></tr><tr><td/><td>4</td><td>-83.85</td><td>she w o u l t very thought after she left the house $</td></tr><tr><td/><td>1</td><td>-32.24</td><td>she walk ed very fast as she left the house $</td></tr><tr><td>LPM</td><td>2 3</td><td>-34.14 -33.10</td><td>she walk ed very fast after she left the house $ she looked very fast as she left the house $</td></tr><tr><td/><td>4</td><td>-36.59</td><td>she looked very fast after she left the house $</td></tr><tr><td>Ref.</td><td>-</td><td>-68.01</td><td>oh if i had imagined him still in such distress sure ly i might have done something to help him $</td></tr><tr><td/><td>1</td><td>-110.11</td><td>i before i had imagined him steal ing such distress sure ly i why have done something to help you $</td></tr><tr><td>Sup.</td><td>2 3</td><td>-107.55 -107.81</td><td>i before i had imagined him steal ing such distress sure ly i want have done something to help you $ i before i had imagined him still ing such distress sure ly i want have done something to help you $</td></tr><tr><td/><td>4</td><td>-107.10</td><td>i before i had imagined him steal ing such distress sure ly i want of done something to help you $</td></tr><tr><td/><td>1</td><td>-72.55</td><td>oh if i had imagined him still in such distress sure ly i would have done something to help you $</td></tr><tr><td>LPM</td><td>2 3</td><td>-71.35 -69.29</td><td>oh if i had imagined him still in such distress sure ly i might have done something to help you $ oh if i had imagined him still in such distress sure ly i would have done something to help him $</td></tr><tr><td/><td>4</td><td>-85.62</td><td>oh if i had imagined him still in such distress sure ly i won't have done something to help you $</td></tr></table>

LM token perplexity of ground truth texts and hypotheses obtained with greedy decoding. A lower perplexity indicates more linguistically plausible decoding results. work builds on a large body of work in semi-supervised learning for ASR. Research in this direction can be classified based on the required modules and objectives used with unpaired data.

Table content: <table><tr><td/><td>D l</td><td>D s u</td><td colspan="2">LM perplexity dev-clean dev-other</td></tr><tr><td>Ground Truth</td><td>N/A</td><td>N/A</td><td>39.94</td><td>43.26</td></tr><tr><td>Supervised</td><td colspan="2">100hr N/A</td><td>96.13</td><td>313.38</td></tr><tr><td>Supervised</td><td colspan="2">460hr N/A</td><td>58.76</td><td>164.77</td></tr><tr><td>Psuedo Label (ASR greedy)</td><td colspan="2">100hr 360hr</td><td>87.36</td><td>273.14</td></tr><tr><td colspan="3">Pseudo Label (ASR+LM stable) 100hr 360hr</td><td>64.07</td><td>170.72</td></tr><tr><td>Local Prior Matching</td><td colspan="2">100hr 360hr</td><td>61.73</td><td>159.72</td></tr><tr><td>Local Prior Matching</td><td colspan="2">100hr 860hr</td><td>59.84</td><td>125.42</td></tr></table>

Peer review (iclr2020):
This work proposed a distillation approach which use ASRs to generate hypotheses for unsupervised data, run a LM to get probability for the hypothesis, and perform distillation with the resulting probability. The ASRs being used for generating hypotheses can be either a model trained with the supervised data or the student model, and can switch between the two during training. In the experiments, ASR models are pre-trained with the subset of Librispeech data and use the rest of Librispeech data as unsupervised data, and the LM is trained with Librispeech LM data. The experiments shown the proposed approach improve baseline model trained with the Librispeech subset significantly. The use of LM to provide soft target is a good idea as LMs can utilize unsupervised text data as opposed to the requirement of training a strong teacher model with paired data, and can be easily integrated with existing distillation approaches for ASRs. The switching to the student model for generating hypotheses when it outperforms the pre-trained ASR also makes a good sense. The overall novelty however is a bit limited compared to the existing work, as the major contribution is to propose to use LMs as teacher rather than ASRs, with the rest of the design to be similar to existing works. The paper relates their method to self-supervised learning, yet I find it having stronger correlation with existing distillation approaches, and can be better understood through the distillation perspective.