Self-Supervised Speech Recognition via Local Prior Matching

1 INTRODUCTION . Fully supervised learning remains the mainstream paradigm for state-of-the-art automatic speech recognition ( ASR ) . These systems require huge annotated data sets ( Li et al. , 2017 ; Chiu et al. , 2018 ; Hannun et al. , 2014 ; Amodei et al. , 2016 ) , which are time-consuming and expensive to collect . This hinders the development of accurate ASR systems for low resource languages ( Precoda , 2013 ) . In fact , out of over 6,000 spoken languages , fewer than 150 are supported by commercial ASR service providers . In sharp contrast to how we teach machines to recognize speech , humans do not learn by listening to thousand hours of speech and simultaneously reading the corresponding transcriptions . Instead , as noted in ( Chomsky , 1986 ; Kuhl , 2004 ; Glass , 2012 ; Dupoux , 2018 ) , humans possess an inherent ability to learn from vast quantities of unlabeled speech . Consider the case of conversing with someone with a strong accent . Even when the speaker pronounces several words in an unusual way , one can often correctly understand the sentence . We argue that the source of indirect supervision in processing unlabeled speech comes from prior knowledge about the world and the context of the speech . Inspired by this , we devise a self-supervised learning framework termed local prior matching ( LPM ) . We apply this framework to speech recognition allowing an ASR model to learn from unlabeled speech by leveraging a strong language model , which serves as the prior for self-supervision . Given an unlabeled utterance , the ASR model proposes multiple hypotheses and the language model provides a learning signal by evaluating the plausibility of each one . We evaluate the LPM method on the LibriSpeech corpus ( Panayotov et al. , 2015 ) , using 100 hours of labeled speech to seed the proposal model . Using 360 hours of additional labeled data reduces the word error rate ( WER ) by 3.8 % and 13.1 % absolute on an easier and a more-challenging test set , respectively . Using the same 360 hours but without labels , LPM reduces the WER by 2.1 % and 9.5 % absolute on the same two test sets , effectively bridging the gap to fully-supervised learning by 54 % and 73 % . In addition , by augmenting LPM with another 500 hours , for a total of 860 hours of unlabeled speech , LPM reduces WER on the noisier test set by 14.2 % in total . Hence , LPM is able to surpass the performance of using 360 hours of labeled data by taking advantage of about twice the amount of unlabeled data . We also conduct extensive ablation studies and analyses in order to demonstrate the significance of each proposed component . For reproducibility , software will be open source and made publicly available in the camera-ready version . 2 METHOD . 2.1 PRELIMINARIES . Let x denote an utterance of speech and y be a transcription . We assume speech is generated following a two-step process : y ∼ py x ∼ px|y , ( 1 ) where the text y is first generated from the language model ( LM ) py , and the speech x is then generated from a text-to-speech ( TTS ) model px|y conditioned on y . The posterior py|x is then the ASR model of interest . For convenience , we refer to pxy ( x , y ) = py ( y ) px|y ( x | y ) as the joint distribution , and px ( x ) = ∑ y pxy ( x , y ) as the marginal distribution . In a typical supervised learning setting , one has access to a labeled dataset Dl , which contains paired samples { ( x ( i ) , y ( i ) ) } Ni=1 drawn from the joint distribution . An ASR model qy|x can be trained by minimizing the marginal-weighted KL-divergence : Ex∼px [ DKL ( py|x || qy|x ) ] , which can be estimated with∑ ( x , y ) ∈Dl − log qy|x ( y | x ) + const from samples ( i.e . the commonly used cross-entropy loss ) . In the self-supervised setting we only have access to unlabeled speech dataset Dsu , yet we still wish to optimize the same marginal weighted KL-divergence . Since we do not have samples from py|x=x directly , we can , under mild assumptions , estimate it . Assumption 1 Different y can lead to the same pronunciation . One reason is the tokenization of words may not be unique . For example y can be the sequence of word-pieces “ c at ” or “ ca t ” , both of which result in the same spoken word . Homophones ( e.g . “ right ” and “ write ” ) also have the same pronunciation . To formalize this property , let f : Y → Z be a function which maps a text sequence to its pronunciation z ∈ Z ( a broader definition of “ pronunciation ” here , where z = f ( y ) is a distribution over phoneme sequences for a given text y ) , and let px|y ( x | y ) = px|z ( x | f ( y ) ) . We let the set Yz = { y : f ( y ) = z } ⊂ Y contain all text samples with the same pronunciation and hence the same conditional distribution , px|z=z . Assumption 2 We assume that two pairs ( y , y′ ) that have different pronunciations ( i.e. , f ( y ) 6= f ( y′ ) ) either sound similar or are very different . For instance , “ lose ” and “ loose ” result in almost the same pronunciation whereas an English speaker would rarely mistake “ winter is coming ” for “ you know nothing ” . In the former case , the conditional distributions px|y ( x | y ) and px|y ( x | y′ ) may overlap substantially whereas in the latter case they barely intersect . More formally , for each z , we assume the pronunciation space Z can be partitioned by acoustic similarity asZ+z ∪Z−z . For all z+ ∈ Z+z , the conditional distributions are similar to that of z , such that DKL ( px|z=z || px|z=z+ ) ≤ for small . On the other hand , we assume the set of dissimilar pronunciations Z−z contributes very little to the marginal probability of non-outlier speech samples . In other words , for any z and x where px|z ( x | z ) ≥ δ , we assume ∑ z−∈Z−z ∑ y−∈Yz− px|y ( x | y− ) py ( y− ) px ( x ) . 2.2 LOCAL PRIOR MATCHING . Equipped with the above assumptions , we next present a tractable approximation of the posterior py|x using unlabeled speech data . Figure 1 shows a toy example of the proposed approach . Let x be a non-outlier speech sample that is generated conditioned on a text sample y∗ , whose pronunciation is f ( y∗ ) = z∗ . The posterior can be derived with Bayes ’ rule as follows : py|x ( y | x ) = py ( y ) px|y ( x | y ) ∑ ŷ∈Y py ( ŷ ) px|y ( x | ŷ ) = py ( y ) px|y ( x | y ) ∑ ẑ∈Z+ z∗∪Z − z∗ ∑ ŷ∈Yẑ py ( ŷ ) px|y ( x | ŷ ) ( 2 ) ≈ py ( y ) px|y ( x | y ) ∑ ẑ∈Z+ z∗ ∑ ŷ∈Yẑ py ( ŷ ) px|y ( x | ŷ ) · 1∪ ẑ∈Z+ z∗ Yẑ ( y ) ( 3 ) = py ( y ) px|z ( x | f ( y ) ) ∑ ẑ∈Z+ z∗ ∑ ŷ∈Yẑ py ( ŷ ) px|z ( x | ẑ ) · 1∪ ẑ∈Z+ z∗ Yẑ ( y ) . ( 4 ) The set Y can be partitioned according to pronunciation to arrive at equation 2 , which is then approximated using by remark 2 to get equation 3 , where 1 is the indicator function . In equation 4 , px|y ( x | ŷ ) is simply replaced with px|z ( x | f ( ŷ ) ) as stated in remark 1 . Despite the simplification , estimating equation 4 requires a language model , a TTS model , and the enumeration of text sequences with similar pronunciations to that of z∗ . To avoid the latter two requirements , we make the following approximations : 1 . Replace px|z ( x | ẑ ) with px|z ( x | z∗ ) for all ẑ ∈ Z+z∗ by exploiting the assumption that these distributions are very similar to each other ( i.e. , from remark 2 we have DKL ( px|z=z∗ || px|z=ẑ ) ≤ ) . 2 . Replace ∪ẑ∈Z+ z∗ Yẑ with hypotheses proposed from a beam search using a bootstrapped ASR proposal model . We refer to the proposal model as ry|x and the set of hypotheses for x generated from the beam search as B ( ry|x , x , k ) , where k is the beam size . The approximated posterior p̄y|x ( y | x ) can then be written as : p̄y|x ( y | x ) = py ( y ) px|z ( x | z∗ ) ∑ ŷ∈B ( ry|x , x , k ) py ( ŷ ) px|z ( x | z ∗ ) · 1B ( ry|x , x , k ) ( y ) ( 5 ) = py ( y ) ∑ ŷ∈B ( ry|x , x , k ) py ( ŷ ) · 1B ( ry|x , x , k ) ( y ) . ( 6 ) The approximated posterior only requires computing language model probabilities of the beam search hypotheses , which is tractable . We refer to equation 6 as the local prior , since it is a renormalized distribution in the neighborhood of y∗ . We also propose local prior matching ( LPM ) as an unsupervised objective for training an ASR model qy|x : Llpm ( qy|x ; x , py , ry|x , k ) = DKL ( p̄y|x=x||qy|x=x ) = −H ( p̄y|x=x ) − ∑ y∈B ( ry|x , x , k ) py ( y ) ∑ ŷ∈B ( ry|x , x , k ) py ( ŷ ) log qy|x ( y | x ) , which minimizes the KL-divergence between the local prior and the model distribution . For clarity , we term qy|x the online model , and let qy|x ( y | x ; θq ) and ry|x ( y | x ; θr ) denote the models that are parameterized by and θq and θr , respectively . Since the LPM objective simply re-weights hypotheses by the local prior , we do not need a continuous relaxation of y to make py differentiable with respect to θq as in Yang et al . ( 2018 ) ; Liu et al . ( 2019 ) . The gradient of the proposed LPM objective is given by : ∂Llpm ( qy|x ; x , py , ry|x , k ) ∂θq = ∑ y∈B ( ry|x , x , k ) py ( y ) ∑ ŷ∈B ( ry|x , x , k ) py ( ŷ ) ∂ log qy|x ( y | x ) ∂θq . We use paired audio and text to estimate ry|x and unpaired speech to compute the LPM objective . The LPM objective can also take advantage of large unpaired text corpora in order to obtain better language models which directly improve the quality of the approximate posterior .

Summary: This work proposed a distillation approach which use ASRs to generate hypotheses for unsupervised data, run a LM to get probability for the hypothesis, and perform distillation with the resulting probability. The ASRs being used for generating hypotheses can be either a model trained with the supervised data or the student model, and can switch between the two during training. In the experiments, ASR models are pre-trained with the subset of Librispeech data and use the rest of Librispeech data as unsupervised data, and the LM is trained with Librispeech LM data. The experiments shown the proposed approach improve baseline model trained with the Librispeech subset significantly.