Audio Lottery: Speech Recognition Made Ultra-Lightweight, Noise-Robust, and Transferable

1 INTRODUCTION . End-to-end automatic speech recognition ( ASR ) ( Wang et al. , 2019 ; Hannun et al. , 2014 ; Graves , 2012 ; Chorowski et al. , 2015 ; Dong et al. , 2018a ) has become an indispensable technology in consumer-interactive devices ( e.g. , smartphones , smart speakers , tablets ) over the past few years ( He et al. , 2019 ; Cohen , 2008 ; Schalkwyk et al. , 2010 ) . Conventional on-device ASR system usually requires the involvement of servers , i.e. , streaming the audio to servers and then streaming the results back to the devices . By contrast , recent studies ( He et al. , 2019 ; McGraw et al. , 2016 ; Kim et al. , 2019 ; Sim et al. , 2019 ; Park et al. , 2018 ; Waibel et al. , 2003 ; Arık et al. , 2017 ) have spurred the success of ASR models fully run on devices , which can be advantageous in terms of computational resources , latency , and user data privacy . Developing on-device ASR models are challenging since the computational resources ( e.g. , CPU , memory , battery ) are typically very limited . A standard design approach to fit ASR model in budget is through applying various neural network compression techniques to the larger ASR models , such as network pruning ( Takeda et al. , 2017 ; Shangguan et al. , 2019 ; Gao et al. , 2020 ) , knowledge distillation ( Li et al. , 2018a ) , and parameter quantization ( He et al. , 2019 ; Sainath et al. , 2020 ) . However , there always exists a trade-off between computational efficiency and the model performance , and the efficiency improvements are usually at the cost of word error rate ( WER ) . In most prior works , we observed a non-negligible degradation of WER in compressed models . A new horizon emerges with the discovery of lottery tickets hypothesis ( LTH ) ( Frankle & Carbin , 2019b ) . LTH empirically demonstrated the existence of highly sparse matching subnetworks ( i.e. , winning tickets ) in full dense networks , that can be independently trained from scratch to match or even surpass the performance of the latter . LTH was widely observed in various models in computer vision ( Liu et al. , 2019b ; Evci et al. , 2019 ; Frankle et al. , 2020c ; Savarese et al. , 2020 ; Wang et al. , 2020a ; You et al. , 2020 ; Ma et al. , 2021 ; Girish et al. , 2020 ) and natural language processing ( Gale et al. , 2019 ; Yu et al. , 2020 ; Prasanna et al. , 2020 ; Chen et al. , 2020b ; c ) . Yet to our best knowledge , it has not been studied nor utilized in the realm of speech processing and recognition . This paper presents the first investigation on LTH for developing on-device ASR models . Despite the rich literature of LTH in vision and language , a practically useful winning ticket for real-world ASR would demand two unique properties : transferability , noise robustness . • As one persistent challenges of ASR , each individual has a different voice and speaking style . Unlike text or images , whose data are much more “ standardized ” , the spoken word varies greatly based on regional dialects , speed , emphasis , even social class and gender . Therefore , scaling up any ASR system has always been a significant obstacle , since the testing utterances may have very different distribution from the training utterances . That has made transferability a crucial demand for ASR in the open world . • In an ideal world , one would have to speak very clearly , slowly , and in an environment with no background noise , for the sounds being recognized : that unfortunately will not happen in the practice . In the real-world ASR applications , noise robustness is becoming another crucially demanded technological factor since ASR is now expected to work in much more difficult acoustic environments than in the past ( Li et al. , 2014 ) . For example , the recognition of speech recorded by distant microphones is challenged by acoustic interference such as noise , reverberation and interference speakers ( Kinoshita et al. , 2020 ; Haeb-Umbach et al. , 2019 ; Kinoshita et al. , 2013 ) . Even in the standard ASR benchmark such as LibriSpeech ( Panayotov et al. , 2015 ) , there are significant background noise in its “ clean ” subset ( Zen et al. , 2019 ) . More importantly , prior LTH studies mostly use unstructured sparsity during model pruning . However , designing chips that speeds up unstructured sparse networks are much more complex than those for structured sparsity ( e.g. , Block-Sparse GPU Kernels ( Gray et al. , 2017 ) ) . Together with the two unique gaps , they make ASR no less challenging than visual recognition or text understanding , if not more . They account for the prior arts ’ difficulty to maintain a satisfactory balance between model efficiency and recognition performance ; and similarly , they question the applicability of LTH in ASR . We provide an affirmative , positive answer of LTH in ASR . As the subject of study , we choose most commonly used ASR architectures in both research and products over the past few years : 1 ) CNN+RNN with connectionist temporal classification ( CTC ) ( Graves et al. , 2006 ) ; 2 ) RNNTransducer ( Graves , 2012 ) ; and 3 ) Convolution-augmented Transformer ( Gulati et al. , 2020 ) . We conducted extensive LTH experiments using these backbones on three popular corpora : TEDLIUM ( Rousseau et al. , 2012 ) , Common Voice ( Ardila et al. , 2020 ) , and LibriSpeech ( Panayotov et al. , 2015 ) . Unlike most of existing LTH studies that only approached to explaining and demonstrating the correctness of LTH theory , in this work , we make the first attempt to apply LTH to real-world use cases . Namely , we investigate three unique properties that were rarely studied in previous LTH research but are key to user-interactive ASR devices , bringing new insights to both LTH theory and lightweight ASR research . Our main contributions are outlined below : • We for the first time reveal the existence of winning tickets in the context of ASR by answering three research questions regarding LTH theory . The most lightweight winning tickets from CTC , RNN-Transducer , and Conformer backbones only possess 21.0 % , 10.7 % , and 8.6 % remaining nonzero weights , respectively . We also show that winning tickets significantly outperform other state-of-the-art network pruning and knowledge distillation methods . • We are the first to explore the use of structured sparsity ( i.e. , block sparsity ( Narang et al. , 2017b ) ) in LTH , and successfully found highly sparse winning tickets ( e.g. , 4.4 % remaining weights ) that have no performance degradation compared to using unstructured sparsity . • Winning tickets have exceptional transferability across different datasets , which are notably better than full models . Also , the winning tickets identified from large source datasets transfer better . • In the presence of various levels of background noise , the winning tickets consistently achieve significantly better WERs than full models . That indicates stronger noise robustness might be an extra bonus of inducing sparsity , and a missing gem by previous LTH works . 2 RELATED WORKS . End-to-End Automatic Speech Recognition . Previous mainstream ASR systems are mostly based on hidden Markov model ( HMM ) -Gaussian mixture model ( GMM ) or HMM-Deep neural network ( DNN ) . These systems can achieve descent performance . However , such system is usually composed of modules ( e.g. , acoustic model , language model , lexicon ) that are needed to be trained separately , which makes it hard to be optimize globally in products ( Wang et al. , 2019 ) . End-to-end ASR can directly transcribe an input audio sequence to a token ( e.g. , grapheme or phoneme ) sequence . Current end-to-end ASR frameworks can be generally categorized into three types : CTC-based ( Hannun et al. , 2014 ; Amodei et al. , 2016 ; Graves & Jaitly , 2014 ; Miao et al. , 2015 ; Eyben et al. , 2009 ) , RNN-transducer ( Graves , 2012 ; Graves et al. , 2013 ; Rao et al. , 2017 ; Dong et al. , 2018b ) , sequence-to-sequence ( Seq2Seq ) model ( Chorowski et al. , 2015 ; Bahdanau et al. , 2016 ; Chan et al. , 2016 ; Zhang et al. , 2017 ; Chiu et al. , 2018 ; Prabhavalkar et al. , 2018 ) , and Transformer model ( Gulati et al. , 2020 ; Dong et al. , 2018a ; Wang et al. , 2020b ; Baevski et al. , 2020 ) . CTC ( Graves et al. , 2006 ) is essentially a loss function , which maximizes the probability of all the paths that correspond to the ground-truth token sequence , with an augmented blank token “ - ” indicating no output . It avoids the need of segmental alignment/labels in training utterances , which makes tons of speech materials usable without additional annotating effort and thus fully exploits the modeling capacity of DNNs . RNN-transducer is composed of an encoder for the input audio sequence , a prediction network to model the interdependencies in between the output token sequence , and a joint network to align the input and output sequence and produce the prediction . Seq2Seq model usually has an encoder-decoder structure , where the encoder maps the input audio sequence to a hidden representation sequence and the decoder autogressively decodes the output token sequence . An attention mechanism ( Chorowski et al. , 2015 ) is trained to learn the alignment between the input and output sequences . Transformer models are similar to Seq2seq models but use multi-head attention ( MHA ) ( Vaswani et al. , 2017 ) layers for encoders and decoders , which has been proven to achieve the state-of-the-art ASR performance . Lottery Tickets Hypothesis . The recently emerged LTH ( Frankle & Carbin , 2019b ) deviates from the common wisdom of after-training pruning ( Han et al. , 2015 ) , and demonstrates the existence of highly sparse subnetworks that are independently trainable from scratch , called winning tickets . Once trained , they are capable of matching or even surpassing the performance of their full models . Follow-up efforts ( Renda et al. , 2020 ; Frankle et al. , 2020b ) introduce the effective weight rewinding techniques to scale up LTH to large networks on large-scale datasets . LTH draws wide attention from various deep learning fields , and has been studied in image classification ( Liu et al. , 2019b ; Evci et al. , 2019 ; Frankle et al. , 2020c ; Savarese et al. , 2020 ; Wang et al. , 2020a ; You et al. , 2020 ; Ma et al. , 2021 ; Chen et al. , 2020a ) , natural language processing ( Gale et al. , 2019 ; Yu et al. , 2020 ; Prasanna et al. , 2020 ; Chen et al. , 2020b ; c ) , object detection ( Girish et al. , 2020 ) , generative adversarial networks ( Chen et al. , 2021d ; Kalibhat et al. , 2020 ; Chen et al. , 2021a ) , graph neural networks ( Chen et al. , 2021b ) , reinforcement learning ( Yu et al. , 2020 ) , and life-long learning ( Chen et al. , 2021c ) . Several pioneer works ( Mehta , 2019 ; Morcos et al. , 2019 ; Desai et al. , 2019 ; Chen et al. , 2020b ; a ) also investigate LTH transferability across datasets and downstream tasks . Yet to the best of our knowledge , LTH in speech models remains untouched – and that would not be a trivial extension for three aspects of reasons . On the task level , ASR has unique demands for individual user transferrability and noise robustness , which has been explained previously . On the model level , compared to CV models , speech models are mostly based on RNN backbones ( Hannun et al. , 2014 ; Chan et al. , 2016 ) , which contain recursive computational graphs and are notoriously unstable to train . That makes the pruning of RNN-based models challenging too ( Zhang & Stadie , 2019 ) , and off-the-shelf methods developed for pruning CNNs are often found ineffective or even inferior to random pruning , when applied to RNNs . Pruning methods customized for RNNs typically call for special sparse structures or stability regularizations ( Narang et al. , 2017a ; b ; Kliegl et al. , 2017 ; Wen et al. , 2018 ; Zhang & Stadie , 2019 ) , and it is hence unclear whether IMP would remain stable and effective for RNN-based models in ASR . On the data level , compared to NLP models , the sequence lengths of speech signals are usually significantly larger than word embeddings ( e.g. , the spectrogram sequence length of a 10-second speech utterance extracted with 10ms shift is 1,000 ; the number of words in a sentence is usually between 10 and 50 ) , which also inevitably costs higher computational complexity . Therefore , finding sparse subnetworks that can maintain the full model performance for ASR models is practically meaningful yet highly non-trivial .

Summary: This paper investigates using the lottery tickets hypothesis (LTH) strategy for pruning neural network weights for speech recognition. The method first explains the general LTH framework and extensions with transfer learning scenarios. The paper shows the effectiveness of the proposed method in the standard ASR tasks, pre-training with other models, and transfer learning, especially in noisy speech recognition tasks.  Some comments: - In the introduction "End-to-end automatic speech recognition (ASR) (Wang et al., 2019)": (Wang et al., 2019) is not a representative paper of end-to-end ASR - In the introduction, "For example, the recognition of speech recorded by distant microphones is challenged by acoustic interference such as noise, reverberation and interference speakers (Kinoshita et al., 2020)": Again not sure (Kinoshita et al., 2020) is a correct citation. I think the following papers are more appropriate   - Haeb-Umbach, Reinhold, et al. "Speech processing for digital home assistants: Combining signal processing with deep-learning techniques." IEEE Signal processing magazine 36.6 (2019): 111-124.   - Kinoshita, Keisuke, et al. "The REVERB challenge: A common evaluation framework for dereverberation and recognition of reverberant speech." 2013 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics. IEEE, 2013. - Currently, data augmentation techniques (artificially contaminate the clean speech with various noisy data) are more standard than transfer learning for noisy speech recognition. Please discuss it. - Section 2 "On the model level, compared to CV models, speech models are mostly based on RNN backbones": I'm confused about this description. As you mentioned in the previous paragraphs, many speech models now use transformer, conformer, and CNN. - Section 2 "with 10ms shift is 1,000": this is not exactly true as we usually use downsampling at the beginning of the encoder layer. - Section 3, RQ3: Frankly, I don't think this is an essential question for me. This sounds very trivial that the use of pre-trained models would improve the performance. - Section 4, What is the main message of Figure 1? I don't find the particular effectiveness of the proposed method from this figure/example. - Section 4, Table 3: Why are the other methods, especially KD-based methods, so worse? - Section 5 "A common way to address this issue is through speaker adaptation (Zhao, 1994;": Again, I don't think (Zhao, 1994) is a representative paper for speaker adaptation. MAP/MLLR adaptation techniques exist in this era, and they were standard techniques in the GMM era. - Section 5: I recommend the authors use the real noisy speech data like the CHiME data