Audio Lottery: Speech Recognition Made Ultra-Lightweight, Noise-Robust, and Transferable
Authors: Shaojin Ding
Tianlong Chen
Zhangyang Wang
Venue: ICLR 2022 Poster (2022)
Keywords: Speech Recognition
Lottery Ticket Hypothesis

Abstract: Lightweight speech recognition models have seen explosive demands owing to a growing amount of speech-interactive features on mobile devices. Since designing such systems from scratch is non-trivial, practitioners typically choose to compress large (pre-trained) speech models. Recently, lottery ticket hypothesis reveals the existence of highly sparse subnetworks that can be trained in isolation without sacrificing the performance of the full models. In this paper, we investigate the tantalizing possibility of using lottery ticket hypothesis to discover lightweight speech recognition models, that are (1) robust to various noise existing in speech; (2) transferable to fit the open-world personalization; and 3) compatible with structured sparsity. We conducted extensive experiments on  CNN-LSTM, RNN-Transducer, and Transformer models, and verified the existence of highly sparse winning tickets that can match the full model performance across those backbones. We obtained winning tickets that have less than 20% of full model weights on all backbones, while the most lightweight one only keeps 4.4% weights. Those winning tickets generalize to structured sparsity with no performance loss, and transfer exceptionally from large source datasets to various target datasets. Perhaps most surprisingly, when the training utterances have high background noises, the winning tickets even substantially outperform the full models, showing the extra bonus of noise robustness by inducing sparsity. Codes are available at https://github.com/VITA-Group/Audio-Lottery.

Published as a conference paper at ICLR 2022
AUDIO LOTTERY: ULTRA-LIGHTWEIGHT, TRANSFERABLE, AND NOISE- ROBUST
SPEECH RECOGNITION MADE
Shaojin Ding"", Tianlong Chen"", Zhangyang Wang*
Teeas A&M University, University of Texas Austin
sh shja@tanu. edu, Itianlong. chen, at Laswang Butexas edu
ABSTRACT
Lightweight speech recognition models have seen explosive demands owing growing amount of speech- features on mobile devices. Since designing such systems from scratch is non-trivial, practitioners typically choose to compress large (pre-trained) speech models. Recently, lottery ticket hypothesis reveals the existence of highly sparse subnetworks that can be trained in isolation without sacrificing the performance of the full models. In this paper, we investigate the tantalizing possibility of using lottery ticket hypothesis speech recognition models, that are (1) robust to various noise existing in speech; (2) transferable to fit the open-world personalization; and 3) compatible with structured sparsity. We conducted extensive experiments CNN-LSTM, RNN- Transducer, and Transformer models, and verified the existence of highly sparse "winming tickets" that can match the full model performance across those backbones. We obtained winning tickets that have less than 20% of full model weights on all backbones, while the most lightweight one only keeps 4.4% weights. Those winning tickets generalize to structured sparsity with no performance loss, and transfer exceptionally from large source datasets to various target datasets. Perhaps most surprisingly, when the training utterances have high background noises, the winning tickets even substantially outperform the full models, showing the extra bonus of noise robustness by inducing sparsity. Codes are available at bttps://gitiin wwwm
discover lightweight
/t///VVIAAAGGOO CErTAA-Grou/// caoooup///iiico -otttez......
INTRODUCTION
End-to-end automatic speech recognition (ASR) (Wang et al., 2019; Hannun al., 2014; Graves, 2012; Chorowski et al., 2015; Dong et al., 2018a) has become an indispensable technology in consumer-interactive devices (e.g., smartphones, smart speakers, tablets) over the past few years (He al., 2019; Cohen, 2008; Schalkwyk et al., 2010). Conventional on-device ASR systems usually require the involvement of servers, i.e., streaming the audio servers and then streaming the results back to the devices. By contrast, recent studies (He et al., 2019; McGraw et al., 2016; Kim et al., 2019; Sim et al., 2019; Park et al., 2018; Waibel et al., 2003; Arık et al., 2017) have spurred the success of ASR models fully run on devices, which can be advantageous in terms of computational resources, latency, and user data privacy.
Developing on-device ASR models is challenging since the computational resources (e.g., CPU, memory, battery) are typically very limited. A standard design approach to fit ASR model under budget is through applying various neural network compression techniques to the larger ASR models, such as network pruning (Takeda et al., 2017; Shangguan et al., 2019; Gao et al., 2020), knowledge distillation (Li et al., 2018a), and parameter quantization (He et al., 2019; Sainath et al., 2020). However, there always exists a trade-off between computational efficiency and the model performance, and the efficiency improvements are usually the cost of word error rate (WER). In most prior works, we observed a neg-nggli degradation of WER in compressed models.
'Equl Contribution. Shaojin Ding is now with Google. This work was finished when he was with Texas A&M University.

---

Published as a conference paper at ICLR 2022
A new horizon emerges with the discovery of lottery tickets hypothesis (LTH) (Frankle & Carbin, 2019b). LTH empirically demonstrated the existence of highly sparse matching subnetworks (i.e., winning tickets) in full dense networks, that can be independently trained from scratch to match or even surpass the performance of the latter. LTT was widely observed in various models in computer vision (Liu et al., 2019b; Evei et al., 2019; Frankle et al., 2020c; Savarese et al., 2020; Wang et al., 2020a; You et al., 2020; Ma et al., 2021; Girish et al., 2020) and natural language processing (Gale al., 2019; Yu et al., 2020; Prasanna al., 2020; Chen al., 2020b;c). Yet our best knowledge, has not been studied nor utilized in the realm of speech processing and recognition
This paper presents the first investigation on LTH for developing on-device ASR models. Despite the rich literature of LTH in vision and language, a practically useful winning ticket for real-world ASR would demand two unique properties: transferability, noise robustness.
As one persistent challenges of ASR, each individual has different voice and speaking style. Unlike text or images, whose data are much more "standardized", the spoken word varies greatly based on regional dialects, speed, emphasis, even social class and gender. Therefore, scaling up any ASR system has always been significant obstacle, since the testing utterances may have very different distribution from the training utterances. That has made transferability a crucial demand for ASR in the open world.
In an ideal world, one would have to speak very clearly, slowly, and in an environment with no background noise, for the sounds being recognized: that unfortunately will not happen in the practice. In the real-world ASR applications, noise robustness is becoming another crucially demanded technological factor since ASR is now expected to work in much more difficult acoustic environments than in the past (Li et al., 2014). For example, the recognition of speech recorded by distant microphones is challenged by acoustic interference such noise, reverberation and interference speakers (Kinoshita et al., 2020; Eueb-Umbbuce et al., 2019; Kinoshita et al., 2013). Even in the standard ASR benchmark such as LibriSpeech (Panayotov et al., 2015), there significant background noise in its "clean" subset (Zen et al., 2019).
More importantly, prior LTH studies mostly use unstructured sparsity during model pruning. However, designing chips that speeds up unstructured sparse networks are much more complex than those for structured sparsity (e.g., Block-Sparse GPU Kernels (Gray et al., 2017)). Together with the two unique gaps, they make ASR no less challenging than visual recognition or text understanding, not more. They account for the prior arts' difficulty to maintain a satisfactory balance between model efficiency and recognition performance; and similarly, they question the applicability of LTH in ASR.
We provide an affirmative, positive answer of LTH in ASR. As the subject of study, we choose most commonly used ASR architectures in both research and products over the past few years: 1) CNN-LSTM with connectionist temporal classification (CTC) (Graves et al., 2006); 2) RNN- Transducer (Graves, 2012); and 3) Convolution-augmmente Transformer (Gulati et al., 2020; Burchi & Viellzuff 2021). We conducted extensive LTH experiments using these backbones on three popular corpora: TED-LIUM (Rousseau et al., 2012), Common Voice (Ardila et al., 2020), and LibriSpeech (Panayotov et al., 2015). Unlike most of existing LTH studies that only approached to explaining and demonstrating the correctness of LTH theory, in this work, we make the first attempt to apply LTH real-world use cases. Namely, we investigate three unique properties that were rarely studied in previous LTH research but are key to user-interactive ASR devices, bringing new insights to both LTH theory and lightweight ASR research. Our main contributions are outlined below:
We for the first time reveal the existence of winning tickets in the context of ASR by answering three research questions regarding LTH theory. The most lightweight winning tickets from CNN-LSTM, RNN-TTansd,uer, and Conformer backbones only possess 21.0%, 10.7%, and 8.6% remaining nonzero weights, respectively. We also show that winning tickets significantly outperform other state-of-the-ar network pruning and knowledge distillation methods.
We are the first to explore the use of structured sparsity (i.e., block sparsity (Narang et al., 2017b) in LTH, and successfully found highly sparse winning tickets (e.g.. 4.4% remaining weights) that have no performance degradation compared to using unstructured sparsity.
Winning tickets have exceptional transferability across different datasets, which are notably better than full models. Also, the winning tickets identified from large source datasets transfer better. In the presence of various levels of background noise, the winning tickets consistently achieve significantly better WERs than full models. That indicates stronger noise robustness might be an extra bonus of inducing sparsity, and a missing gem by previous LTH works.
2

---

Published as a conference paper at ICLR 2022
2 RELATED WORKS
End-to-End Automatic Speech Recognition. Previous mainstream ASR systems mostly on hidden Markov model (HMM)-Gausian mixture model (GMM) or HMM-Deep neural network (DNN). These systems can achieve descent performance. However, such system is usually composed of modules (e.g., acoustic model, language model, lexicon) that are needed to be trained separately, which makes it hard be optimize globally in products (Wang et al., 2019).
based
End-to-end ASR can directly transcribe an input audio sequence to a token (e.g., grapheme or phoneme) sequence. Current end-to-end ASR frameworks can be generally categorized into three types: CTC-based (Hannun et al., 2014; Amodei et al., 2016; Graves & Jaitly, 2014; Miao et al., 2015; Eyben et al., 2009), RNN-transdueer (Graves, 2012; Graves al., 2013; Rao et al., 2017; Dong al, 2018b), sequence-le weeeenccett-seeueeee sequencee (Seq2Seq) model (Chorowski et al., 2015; Bahdanau et al., 2016; Chan et al., 2016; Zhang et al., 2017; Chiu al., 2018; Prabhavalkar et al., 2018), and Transformer model (Gulati et al., 2020; Dong et al., 2018 1a; Wang et al., 2020b; Baevski et al., 2020).
CTC (Graves et al., 2006) essentially loss function, which maximizes the probability of all the paths that correspond to the ground-truth token sequence, with an augmented blank token "." indicating no output. It avoids the need of segmental alignment/labels in training utterances, which makes tons of speech materials usable without additional annotating effort and thus fully exploits the modeling capacity of DNNs. RNN-transducer is composed of an encoder for the input audio sequence, prediction network to model the interdependencies in between the output token sequence, and a joint network to align the input and output sequence and produce the prediction. Seq2Seq model usually has encoder-decoder structure, where the encoder maps the input audio sequence to a hidden representation sequence and the decoder autogressively decodes the output token sequence. An attention mechanism (Chorowski et al., 2015) is trained to learn the alignment between the input and output sequences. Transformer models are similar to Seq2seq models but use multi-head attention (MHA) (Vaswani et al., 2017) layers for encoders and decoders, which has been proven achieve the ofte-offthheart ASR performance.
Lottery Tickets Hypothesis. The recently emerged LTH (Frankle & Carbin, 2019b) deviates from the common wisdom of after-training pruning (Han et al., 2015), and demonstrates the existence of highly sparse subnetworks that are independently trainable from scratch, called winning tickets. Once trained, they are capable of matching or even surpassing the performance of their full models. Follow-up efforts (Renda et al., 2020; Frankle et al., 2020b) introduce the effective weight rewinding techniques to scale up LTH to large networks on large-scale datasets, LTH draws wide attention from various deep learning fields, and has been studied in image classification (Liu et al., 2019b; Evei et al., 2019; Frankle et al., 2020c; Savarese et al., 2020; Wang et al., 2020a; You et al., 2020; Ma et al., 2021; Chen et al., 202200212 ld; 2022a;b), natural language processing (Gale et al., 2019; Yu et al., 2020; Prasanna al., 2020; Chen et al., 2020b;e), object detection (Girish et al., 2020), generative adversarial networks (Chen et al., 2021e; Kalibhat et al., 2020; Chen et al., 2021a), graph neural networks (Chen et al., 2021b), reinforcement learning (Yu et al., 2020), and life-long learning (Chen et al., 2021c). Several pioneer works (Mehta, 2019; Morcos et al., 2019; Desai et al., 2019; Chen et al., 2020b;a) also investigate LTH transferability across datasets and downstream tasks.
Yet to the best of our knowledge, LTH in speech models remains untouched and that would not be a trivial extension for three aspects of reasons. On the task level, ASR has unique demands for individual user transferrability and noise robustness, which has been explained previously. On the model level, compared to CV models, speech models are mostly based on RNN backbones (Hanoun et al., 2014; Chan et al., 2016), which contain recursive computational graphs and are notoriously unstable to train. That makes the pruning of RNN-based models challenging too (Zhang & Stadie, 2019), and off-the-shelf methods developed for pruning CNNs are often found ineffective or even inferior to random pruming, when applied to RNNs Pruning methods customized for RNNs typically call for special sparse structures or stability regularizations (Narang et al., 20;b;; Kliegl et al., 2017; Wen et al., 2018; Zhang & Stadie, 2019), and it is hence unclear whether IMP would remain stable and effective for RNN-based models in ASR. On the data level, compared to NLP models, the sequence lengths of speech signals are usually significantly larger than word embeddings (e.g.. the spectrogram sequence length of a 10-second speech utterance extracted with 10ms shift is 1,000; the number of words in a sentence is usually between 10 and 50), which also inevitably costs higher computational complexity. Therefore, finding sparse subnetworks that can maintain the full model performance for ASR models is practically meaningful yet highly non-trivial.
3

---

Published as a conference paper at ICLR 2022
3 PRELIMINARIES AND SETUPS
Backbone Network. We investigate three ASR backbone networks that are widely used in both academia and productions: 1) CNN-LSTM (Amodei et al., 2016) model with CTC loss (Graves et al., 2006); 2) RNN-Transducer (Graves, 2012); and 3) Convolution-augmennte Transformer with CTC (Conformer) (Gulati et al., 2020). Please see Appendix A.1 for details of the backbones.
Datasets, Training, and Evaluations. We conducted experiments on three commonly used ASR corpora: TED-LIUM (Rousseau et al., 2012) (118 hours). Common Voice (Ardila et al., 2020) (582 hours), and LibriSpeech (Panayotov et al., 2015) (960 hours). Note that LibriSpeech has two test sets: test-clean little noise inutterances; test-other considerable noise in utterances. We test the LibriSpeech model on the two test sets hereinafter. During training, we set the batch size to 32 and an initial learning rate 0.0003, which is annealed down by factor of 1.1 the end of each epoch. All the models were trained for 16 epochs. To evaluate the performance, we consider two measurements:
Word Error Rate. WER is the standard metric measuring the accuracy of ASR models. WER is computed as: WER - (S + I + D)/N, where S. I. D, and N denote the number of substitutions, insertions, deletions, and the total number of words, respectively.
Number of Purameters. The number of parameters measures the complexity of a model. In our case, since all subnetworks were pruned from the full models, we use the percentage of Remaining Weights as an alternative measurement. We define Sparsity as Sparsity(%) = 100% - Remmanine Weights (%).
Subnetworks. For dense model fa; its subnetworks can be derived as f(r; O 0) with binary pruning mask m E {0, 13 where @ is the element-wise product and d is the dimension of prumeable model parameters. We use (f 8)) to represent the training algorithm (e.g.. Adam (Kingma & Ba, 2017) with grid searched hyperparameters) that trains a network J (x; 0) on dataset (e.g., LibriSpeech) for iterations. Let Og be the random initialized network weights.
Subnetwork Evaluation. To measure the generalization ability of obtained subnetworks, we define EP (AP (f(;; 0))) as the evaluation function of model f returned from AP on the dataset D. Then, we further introduce:
Matching subnetworks. Following the definition in (Frankle et al., 2020a; Chen et al., 2020b;a), a subnetwork f(x; @ 0) is matching if it satisfies the following condition that indicates matching subnetworks achieve no worse performance than its dense counterpart under the same training algorithm AP and evaluation metric & ED (AP (f (x; m o 0))) 2 ED (AP (z;)
Winning ticket. far; m 0) is a winning ticket for AP, if is (i) matching subnetwork and (ii) 0 = Oo for AP.
Transferable Winning ticket. subnetwork f(x; m @ @0) is transferable target datasets {D.) N
if and only it is a winning ticket for each AP: The subnetwork f(x; m @ 0) is derived from the source dataset D. & (D.)
Pruning Method for Subnetwork Searching. Iterative weight magnitude pruning (IMP) is the widely used algorithm in previous LTH literature (Frankle & Carbin, 2019a; Frankle et al., 2020a; Chen et al., 2020b). To identify subnetworks f(z; m @ 0). IMP performs following three steps: (1) training a unpruned dense network to completion on a dataset 'D (i.e., applying AP): )P) (2) eliminating a portion of insignificant weights with the globally smallest magnitudes (Han et al., 2015; Renda et al., 2020) so that the model only has 8,% of weights remaining (i.e., the sparsity); (3) rewinding model weights 0 (0- = 6o. the original random initialization; or 0 = 0pre. the weights from a pre-trained model) and finetuning the subnetwork to converge by leveraging AP Note that steps (2) and (3) usually needs to be iteratively repeated for several rounds for finding highly competitive winning tickets. In all experiments, we set s, % = (1 0.8*) x 100%, where is the number of iterations.
4 THE EXISTENCE OF WINNING TICKETS IN SPEECH RECOGNITION
In this section, we explore the existence of winning tickets in the three ASR backbones. Namely, we would like to answer the following research questions from an empirical perspective:
ROI: Can we find winning tickets f(x; MMMP @ 8) for speech recognition model using IMP? How much do they improve model complexity?
4

---

Published as a conference paper at ICLR 2022
Table 1: Performance of three backhones the extreme sparsity at the best performance LibriSpeech test-clean subset. The performance on fest-other subset has a similar trend (see Appendix A.3). # * number of parameters in full model, in which we use Mega (x (x10%) as the unit; WER WER of full models; WER WER of the winning tickets at extreme sparsity; bER WER of the best performing winning tickets. Remaining Weight (RW) is included as model complexity measurement.
Backbone
#Paramsuul
WER
WER
RW oxt (#Params)
WER
RWbess (#Params)
CNN-LSTM
86.62M
8.02
7.98
21.0% (18.19M)
7.13
51.2% (44.34M)
RNN-Transdwer
132.23M
5,90
5.71
10.7% (14.14M)
5.39
41.0% (54.2IM)
Conformer
65.84M
2.55
2.49
16.8% (11.06M)
2.26
51.2% (33.7IM)
Table 2: Performance CNN-LSTM backbone (86.62M parameters) at the extreme sparsity or at the best performance on TED-LIUM, Common Voice, and LibriSpecch datasets.
Dataset
WER
WFBRea
R8Wox oxt (*Params)
WER
RW (#Params)
TED-LIUM
15.93
15.70
4,4% (3.81M)
14.04
16.8% (14,55M)
Common
5.57
5.41
16.8% (14.55MM)
4.17
64.0% (55,43M)
LibriSpeech (test-clean)
8.02
7,98
21.0%
(18. (18.19M)
7.13
51.2% (44,34M)
LibriSpeech (test-other)
20.59
20.53
21.0% (18.19M)
19.21
51.2% (44.34M)
RQ2: Do winning tickets identified by IMP have less complexities or better performance, compared to random pruning/random tickets and other compression methods?
RQ3: Instead of using randomized weights Og as the initialization of IMP, does it improve the performance of winning tickets if we use weights pre from a pre- pre-trained model?
Does winning tickets exist in speech recognition models? To answer the questions, we conducted experiments three backbones and three datasets. For each trial, we first run IMP to extract a binary pruning mask at each sparsity. Then, we generate one subnetwork at each sparsity by applying the corresponding mask to the model and reset the weights the original random initialization Og- Finally, we train each subnetwork and computes their WER on the test set to determine if they are winning tickets. All the training hyperparameters training a subnetwork are the same as those in training the full model.
RQI:
As shown in Table 1, winning tickets can be identified on all three backbones. The most lightweight winning tickets on CNN-LSTM, RNN- -ranTuan,, and Conformer have 21.0%, 10.7%, and 16.8% remaining weights, respectively. In addition, we noticed that the RNN -rannrrnseeeer subnetworks at the extreme sparsity has less percentage of remaining weights than CNN-LSTM and Conformer subnetworks, likely due to this model being more werrparaaeterizze (RNN-Transdun Transducer: 132.23M; CNN-LSTM: 86.62M parameters; Conformer: 65.84M). These results show that, for fixed dataset, the winning tickets extracted from larger models are sparser.
Similarly, winning tickets can also be identified on all three datasets, as shown in Table 2. We also found that the sparsity of a winning ticket is correlated to the size the dataset. For example, TED- LIUM has relatively small size (118 hours) compared to Common Voice (582 hours) and LibriSpeech (960 hours). Accordingly, the remaining weights of TED-LIUM winning ticket are significantly lower than that of Common Voice and LibriSpeech winning tickets. A possible explanation is models become relatively more overparameterized for smaller training sets, which allows them to be more amenable for sparsification (Li et al., 2020). Similar observations can be found in (Chen et al., 2020b).
From Table and 2, another interesting finding is that the subnetworks with low sparsity (remain most of the weights) always achieve preferable performance than the full model. We also provided visualization of the outputs from the full model, the most sparse subnetwork, and the best performing subnetwork in Figure 1 (see Appendix A.4 for more examples.). Similarly, we observed larger performance improvement on smaller datasets, possibly also due to the over parameterization issue. Results indicate that LTH is also a potential research direction in improving overall ASR performance.
RQ2: Does IMP winning tickets have lower complexity or better performance than random pruning/tickets and other compression methods? As suggested in previous studies (Frankle & Carbin, 2019a; Chen et al., 2020b; 2021e; 2020a), the two key aspects for a winning ticket to achieve the desired performance are: 1) initial weight 0, and 2) mask generated from IMP MIMP. In this subsection, we test if this argument hold in ASR winning tickets. To achieve this, we compare the winning tickets against two baseline pruning approaches: random pruning and random tickets. The subnetworks identified with random pruning are initialized with 6g but the masks are randomly generated MRAND. By contrast, the subnetworks identified with random tickets are randomly
5

---

Published as a conference paper at ICLR 2022
Waveform
ogram
Target
84's the romsanie
lind
mst
by
pumps
bot
by
fresh
water
Fomtaniic the
nuw
by
pamps
but
by
fresh
watter
the romtnttc
kind
byy
pumps
fresh
walter
rontantcc
pazztgg
Waveform
Speetrogram an
Targer
I'm
expert
relationships
farmss
aren't
worlds
themsselves
Full
relationships
Earms
that
art
worlds
into
thems
Extreme
of
In
relationssiigs
Earass
that
art worlds
indo
(leanscive
Best
I'm
expert
in
relatisoships
farmss
that
aren't
worlds
into
themseevve es
Figure I: Example outputs from full model, subnetwork at extreme sparsity, and best performing subnetwork on TED-LIUM test utterances. Target: ground-truth transcriptions. Recognition errors are highlighted in red.
initialized e but the masks are generated by IMP mIMP. As result, with these two comparisons, we can explore if the two aspects are necessary to a winning ticket.
We tested the three pruning approaches on TED-LIUM dataset. Figure 2 shows the WER curves of the three approaches. From the figure, we found that random pruning and random ticket could find winning tickets, but the extreme remaining weights are much higher than the one identified by IMP (IMP: 4.4%, Random Pruning: 26.2%, Random Ticket: 10.7%). Random pruning can roughly retain its performance when the remaining weights are larger than 26.2%. However, the performance drops dramatically as the remaining weights continue decreasing. Random ticket achieves very similar performance to IMP when the remaining weights larger than 10.7%, but degrades much faster than IMP after this point. Results verified the previous statement in ASR models that both initial weight 0 and mask generated from IMP MIMP necessary for winning tickets. Additionally, the comparison between random pruning and random ticket suggests that myMP more important than 0 for winning tickets.
Additionally, we compared our proposed approach against four state-of-the-ar neural network compres- sion baselines: 1) Standard Pruning (Han et al., 2015;
Table 3: Comparison to state-of ofte-oofthhearr pruning and distillation methods on Conformer backhone.
Models are evaluated on I LibriSpeech test-clean subset. See Appendix A.5 for test-clean results.
Blalock et al., 2020; Shangguan et al., 2019); 2) Tutor- Net (Yoon et al., 2021): :) 3) MLKD + multi-teacher (Li et al., 1):01)::400) 4) Sequence-level KD (Takashima al.,
System
WER
Paarmss
Proposedexc (16.8% RW)
2.51
11.06M
2018). Baseline is the most commonly used and best performing network pruning approach, which iteratively prunes the lowest magnitude weights and train the network until reaching the target sparsity (without any rewinding). Baseline 2, 3, and 4 are three recent knowledge distillation approaches. As shown in Table 3, our winning ticket at extreme sparsity (Proposedext) achieves superior WER than all the baselines, while using less parameters.
Standard Pruning (16.8% RW)
3.96
11.06M
TutorNet
3.86
13.09M
MLKD multi-teacher
13.72
11.60M
Sequence- KD
17.58
11.60M
RQ3: Can IMP find better subnetworks by initializing from pre-trained model? Training ASR model from a good initialization usually results in more satisfactory performance (Jaitly et al., 2012), which is commonly achieved by finetuning pre-trained model in practice. Being aware of this, we would like investigate if IMP can find better subnetworks for ASR by initializing with weights from pre-trained model 0 = 0 pre- To verify this, we ran IMP on CNN-LSTM backbone with TED-LIUM dataset, where the weights were initialized from either LibriSpeech pre-trained model 0 pre OLabr or CommonVoice pre-trained model Opre Ocv.
WER curves of the subnetworks in this experiment are shown in Figure 3. Both initializing from Orabii and dev significantly improves the performance of the subnetworks at any sparsity. More importantly, we found rapid WER increase when the remaining weights are less than 4.4% for IMP initialized from 8g In contrast, the WERs of subnetworks in f and dcy systems degrades much slower when the remaining weights are extremely low, indicating that the subnetworks that are identified from pre-trained models can utilize the parameters more efficiently. possible reason is

---

Published as a conference paper at ICLR 2022
that the flatter loss surfaces of the pre-trained models (Liu et al., 2019a) make it more traceable for compression, leading to highly quality subnetworks, as also evidenced in (Chen et al., 2020b;a).
60
fx; MMP
0₂)
50
Pruning
50
f(;; MMMP
0 Libri)
WER (8)
WER 180
40
f(x; MIMP
0 0cv)
30
30
20
10
10
100.0
64.0
41.0
26.2
16.8
1.8
1.2
100.0
64.0
41.0
26.2
16.8
Remaining Weights (%)
Remaining Weights (16
Figure 2: The WER curves of the best subnetworks produced by different pruning approaches. IMP: itera- magnitude pruning, which is the pruning approach we used in subnetwork searching. Random pruning and Random Ticket are the two baseline pruning ap- proaches we evaluated.
Figure The WER curves of initialization with pre- trained models. We test CNN-LSTM backbone on TED- dataset, Oo: random initialization; Brabest initialized from LibriSpeech pre-trained model; Acv: initialized from CommonVoies pre-trained model.
Table 4 shows quantitative results of these sys- tems. Initializing from Labri achieves the best performance among all three systems. The WERs of subnetworks initialized from either Orabri or Ocv at different sparsities varies marginally, which makes it hard find the win- ning ticket at extreme sparsity. Therefore, we show the WER of all the system with 4.4% re- maining weights instead, corresponding to the extreme sparsity of the subnetworks initialized with fg- Although WER4.%% of 0 Labri and Ocv systems are equal or higher than WERs the corresponding full models, they are still significantly lower than Oo system. Additionally, the remaining weights of best performing subnetworks in 0rabri
Table Results of initialization with pre-traines mod- els. WERenn: WER of full models; WERbest? WER of the best performing matching subnetworks; 44R4.4% WER of subnetworks with 4.4% remaining weights. The brackets shows the remaining weights of the best per- forming subnetworks.
Initialization
WER
WER,,4%
WERnen
Ho 15.93
15.70
14,04 (16.8%)
L dcv 14,30
12.32
13.45
11.69 (32.8%)
14.28
13.3 (51.2%)
and Bey systems are higher than f.. however, when reducing the amount of remaining weights, the WERs only have minimal degradations.
Summary. We conducted extensive experiments to answer the three research questions about the existence of winning tickets in three ASR backbones. First, our results verified the existence of winning tickets in CNN-LSTM, RNN-Transducer, and Conformer models, 4.4% remaining weights). Second, we compared IMP with random pruning/tickets and other state-of- the-art network compression methods. The results suggest that matching subnetworks extracted by IMP significantly exceed those extracted by random pruning and random tickets in all measurements, which corroborates both binary pruning mask and weight initialization are indispensable in finding the winning tickets. Our approach also significantly outperforms other compression approaches, achieving state-of-the-ee performance on ASR model compression. Lastly, we explored the use of weights from pre-trained models to initialize IMP. We found that, in this way, IMP can identify more effective winning tickets compared to random initialization, and more importantly, these winning tickets have a considerably higher parameter efficiency. These results collectively advocate the profound benefits that LTH can bring to both server-side and on-device ASR.
high sparsity (e.g..
5 TOWARDS PRACTICAL ASR WITH WINNING TICKETS: STUDYING STRUCTURED SPARSITY, TRANSFERABLITY, AND NOISE ROBUSTNESS
After we proved the existence of winning tickets in ASR models, three more properties remain to be verified: structured sparsity, transferability, and noise robustness, which are key to ASR applications.
Study of structured sparsity As we mentioned earlier, designing chips that speeds up unstructured sparse networks are much more complex than those for structured sparsity (e.g.. Block-Sparse GPU Kernels (Gray et al., 2017)). Therefore, it is critical to verify if we can also identify winning tickets using structured pruning. However, the exploration of structured pruning in prior LTH studies is very limited. (You et al., 2020) is only able to identify limited winning tickets with structured sparsity
7

---

Published as a conference paper at ICLR 2022
(b)
Figure 4: Visualizations of weight matrix pruned with (a) unstructured sparsity (b) block sparsity. Pruned
weights are shown white color.
at low sparsity levels (70% remaining weights), which again emphasizes the difficulty in finding computation-friendgy sparse patterns.
Table 5: Results of structured sparsity study on TED-LIUM dataset.
In this experiment, we applied block sparsity (Narang et al., 2017b; Shang- guan et al., 2019; Wu et al., 2021) with 1x4 block to subnetwork searching. and then evaluated the subnetworks to see if they can match the full model performance. Results in Table 5 show that using block sparsity during subnetwork searching process lead to comparable results to using unstructured pruning. Also, we visualized the weights of winning ticket models that were discovered with unstructured sparsity and block sparsity, as shown in Figure 4. We can clearly observe I 1 block patterns from the weight matrix pruned with block sparsity.
also show the results with unstructured sparsity as reference.
Sparsity type
WER
WER
RWast
WER
RWhest
Unstructured Block sparsity
15.93
15.70
4.4%
14.04
16.8%
15.93
15.66
4.4%
13.96
21.0%
Study of Transferability In practical scenarios, the testing utterances are directly recorded from users in the wild, which may have very differrnnt distributions from the training utterances. common way to address this issue is through speaker adaptation (Gauvain & Lee, 1994; Leggetter & Woodland, 1995; Woodland, 2001; Liao, 2013; et al., 201 8b; Meng et al., 2019; Weninger et al., 2019), which is usually achieved by finetuning pre-trained model on utterances whose distributions are similar to test utterances directly, or with additional loss terms (Li et al., 2018b; Meng et al., 2019; Ding et al., 2020). In the context of winning tickets, these sparse architectures have to be transferable to new datasets so that we are able perform speaker adaptation on these models.
To examine the transferabilityy, we conducted the following experiment: First, we identify subnetworks fa; MPMP P @ 0o) on a source dataset D. at different sparsities using IMP. Then, we re-train each subnetwork target dataset D, and evaluate their performance. We tested the transferability between LIUM, Common Voice, and LibriSpeech datasets. These datasets are different in terms of recording scenario, noise level, speaker coverage, training set size, etc., and therefore, their utterances have very difference distributions. For example, TED-LIUM is created using TED talks, Common Voice is composed of volunteer's voice recorded through website mobile apps, and LibriSpeech is extracted from LibriVox audio books.
Figure 5 shows the performance of subnetworks transferring to the three target datasets. Results on the three datasets consistently suggest that the winning tickets are transferable across different datasets. When the remaining weights larger than around 26.2%, the transferring tickets can generally replicate the performance of the winning tickets identified on the target datasets. However, extremely high sparsity, the performance of transferring tickets degrades faster than the winning tickets identified on the target datasets, corresponding to the observations from previous studies (Chen et al., 2020c; Morcos al., 2019; Chen et al., 2021e). For example, Table 9 in Appendix A.6 shows WER and remaining weight of Common Voice and LibriSpeech winning tickets transferring to TED-LIUM dataset at extreme sparsity. Although the remaining weights of transferring winning tickets are higher than TED-LIUM winning ticket, they are still significantly lower than the full model. We also noticed is that the winning tickets identified from a larger dataset usually have better transferability, which is in accordance with (Morcos al., 2019).
Study of Noise Robustness The training/adaptation speech utterances are mostly collected from users, which are usually recorded from uncontrolled environments with notable background noise. Even in standard ASR benchmarks such LibriSpeech (Panayotov et al., 2015), there significant background noise even in its "clean" " subset (Zen et al., 2019). To test the noise robustness of winning tickets, conducted experiment on TED-LIUM dataset. Namely, we re-trained the
8

---

Published as a conference paper at ICLR 2022
60
Ticket Source
Tisket Source
Ticket Source
TED-LUM
CommonVoice
Libräpesch
CommoniVicc
TED UIM
TED-IIUM
Librispacch
Commonvorce
20
20
10
10
41.0
180.6
64.
198.
640
Bermanning Weiights
Remaininn We Weights [8]
Ramaining Weights (S4
TED-LIUM
(b) CommonVoice
(c) LibriSpeech
Figure 5: The WER curves of transferring winning tickets target datasets: (a) TED-LIUM, (h) Common Voice, (c) LibriSpeech (test-clean). Each curve represents the winning tickets extracted from a source dataset. The dashed lines indicate the WERs of full models on target datasets.
winning tickets identified from TED-LIUM, Common Voice, and LibriSpeech on resynthesized TED- LIUM dataset with background noise. We used DESED dataset (Serizel al., 2020) as the noise source, which consists of various background sounds in domestic environments such as bell, animals, running water, speech, vacuum cleaners, etc. During resynthesis, the noise audio is added to the original speech utterance with a random noise level, which is drawn from uniform distribution from [0, N.maa. In addition, we considered three different noise levels: Nmaz = (no noise), Neaa = 0.2 (low noise), Nmax 0.5 (high noise).
Table 6 shows WERs on TED-LIUM test set of the re-trained winning tickets. First, found the performance of the full model dramati- cally susceptible to background noise. When the noise level is increase from Maz = 0 to Nmax 0.5, WER increases from 15.93% to 38.21%. In contrast, WERs of TED-LIUM win- ning ticket at different noise level are very close, indicating that the sparse subnetwork is sub- staintially more robust to noise. Although the winning tickets from Common Voice and Lib- riSpeech cannot reach the WER of TED-LIUM winning tickets noise data, their WERs are still significantly lower than full model. More essen- tially, we can adapt transfer tickets to any target users without the need of re-finding winning tickets, which is more feasible for practical scenarios. Our findings coincide with the ones in (Ye et al., 2019; Gui et al., 2019) that an appropriate sparsity connectivity severs as implicit regularization for network training, which improves generalization on shifted data distributions (e.g., noisy perturbed data).
Table 6: Results of noise robustness study on TED- LIUM dataset. The noise level is drawn from uniform distribution from [0, Nmar), and we evaluate three noise conditions: Nmaz
(no noise), Nmaz 0.2 (low noise), Nmas = 0.5 (high noise).
Ticket Source
Nmaz
No
Nmer
Full model
15.93
16.80
38.21
TED-LIUM,tt
15,70
15.75
15.82
Common Voice
15.93
16.73
17.53
LibriSpeech
15.88
16.23
17.38
TED-LJUMpest
14.04
14.89
14.09
Common Voicquest
15.32
16.27
16.79
LibriSpeechheet 15.06
16.01
16.89
Summary In this section, we examined the structured sparsity, noise robustness, and trasferability of the winning tickets. Throughout the studies, we found that winning tickets can generalize to structured sparsity with no performance degradation. In addition, winning tickets identified from source datasets can achieve matching performance on target datasets, which verifies the transferability. Lastly, winning tickets (identified from either target dataset source datasets) are significantly more robust to noise compared to full models, especially when the noise level is high. These results jointly demonstrate the eligibility and benefits of winning tickets on-device ASR applications.
CONCLUSIONS AND FUTURE WORKS.
In this work, we examine and leverage lottery ticket hypothesis in speech recognition for the first time. Our extensive results show that the winning tickets not only ultra-lightweight, but also highly transferable and (even more) noise-robust, compared to the full models. These results collectively propound the of LTH into ASR models, bringing new insights to both L.TH theory and portable ASR research. In addition, we also would like to generalize THE LTH compression paradigm to other speech tasks such text-to-speech synthesis (Wang et al., 2017; Shen al., 2018) and voice-to-voice conversion (Jia et al., 2019; Biadsy et al., 2019; Zhao et al., 2019). These new directions require further customized studies and point to new opportunities for LTH in speech research.
LEDEENTE
Z. Wang is in part supported by the NSF EPCN grant #2053272, and the U.S. Army Research I Laboratory Cooperative Research Agreement W911NF177 1991NN977--0119 (IOBT REIGN).

---

Published as a conference paper at ICLR 2022
REFERENCES
Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, al. Deep speech End-to-end speech recognition in english and mandarin. In ICML, pp. 173-182. PMLR, 2016.
Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. Common Voice: A massively- multilingual speech corpus. Proceedings of The 12th Language Resources and Evaluation Conference, pp. 4218 4222, 2020.
Sercan Ô Anı, Markus Kliegl, Rewon Child, Joel Hestness, Andrew Gibiansky, Chris Fougner, Ryan Prenger, and Adam Coates. Convolutional recurrent neural networks for small-footprint keyword spotting. In INTERSPEECH, pp. 1606-1610, 2017.
Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vee 2.0: A framework for supervuper learning of speech representations. arXiv preprint arXiv:2006. 11477, 2020.
Dumitry Bahdanau, Jan Chorowski, Dmitriy Serdyuk, Philemon Brakel, and Yoshua Bengio. End- to-end attention-based large vocabulary speech recognition. In ICASSP, pp. 4945-4949, IEEE, 2016.
Fadi Biadsy, Ron J Weiss, Pedro Moreno, Dimitri Kanvesky, and Ye Jia. Parrotron: An end-to-end speechh conversion model and its applications to mearing-impaine speech and speech separation. In INTERSPEECH, pp. 4115-4119, 2019.
Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state of neural network pruning? arXiv preprint arXiv:2003.03033, 2020.
Maxime Burchi and Valentin Vielzeuf. Efficient conformer: Progressive downsampling and grouped attention for automatic speech recognition. arXiv preprint arXiv:2109.01163. 2021.
William Chan, Navdeep Jaitly, Quoc Le, and Oriol Vinyals. Listen, attend and spell: A neural network for large vocabulary conversational speech recognition. In ICASSP, pp. 4960-4964 IEEE, 2016.
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Michael Carbin, and Zhangyang Wang. The lottery tickets hypothesis for supervised and self-supervised pre-training in computer vision models. arXiv preprint arXiv:20122.06908 2020a.
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. The lottery ticket hypothesis for pre-trained BERT networks. In NeurIPS, 2020b.
Tianlong Chen, Yu Cheng, Zhe Gan, Jingjing Liu, and Zhangyang Wang. Ultra-data-efficien GAN training: Drawing lottery ticket first, then training it toughly. arXiv preprint arXiv:2103.00399 2021a.
Tianlong Chen, Yongduo Sui, Xuxi Chen, Aston Zhang, and Zhangyang Wang. A unified lottery ticket hypothesis for graph neural networks. arXiv preprint arXiv:2102.06790 2021b.
Tianlong Chen, Zhenyu Zhang, Sijia Liu, Shiyu Chang, and Zhangyang Wang. Long live the lottery: The existence of winning tickets in lifelong learning. In ICLR, 2021c
Tianlong Chen, Xuxi Chen, Xiaolong Ma. Yanzhi Wang, and Zhangyang Wang. Coarsening the granularity: Towards structurally sparse lottery tickets. arXiv preprini arXiv: 04736, 2022a.
Tianlong Chen, Zhenyu Zhang, pengjun wang, Santosh Balachandra, Haoyu Ma, Zehao Wang, and Zhangyang Wang Sparsity winning twice: Better robust generalization from more efficient training. In International Conference on Learning Representations, 2022b. URL https:// openreview. wttp:/oormm.dd--/Suu.Xt.xx
Xiaohan Chen, Yu Cheng, Shuohang Wang. Zhe Gan, Zhangyang Wang, and Jingjing Liu. Earlybert: Efficient bert training via early-bird lottery tickets. arXiv preprint arXiv: 100.000000000 00.00063, 2020c.
10

---

Published as a conference paper at ICLR 2022
Xuxi Chen, Tianlong Chen, Zhenyu Zhang, and Zhangyang Wang. You are caught stealing my winning lottery ticket! making lottery ticket claim its ownership. Advances in Neural Information Processing Systems, 34, 2021d.
Xuxi Chen, Zhenyu Zhang, Yongduo Sui, and Tianlong Chen. GANs can play lottery tickets too. In ICLR, 2021e.
Chung-Cheng Chiu, Tara N Sainath, Yonghui Wu, Rohit Prabhavalkar, Patrick Nguyen, Zhifeng Chen, Anjuli Kannan, Ron Weiss, Kanishka Rao, Ekaterina Gonina, et al. Sfate-of-the-are speech recognition with sequenceeto--sequeece models. In ICASSP, pp- 4774-4778. IEEE, 2018
Jan Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. intention-based models for speech recognition. In ICONIP, pp. 577-585, 2015.
Jordan Cohen, Embedded speech recognition applications in mobile phones: Status, trends, and challenges. In ICASSP, pp. 5352-5355. IEEE, 2008
Shrey Desai, Hongyuan Zhan, and Ahmed Aly. Evaluating lottery tickets under distributional shifts. arXiv preprint arXiv: 1910. 12708, 2019.
Fenglin Ding, Wu Guo, Bin Gu, Zhenhua Ling, and Jun Du Adaptive speaker normalization for cte-based speech recognition. In INTERSPEECH, pp. 1266-1270, 2020.
Linhao Dong, Shuang Xu, and Bo Xu. Speech-transformerr: no-recurrence sequence-to-sequene model for speech recognition. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3884-5888, IEEE, 2018a. Sa.
Linhao Dong, Shiyu Zhou, Wei Chen, and Bo Xu. Extending recurrent neural aligner for streaming end-to-end speech recognition in Mandarin. In INTERSPEECH, pp. 816-820, 2018b.
Utku Evei, Fabian Pedregosa, Aidan Gomez, and Erich Elsen. The difficulty of training sparse neural networks. arXiv preprint arXiv: 10Xiv:1900.. 2019.
Florian Eyben, Martin Wöllmer, Björn Schuller, and Alex Graves. From speech to letters-using novel neural network architecture for grapheme based ASR. 2009 IEEE Workshop on Automatic Speech Recognition & Understanding, pp. 376-380. IEEE, 2009.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In ICLR, 2019a. URL https: ttttps://openreee
net htt//foummp/ wowuummii-riii d-rJl- -ec-rrr1-33cFF----- 7.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In ICLR, 2019b.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In ICMI. pp. 3259-3269. PMLR, 2020a.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In ICML, 2020b.
Jonathan Frankle, David J. Schwab, and Ari S. Morcos. The early phase of neural network training. In ICLR, 2020c. URL /ttp://:////eeeeeeeeee/eo net wttp::.eee/ffoumm//// for.mmormmmmm ii -k MalkkliiNN LiRNFWS.
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv: 110.::/99020055..1.. 09574, 2019.
Dawei Gao, Xiaoxi He, Zimu Zhou, Yongxin Tong, Ke Xu, and Lothar Thiele. Rethinking pruning for accelerating deep inference the edge. In SIGKDD, pp. 155-164, 2020.
J-L Gauvain and Chin-Hui Lee. Maximum a posteriori estimation for multivariate gaussian mixture observations of markov chains. IEEE transactions on speech and audio processing, 2(2):291-299 1994.
Sharath Girish, Shishira Maiya, Kamal Gupta, Hao Chen, Larry Davis, and Abhinav Shrivastava. The lottery ticket hypothesis for object recognition. arXiv preprint arXiv:20/2.06 0,0443, 2020.

---

Published as a conference paper at ICLR 2022
Alex Graves. Sequence transduction with recurrent neural networks. arXiv preprint arXiv: arXiv: 2012.
Alex Graves and Navdeep Jaitly. Towards end-to-end speech recognition with recurrent neural networks. In ICML, pp. 1764-1772. PMLR, 2014.
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks. ICMI., pp. 369-376, 2006.
Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recurrent neural networks. In ICASSP, pp.
Icee, 2013.
Scott Gray, Alec Radford, and Diederik P Kingma. Gpu kernels for block-sparse weights, arXiv preprint arXiv:1711.0922 3, 2017.
Shupeng Gui, Haotao Wang, Chen Yu, Haichuan Yang, Zhangyang Wang, and Ji Liu. Model compression with adversarial robustness: A unified optimization framework. arXiv preprint arXiv:1902.08538, 2019.
Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. Conformer: Convolution-augmennte transformer for speech recognition. arXiv preprint 2005:00000000800 2020.
Reinhold Haeb-Umbache Umbach, Shinji Watanabe, Tomohiro Nakatani, Michiel Bacchiani, Bjorn Hoffmeister, Michael L Seltzer, Heiga Zen, and Mehrez Souden. Speech processing for digital home assistants: Combining signal processing with deep-learning techniques. IEEE Signal processing magazine, 36(6):111-122) 2019.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv: 1500000149 2015.
Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, et al. Deep speech: Scaling up end-to-end speech recognition. arXiv preprint arXiv: 1412:544 2014.
Yanzhang He, Tara N Sainath, Rohit Prabhavalkar, Ian McGraw, Raziel Alvarez, Ding Zhao, David Rybach, Anjuli Kannan, Yonghui Wu, Ruoming Pang, et al. Streaming end-to-end speech recogni- tion for mobile devices. In ICASSP, pp. 6381-6385. IEEE, 2019.
Navdeep Jaitly, Patrick Nguyen, Andrew Senior, and Vincent Vanhoucke. Application of pretrained deep neural networks to large vocabulary speech recognition. In INTERSPEECH, 2012.
Ye Jia, Ron Weiss, Fadi Biadsy, Wolfgang Macherey, Melvin Johnson, Zhifeng Chen, and Yonghui Wu. Direct speech-to-speed translation with a sequence-to-sequenee model. In INTERSPEECH. pp. 1123-1122 2019.
Neha Mukund Kalibhat, Yogesh Balaji, and Soheil Feizi. Winning lottery tickets in deep generative models. arXiv preprint arXiv:2010.02350, 2020.
Kwangyoun Kim, Kyungmin Lee, Dhananjaya Gowda, Junmo Park, Sungsoo Kim, Sichen Jin, Young- Yoon Lee, Jinsu Yeo, Daehyun Kim, Seokyeong Jung, et al. Attention based on-device streaming speech recognition with large speech corpus. In ASRU, pp.
IEEE, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
Keisuke Kinoshita. Mare Delcroix, Takuya Yoshioka, Tomohiro Nakatani, Emanuel Habets, Reinhold Slaeb-Umbach, Volker Leutnant, Armin Sehr, Walter Kellermann. Roland Maas, et al. The reverb challenge: A common evaluation framework for dereverberation and recognition of reverberant speech. In 2013 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, pp. 1-4. IEEE, 2013.
Keisuke Kinoshita, Tsubasa Ochiai, Mare Delcroix, and Tomohiro Nakatani. Improving noise robust automatic speech recognition with single-channel time-domain enhancement network. In ICASSP, pp. 7009-7013. IEEE, 2020.
12

---

Published as a conference paper at ICLR 2022
Markus Kliegl, Siddharth Goyal, Kexin Zhao, Kavya Srinet, and Mohammad Shoeybi. Trace norm regularization and faster inference for embedded speech recognition RNNs. arXiv preprint arXiv:1710.00026, 2017. 2017.
Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In EMNLP (Demonstration), 2018.
Christopher J Leggetter and Philip C Woodland. Maximum likelihood linear regression for speaker adaptation of continuous density hidden markov models. Computer speech & language, 9(2): 171-185, 1995.
Chenxing Li, Lei Zhu, Shuang Xu, Peng Gao, and Bo Xu. Compression of acoustic model via knowledge distillation and pruning. In ICPR, pp. 2785-2790. IEEE, 2018a.
Jinyu Li, Li Deng, Yifan Gong, and Reinhold bubbch. An overview of noise-robust automatic speech recognition. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 22(4): 745-777, 2014.
Ke Li, Jinyu Li, Yong Zhao, Kshitiz Kumar, and Yifan Gong. Speaker adaptation for end-to-ent CTC models. In SLT, pp. 542-549, IEEE, 2018b.
Zerui Li, Yue Ming, Lei Yang, and Jing-Hao Xue. Mutual-learning sequence-level knowledge distillation for automatic speech recognition. Neurocomputing, 428:259-267, 2021.
Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein, and Joey Gonzalez.
Train big, then compress: Rethinking model size for efficient training and inference of transformers. In ICML, pp. 5958-5968. PMLR, 2020.
Hank Liao. Speaker adaptation context dependent deep neural networks. ICASSP, pp. 7947- 7951. IEEE, 2013.
Hong Liu, Mingsheng Long, Jianmin Wang, and Michael Jordan. Towards understanding the transferability of deep representations. arXiv preprint arXiv: 12031, 2019a.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. In ICLR, 2019b.
Haoyu Ma, Tianlong Chen, Ting-Kuei Hu, Chenyu You, Xiaohui Xie, and Zhangyang Wang. Good students play big lottery better. arXiv preprint (=XXx::211.03255: 2021.
Ian McGraw, Rohit Prabhavalkar, Raziel Alvarez, Montse Gonzalez Arenas, Kanishka Rao, David Rybach, Ouais Alsharif, Haşim Sak, Alexander Gruenstein, Françoise Beaufays, al. Personalized speech recognition on mobile devices. In ICASSP, pp. 5955-5959, IEEE, 2016.
Rahul Mehta. Sparse transfer learning via winning lottery tickets. arXiv preprint 1rXhv:1905.07785 2019.
Zhong Meng, Yashesh Gaur, Jinyu Li, and Yifan Gong. Speaker adaptation for attention-based end-to-end speech recognition. In INTERSPEECH, pp. 241-245, 2019.
Yajie Miao, Mohammad Gowayyed, and Florian Metze. EESEN: End-to-end speech recognition using deep RNN models and WFST-based decoding. In ASRU, pp. 167-174. IRBE, 2015.
Ari Morcos, Haonan Yu, Michela Paganini, and Yuandong Tian. One ticket to win them all: Generalizing lottery ticket initializations across datasets and optimizers. In NewrIPS, pp. 4932- 4942, 2019.
Sharan Narang, Erich Elsen, Gregory Diamos, and Shubho Sengupta. Exploring sparsity in recurrent neural networks. arXiv preprint arXiv:1704.05/18 201
Sharan Narang, Eric Undersander, and Gregory Diamos. Block-sparse recurrent neural networks. arXiv preprint arXiv: 171100782, 2017b.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An ASR corpus based on public domain audio books. In ICASSP, pp. 5206-5210. IEBE, 2015.
13

---

Published as a conference paper at ICLR 2022
Jinhwan Park, Yoonho Boo, Iksoo Choi, Sungho Shin, and Wonyong Sung. Fully neural network based speech recognition on mobile and embedded devices. In NeurIPS, pp. 10620-10630, 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. NeurIPS, 32:8026-8037, 2019.
Rohit Prabhavalkar, Tara Sainath, Yonghui Wu, Patrick Nguyen, Zhifeng Chen, Chung-Chenny Chiu, and Anjuli Kannan. Minimum word error rate training for attention- based sequence-tor -eeuueeece models. In ICASSP, pp. 4839-4843, IEEE, 2018.
Sai Prasanna, Anna Rogers, and Anna Rumshisky. When BERT plays the lottery, all tickets are winning. arXiv preprint wrXiv:2055005561, 2020.
Kanishka Rao, Haşim Sak, and Rohit Prabhavalkar. Exploring architectures, data and units for streaming end-to-end speech recognition with RNN-Transducer. In ASRU, pp. 199, IEEE, 2017.
Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. In ICLR, 2020.
Anthony Rousseau, Paul Deléglise, and Yannick Esteve. TED-LIUM: automatic speech recognition dedicated corpus. In LREC, pp. 125-129, 2012.
Tara N Sainath, Yanzhang He, Bo Li, Arun Narayanan, Ruoming Pang, Antoine Bruguier, Shuo-yiin Chang, Wei Li, Raziel Alvarez, Zhifeng Chen, al. A streaming on-device end-to-end model surpassing server-side conventional model quality and latency. In ICASSP, pp. 6059-6063. IEEE, 2020.
Pedro Savarese, Hugo Silva, and Michael Maire. Winning the lottery with continuous sparsification. In NeurlPS, 2020.
Johan Schalkwyk, Doug Beeferman, Françoise Beaufays, Bill Bymee Ciprian Chelba, Mike Cohen, Maryam Kamvar, and Brian Strope. "Your word is my command": Google search by voice: A case study. In Advances in speech recognition, pp. 61-90. Springer, 2010.
Romain Serizel, Nicolas Turpault, Ankit Shah, and Justin Salamon. Sound event detection in synthetic domestic environments. In ICASSP, pp. 86-90. IEEE, 2020.
Yuan Shangguan, Jian Li, Qiao Liang, Raziel Alvarez, and Ian McGraw. Optimizing speech recognition for the edge. arXiv preprint arXiv: 1909. 2019.
Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al. Natural TTS synthesis by conditioning Wavenet on Mel spectrogram predictions. In ICASSP, pp. 4779-4783. IEEE, 2018
Khe Chai Sim, Petr Zadrazil, and Françoise Beaufays. An investigation into on-device personalization of end-to-end automatic speech recognition models. In INTERSPEECH, pp. 774-778, 2019.
Ryoichi Takashima, Sheng Li, and Hisashi Kawai. An investigation of a knowledge distillation method for ctc acoustic models. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5809-5813. IEEE, 2018.
Ryu Takeda, Kazuhiro Nakadai, and Kazunori Komatani. Node pruning based on entropy of weights and node activity for small-footprint acoustic model based deep neural networks. In INTERSPEECH, pp. 1636-1640, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Alex Waibel, Ahmed Badran, Alan W Black, Robert Frederking, Donna Gates, Alon Lavie, Lori Levin, Kevin A Lenzo. Laura Mayfield Tomokiyo. Jurgen Reichert, et al. Speechalator: Two-way translation on a consumer PDA. In Eighth European Conference on Speech Communication and Technology, 2003.
14

---

Published as a conference paper at ICLR 2022
Chaoqi Wang, Guodong Zhang. and Roger Grosse. Picking winning tickets before training by preserving gradient flow. In ICLR, 2020a.
Dong Wang, Xiaodong Wang, and Shaohe Lv. An overview of end-to-end automatic speech recogni- tion. Symmetry, 11(8):1018, 2019.
Yongqiang Wang, Abdelrahman Mohamed, Due Le, Chunxi Liu, Alex Xiao, Jay Mahadeokar, Hongzhao Huang, Andros Tjandra, Xiaohui Zhang, Frank Zhang, et al. Transformer-base acoustic modeling for hybrid speech recognition. In ICASSP 2020-2020 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 6874-6878. IEEE, 2020b.
Yuxuan Wang, RJ Skerry-Raan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, et al. Tacotron: Towards end-to-end speech synthesis. In INTERSPEECH, pp. 4006-4010, 2017.
Wei Wen, Yuxiong He, Samyam Rajbhandari, Minjia Zhang, Wenhan Wang, Fang Liu, Bin Hu, Yiran Chen, and Hai Li. Learning intrinsic sparse structures within long short-term memory. In ICLR, 2018.
Felix Weninger, Jesús Ferrés-Ferrer, Xinwei Li, and Puming Zhan. Listen, attend, spell and adapt: Speaker adapted sequence-to snseence-to-equocee se-ueneecce ASR. In INTERSPEECH, pp. 3805-3809, 2019.
Phil C Woodland. Speaker adaptation for continuous density hmms: A review. In ISCA Tutorial and Research Workshop (ITRW) on Adaptation Methods for Speech Recognition, 2001.
Zhaofeng Wu, Ding Zhao, Qiao Liang, Jiahui Yu, Anmol Gulati, and Ruoming Pang. Dynamic sparsity neural networks for automatic speech recognition. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6014-6011 6018. IEEE, 2021.
Shaokai Ye, Kaidi Xu, Sijia Liu, Hao Cheng, Jan-Henrik Lambrechts, Huan Zhang, Aojun Zhou, Kaisheng Ma. Yanzhi Wang, and Xue Lin. Adversarial robustness vs. model compression, or both? In ICCV, October 2019.
Ji Won Yoon, Hyeonseung Lee, Hyung Yong Kim, Won Ik Cho, and Nam Soo Kim. Tutornet: Towards flexible knowledge distillation for end-to-end speech recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29: 29:1626-1638, 2021.
Haoran You, Chaojian Li, Pengfei Xu, Yonggan Fu, Yue Wang, Xiaohan Chen, Richard G. Baraniuk, Zhangyang Wang, and Yingyan Lin. Drawing early-bird tickets: Toward more efficient training of deep networks. ICLR, 2020.
Haonan Yu, Sergey Edunov, Yuandong Tian, and Ari S Morcos. Playing the lottery with rewards and multiple languages: Lottery tickets in RL and NLP. In ICLR, 2020.
Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. LibriTTS: A corpus derived from LibriSpeech for Text-to-Speech. In INTERSPEECH, pp. 1526-1530, 2019.
Shunshi Zhang and Bradly C Stadie. One-shot pruning of recurrent neural networks by jacobian spectrum evaluation. In ICLR, 2019.
Yu Zhang, William Chan, and Navdeep Jaitly. Very deep convolutional networks for end-to-end speech recognition. In ICASSP, pp. 4849.4849. IEEE, 2017.
Guanlong Zhao, Shaojin Ding, and Ricardo Gutierrez-Osuna. Foreign accent conversion by synthe- sizing speech from phonetic posteriorgrams. In INTERSPEECH, pp. 2843-2847, 2019.
15

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.1 DETAILS IN BACKBONE ARCHITECTURES
CNN-LSTM. The network is composed of two convolutional layers and five recurrent layers, fol- lowed by final fully-comnected layer. The first convolutional layer has 32 41 x1 (in frequency xtime) kernels, with 2x2 stride. The second convolutional layer has 32 21 kernels, with 2x1 stride. For each convlutional layers, tanh is used nonlinearity. Following these, there five bi-directional RNN layers, each of them having 1,024 units. A final fully-connected layer with softmax operator outputs a probability distribution over characters. The total number of parameters the full model is 86.62M.
The input to the networks is sequence of 161-dimensional magnitude spectrogram (160 magnitude points plus DC component), extracted with 20ms window, 10ms shift, and 320-point fast fourier transform. The ground-truth labels are represented by 28 graphemes, including 26 English characters, space, and apostrophe symbol.
RNN-Transducer. In RNN-Transducer, we set the encoder and decoder/joint model have 5 and bidirectional-LSSTT layers, each of them having 1,024 units. The total number of parameters in the full model 132.23M.
The input to the networks is sequence of 80-dimensional Mel-filterbanks, extracted with 25ms window, 10ms shift, and 512-point fast fourier transform. We used a 1000-dimensional sentence-piece embedding (Kudo & Richardson, 2018) to represent the labels.
Conformer. In Conformer, our parameter settings are based on EffiicientConfoomee-CCCC in (Burchi & Vielzeuf, 2021). Additionally, we reduce the encoder dimensions from [360, 512, 720] to (256, 384, 512], which we found to have no performance degradation and speed up training. The model has 17 encoder layers, each of them is composed of 8-head convolution augmented attention. The convolution kernel has a shape of x32. The total number of parameters the full model is 65.84M. During the initial submission, et al., 2020). However, we found that this model is susceptible to gradient explosion when the batch size is limited (e.g., 256). To improve the reproducibility, we switch in the camera ready version, which we found to be more stable during training with smaller batch size.
using the configurations of Conformer(L) in (Gulati
Conformer-CTCLL
The input to the networks is sequence of 80-dimensional Mel-filtrbbakss extracted with 25ms window, 10ms shift, and 512-point fast fourier transform. We used 256-dimensional sentence sentence-piece embedding (Kudo & Richardson, 2018) to represent the labels.
A.2 MORE IMPLEMENTATIONAL DETAILS
As we mentioned in abstract, we will open-source all of the code once the paper is peer-reveiwed. Here, we provide key references and tools that we used during our implementation.
We implement neural network training using PyTorch library (Paszke et al., 2019). Our implementa- tion of CTC backbone is based on:
/ htt.attcccmeeeeeaaaeeeeeeeeeeeep
Our implementation of RNN-Transduce is based on:
Our implementation of Conformer is based on:
We used PyTocc pruning library for unstructured sparsity:
https::/pytoott http://cww..r..ectttaa///////aaa util.. prune.gloee obobl. www.tuuuuuuueddd
trune.gllbbl_mmmttuuutrreed html
16

---

Published as a conference paper at ICLR 2022
We implemented pruning functions for structured sparsity based on:
utils.
prune arunaeBandoostrrutturd....t
A.3 EVALUATING WINNING TICKETS THREE ASR BACKBONES ON LIBRISPEECH TEST-OTHER SUBSET
The performance of the three backbones on LibriSpeech test-other subset is shown in Table 7.
Table 7: Performance of three backhones at the extreme sparsity at the best performance LibriSpeech lest-other subset. #Prramse.nt number of parameters in full model. in which WER WER full models: WER WER of the winning tickets extreme sparsity: WER WER the best performing winning tickets. Remaining Weight (RW) is included as model complexity measurement.
Mega (x 10%) the unit;
Backbone
# Parammsan
WER
WER...
RW (#Params)
WER
RWh (#Params)
CNN-LSTM
86.62M
20.59
20.53
21.0% (18.19M)
19.21
$1.2% (44.34M)
MONN-Transduce
132.23M
16.93
16.30
10.7% (14.14M)
15.55
41.0% (54.21M)
Conformer
65.84M
6.55
6.47
16.8% (11.06M)
6,28
51.2% (33.71M)
A.4 MORE EXAMPLES OF THE MODEL OUTPUTS
We provided a couple more examples of the outputs from full model, winning tickets at extreme sparsity, and best performing winning tickets on TED-LIUM test utterances in Figure 6. The examples the first row show the cases where the winning tickets avoid the errors appeared in the full model outputs. Meanwhile, we also noticed a few cases where winning tickets produce erroneous recognition results, as shown the second and the third rows. Most of these utterances have either a very fast speaking rate unclear pronunciations of words. Therefore, neither full model nor winning tickets reasonably transcribe the utterances in challenging cases, showing the limited capability of the models.
A.5 COMPARISON TO OTHER MODEL COMPRESSION APPROACHES ON LIBRISPEECH TEST-OTHER SUBSET
Comparison between the proposed approach and state-of-the-art distillation and pruning methods on LibriSpeech test-other subset is shown in Table 8. Sequence-level KD approach (Takashima et al., 2018) does not report their evaluation results on test-other subset, and therefore, we use "N/A" the table.
Table 8: Comparison to state-of oh-the-rtt art distillation and pruning methods on Conformer backbone. Models are evaluated on LibriSpeech test-other subset.
System
WER
#Params
Propused (16.8% RW)
6.47
11.06M
Standard Pruning (16.8% RW)
8.79
1106M
TutorNet
11.14
13.09M
MLKD multi-tcacher
34.28
11.60M
Sequence-leve KD
N/A
11.60M
A.6 MORE RESULTS IN TRANSFERRING STUDY
We have shown the WER curves of the transferring winning tickets transferred in Figure 5. Here, we included the WER and RW the extreme sparsity in Table 9, Table10 and Table 11, respectively. The transferring tickets to both TED-LIUM and Common Voice have a extreme remaining weights of 32.8%, and those to LibriSpeech have extreme remaining weights of 32.8% compared to the full
17

---

Published as a conference paper at ICLR 2022
Speclrugram
Turget
out
they
are
Full
Estrome
fids
Best
fell
love
with
fidh
Wiveform
Spectrugram
didw't
smything
kind
Pull
thing
about
fish
and
Extreme
thing
about
fish
Best
don't
knoww
thing
about
Fubh bec
con of
set
Waveform
Spectrugram
and
miguel
and
this
and
Farm that its
Pull
and
tbe
and
and
this
and
Farm that
measures measures
SUICCSS
Extreme
and
tie
process
began
and
this
company
and
Farm
that
of
Best
and
process
began
and
this
company
on
Farm
thai
micssurss
$6 SUCCOO
Figure 6: Example outputs from full model, subnetwork at extreme sparsity, and best performing subnetwork on TED-LIUM test utterances. Target: ground-truth transcriptions. Recognition errors are highlighted in red.
models. Although they are not sparse the winning tickets identified from the target datasets, the transferring tickets are still much more efficient than full models. These results are in consensus with findings in Section 5.
Table Performance of transferring Common Voice and LibriSpeech winning tickets to TED-LIUM dataset at the extreme sparsity. Remaining Weight (RW) is included as the spatial and temporal complexity measurements.
Ticket Source
WER
WER
RW
TED-LIUM
15.93
15.70
4.4%
Common Voice
N/A
15.93
41.0%
LibriSpeech
N/A
15,88
32.8%
Table 10: Performance of transferring TED-LIUM and LibriSpeech winning tickets Common Voice dataset at the extreme sparsity. Remaining Weight (RW) is included as the spatial and temporal complexity measurements.
Ticket Source
WER
WER
RW
Common Voice
5.56
5.41
16.8%
TED-LIUM
N/A
4.25
51.2%
LibriSpeech
N/A
5.53
32.8%
18

---

Published as a conference paper at ICLR 2022
Table 11: Performance of transferring TED-LIUM and Common Voice winning tickets to LibriSpeech dataset at the extreme sparsity. Remaining Weight (RW) is included as the spatial and temporal complexity measurements.
Ticket Source
WERfull
WER
RW
LibriSpeech
8.02
7.98
21.0%
TED-LIUM
N/A
7.72
32.8%
Common Voice
N/A
7.19
41.0%
A.7 EXTRA NEW RESULTS
A.7.1 RESULTS ON THE EXISTENCE OF THE THREE BACKBONES ON TED-LIUM AND COMMON
Table Performance three backhones at the extreme sparsity or at the best performance on TED-LIUM. # number of parameters in full model, in which we use Mega (x10%) as the unit: WER WER of full models; WER WER of the winning tickets at extreme sparsity; WER WER of the best performing winning tickets. Remaining Weight (RW) included model complexity measurement.
Backbone
Parramsall
WER4an
WER
RW exco
WER
RW best
CNN-LSTM
86.62M
15.93
15.07
4.4%
14.04
16.8%
132.23M
12.43
12.26
2.2%
13.96
41.0%
Conformer
65.84M
7.40
7.36
3.5%
7.01
21.0%
Table 13: Performance of three backbones al the extreme sparsity or at the best performance on Common Voice. #Prramssnn: number of parameters in full model, in which models; WER WER the winning tickets extreme sparsity; WERtaa:: WER the best performing winning tickets. Remaining Weight (RW) is included as model complexity measurement.
Mega (x10") as the unit; WER WER of
Backbone
#Paramstnn
WER
WER
RW RW
WERnest
RW best
CNN-LSTM
86.62M
5.57
5.41
16.8%
4.17
64.0%
Rrnnsuann
132.23M
3.41
3.39
10.7%
3.02
26.2%
Conformer
65.84M
1.37
1.35
8.6%
1.20
20.9%
A.7.2 RUN TIME EVALUATIONS
Table 14: Run time evaluation of the three backbones on LibriSpeech at the extreme sparsity or at the best performance. Here we use the Number of Coltiply-Accuminn Operations (MACs) in Giga (G) to measure the time complexity. We compute the percentage compared full model for all the subnetworks. M AC MACs of thefull model. AC MACs of the winning tickets extreme sparsity. M. AC MAC's of the best performing winning tickets.
Backbone
MAC's
MACseat
MACsbest
CNN-LSTM
77.88G
20.
49.9%
-NN-Tuunnse
124.56G
9.6%
39.8%
Conformer
62.27G
15.9%
49.4%
19

---

Published as a conference paper at ICLR 2022
Table 15: Run time evaluation of the three backhones TEDLIUM the extreme sparsity the best performance. Here we use the Number of Muliply-Acuuulaat Operations (MACs) in Giga (G) to measure the run time complexity. We compute the percentage compared to full model for all the subnetworks. M AC&srutt MACs of thefull model. M.C's MACs of the winning tickets at extreme sparsity. MAC Sh'nettt MACs of the best performing winning tickets.
Backbone
MACs fuil
MACseat
MACShest
CNN-LSTM
24.34G
16.0%
63.1%
ENN-Transduces
38.92G
1.5%
39.9%
Conformer 19.45G
2.9%
19.7%
Table 16: Run time evaluation of the three backbones on Common Voice at the extreme sparsity the best performance. Here we use the Number of run time complexity. We compute the percentage compared to full model for all the subnetworks, M 4f MACs of thefull model. MCrr MACs of the winning tickets extreme sparsity. MAC AC MACs of the best performing winning tickets.
Operations (MACs) in Giga (G) to measure the
Backbone
MAC' full
MACsext
MACsbest
CNN-LSTM
53.54G
20.
49.9%
RNN-Transduuer
85,63G
9.2%
25.0%
Conformer
42.81G
7.1%
19.7%
A.7.3 UPDATED NOISE ROBUSTNESS RESULTS
Table 17: Performance of ASR models when adding noise only at test time. Results are shown the hest performance.
Ticket no SNR=10dB
SNR=5dB
SNR=3dB
SNR=0dB
SNR=-5dB
Full model
15.93
15.99
19.65
44.77
67.58
100
TEDLIUM 14.04
14.12
16.30
20.88
40.35
100
Common Voiceaust
15.32
15.44
17.24
21.01
45.54
100
LibriSpeechuaat
15.06
15.25
17.49
21.50
43.98
> 100
A.7.4 BLOCK SPARSITY EXPERIMENTS WITH LARGER BLOCK SIZE
Table 18:
Results of structured sparsity study TED-LIUM dataset. We also show the results with unstructured sparsity as a reference.
Spursity type
WER
WER...
RW...
WER
RW best
Unstructured
15.93
15.70
4.4%
14.04
16.8%
Block sparsity x
15.93
15.66
21.0% 4.4%
13.96
Block spursity x 16
15.93
15.72
4.4%
14.33
16.8%
A.7.5 PSEUDO-CODE OF LTH PRUNING ALGORITHM
Algorithm Lottery Ticket Hypothesis Pruning
1: Set initial mask m, with the weight initialization
2: repeat
3: Rewind the weight to 0
4: Train f(x; m @ 0) for i epochs with algorithm AP, i.e., AP (x 0))
5: Prune 20% of remaining weights in AP (f( m @ 0)) and update m accordingly
6: until the sparsity of reaches the desired sparsity level s
7: Return ((; m 0).
20

---

Published as a conference paper at ICLR 2022
A.7.6 FURTHER EXPLANATION OF BLOCK SPARSITY.
In terms of block sparsity, it prunes block of parameters (e.g., 1x4 block) instead of just one parameter, which has the minimal magnitude. Unstructured pruning could be thought as block sparsity with Ixl block.
A.7.7 EXPLANATION OF THE CHOICE OF SPARSITY LEVELS
In our current setting, prune the 20% of remaining weights that have the lowest magnitude. The reason for choosing the value to be 20% is based on considerations of model performance and computation resources. When the amount of weights pruned each iteration becomes larger, less iterations will be needed. However, it is more likely to have larger performance regressions, since the pruned weights could be useful in the subnetwork at higher sparsity. On the other hand, the amount of weight pruned at each iteration is too small, the number of iterations required would be very large, which is extremely resource demanding. As a result, feel 20% is good one considering both factors.
21


Reviews:
[{"review_id": "vEXhKZUpp8K", "reviewer": "Reviewer_5kPn", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "strengths:\n- straightforward application of the LTH strategy to speech recognition\n- showing the significant performance (very high compression without losing the performance so much). This is especially important for on-device scenarios.\n- extensions for transfer learning scenarios\n\nweaknesses:\n- the novelty of the method itself is incremental from the original LTH and related work.\n- some references are not appropriate —> (correction) the reference part was improved through the discussions", "summary_of_the_paper": "This paper investigates using the lottery tickets hypothesis (LTH) strategy for pruning neural network weights for speech recognition. The method first explains the general LTH framework and extensions with transfer learning scenarios. The paper shows the effectiveness of the proposed method in the standard ASR tasks, pre-training with other models, and transfer learning, especially in noisy speech recognition tasks.\n\nSome comments:\n- In the introduction \"End-to-end automatic speech recognition (ASR) (Wang et al., 2019)\": (Wang et al., 2019) is not a representative paper of end-to-end ASR\n- In the introduction, \"For example, the recognition of speech recorded by distant microphones is challenged by acoustic interference such as noise, reverberation and interference speakers (Kinoshita et al., 2020)\": Again not sure (Kinoshita et al., 2020) is a correct citation. I think the following papers are more appropriate\n  - Haeb-Umbach, Reinhold, et al. \"Speech processing for digital home assistants: Combining signal processing with deep-learning techniques.\" IEEE Signal processing magazine 36.6 (2019): 111-124.\n  - Kinoshita, Keisuke, et al. \"The REVERB challenge: A common evaluation framework for dereverberation and recognition of reverberant speech.\" 2013 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics. IEEE, 2013.\n- Currently, data augmentation techniques (artificially contaminate the clean speech with various noisy data) are more standard than transfer learning for noisy speech recognition. Please discuss it.\n- Section 2 \"On the model level, compared to CV models, speech models are mostly based on RNN backbones\": I'm confused about this description. As you mentioned in the previous paragraphs, many speech models now use transformer, conformer, and CNN.\n- Section 2 \"with 10ms shift is 1,000\": this is not exactly true as we usually use downsampling at the beginning of the encoder layer.\n- Section 3, RQ3: Frankly, I don't think this is an essential question for me. This sounds very trivial that the use of pre-trained models would improve the performance.\n- Section 4, What is the main message of Figure 1? I don't find the particular effectiveness of the proposed method from this figure/example.\n- Section 4, Table 3: Why are the other methods, especially KD-based methods, so worse?\n- Section 5 \"A common way to address this issue is through speaker adaptation (Zhao, 1994;\": Again, I don't think (Zhao, 1994) is a representative paper for speaker adaptation. MAP/MLLR adaptation techniques exist in this era, and they were standard techniques in the GMM era.\n- Section 5: I recommend the authors use the real noisy speech data like the CHiME data\n\n\n", "summary_of_the_review": "Although the paper shows significant performance achieving high model compression rates, the algorithmic novelty of this paper is not strong. The application is only ASR and it may not attract so many AI/ML researchers. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "oII44esNI_4", "reviewer": "Reviewer_j7YG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. This paper is clearly written. The idea is simple and straightforward. Experiments are extensively performed, making the paper solid. \n\nCons:\n1. The LTH basically follows the original LTH work and validates the claims from original LTH on ASR models. Though the experimental results are solid, the technical novelty of the paper is a bit thin.\n2. The description of IMP could be elaborated. An algorithm or pseudo-code would make the pipeline more clear. Given that the overall framework does not change much, you should more focus on the ASR-specific details, e.g. how the pruning is different for 3 models? Besides, how does block sparsity work exactly?\n3. In section 2, you mention pruning RNN-based models may be more challenging than pruning CNN-base models. But the method used in this paper is indeed similar to prior works. This claim is a bit inconsistent.\n4. For the random tickets, do you randomly initialize a new $\\theta_0'$ every time, or just use a $\\theta_0'$ different from $\\theta_0$?\n5. As an empirical paper, it is important to give thorough experiments. For example, it's better for you to provide \"existence\" tables on CommonVoice and TED-LIUM as well, as Table 1.\n6. \"Table 4 shows qualititive results of ...\". I think you mean \"quantitative\"?\n7. It would be better if you can show some run time comparison tables.\n8. Given that many modern ASR models now involves quantization (wav2vec2), would this affect how LTH works?", "summary_of_the_paper": "In this paper, authors propose to use lottery ticket hypothesis (LTH) for ASR model pruning. The whole idea generally inherits from the original LTH paper. The model is first trained from scratch. Then you collect all the model weights and sort them. A proportion of the smallest weights are set and fixed to 0. Given this sparsified subnetwork, one further repeats the whole process and prune more parameters. After several rounds, if the pruned subnetwork still gives at least the same performance of the original full network , it is called a \"winning ticket\". The \"winning ticket\" subnetwork should be lightweight, transferrable, and noise-robust.\nThis work applies LTH to 3 models (CNN-RNN CTC, RNN-T, Conformer) on 3 datasets (TED-LIUM, Common Voice, LibriSpeech). The authors empirically show \n1. The winning ticket exists for each method. The subnetwork could be only 4.4% size of the original full model, but still matches the performance.\n2. Compared to other pruning methods, LTH performs best.\n3. LTH with pretrained model performs better than LTH with random initialization.\n\nFurthermore, the paper shows\n1. One can enforce block sparsity for pruning to facilitate chip design.\n2. The winning ticket can transfer to a new dataset and work well.\n3. The pruned network is noise-robust.\n", "summary_of_the_review": "The paper is well-written with a clear and simple idea inherited from prior works. LTH does perform well on ASR and this is a very nice finding. Though the paper claims pruning RNN-based models is different from pruning CNN in CV, in the end the method is still quite similar to the original LTH, which makes the paper's novelty a bit thin. But the work does have a pretty thorough experimental section including all kinds of verification tasks, and proves the effectiveness of LTH on ASR models. The paper would be better if you can give more details on how the smallest weights (or block sparsity) are pruned for different models.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ci3fZsp0ZVN", "reviewer": "Reviewer_etjM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The first investigation of lottery tickets hypothesis on automatic speech recognition. No previous works have studied this task.\n2. Paper is well-structured, well-written, and easy to follow.\n3. They conduct comprehensive analyses on several research questions and the verification on effectiveness of the two components (initial weight, and mask generated from IMP) in iterative weight magnitude pruning (IMP). \n4. They conduct advanced analyses on structured sparsity, transferability, and noise robustness, which echo the story told in the introduction part.\n\nWeaknesses:\n1. The WER of full models using the three model structures (CTC, RNN-T, Transformer) are not state-of-the-art. For example, the SOTA WER on LibriSpeech test-clean for the three models should be nearly (4.0, 3.0, 2.0) respectively according to my experiences, while the numbers the paper is (8.02, 5.90, 2.53). The authors should explain why the WERs have large gap to SOTA.\n2. The method seems straightforward, and the paper just do some analyses to verify the lottery tickets hypothesis on this task. The authors should explain why it is challenging for LTH to generalize to ASR, so the authors need study it separately. Otherwise, I would assume these analysis are somewhat trivial, since it does not provide something special. \n3. It is straight-forward that random masks are worse than masks generated from IMP, and initializing from pre-trained model is better than random initialization. Do we find some something specific and different? \n4. In the structured sparsity analysis, I am not sure only using 1×4 block can cover the practical scenario? I assume that the block should be larger to be GPU-friendly.\n", "summary_of_the_paper": "This paper conducts comprehensive analyses of lottery tickets hypothesis on automatic speech recognition. The authors verified the existence of highly sparse “winning tickets” in ASR task, and analyze its robustness to noise, transferable to other datasets, and supports with structured sparsity. They conduct experiments on different model structures (CTC, RNN-T, Transfomer) and different datasets (LibriSpeech, TED-LIUM, Common Voice) with extensive analyses. ", "summary_of_the_review": "Given the above comments, I think this paper do have some strengths such as the paper is well-written and self-contained, and is the first to study on ASR. However, I have concerns on the soundness of the results since the WER is not close to SOTA and the novelty of the method is limited. Overall, I recommend weak reject of this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0Ez_hAoKvA", "reviewer": "Reviewer_5xPW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a well written paper presenting several interesting results. The novelty might not be high, it's more like an application of the lottery ticket hypothesis to the ASR problem but the conclusions are interesting. \n\nThe main weakness in the paper is that it seems the authors train and test the models (in the robustness study) with the same type of noise.\n\"Namely, we re-trained the winning tickets identified from TED-LIUM, CommonVoice, and LibriSpeech on resynthesized TEDLIUM dataset with background noise. We used DESED dataset (Serizel et al., 2020) as the noise source, which consists of various background sounds...\"\nMy interpretation of the above statement is that the winning tickets are re-trained with noise. A much more interesting experiment would be that noise is added only on the test data, this will demonstrate how robust the models are on out-of-distribution samples. \n\nIt would also be more helpful if the authors presented the performance (Table 6) on different SNR levels, eg., -10 to 10 dB with a step of 5 dB, rather that showing 2 values for N_max which are not really interpretable.\n\nIt is not clear which parameters are masked in each architecture (A1). My understanding is that all parameters, no matter if they belong to the CNNs, RNNs or Conformer, can be masked. This is not clear in the paper and authors also mention (p. 3) that pruning RNNs might be harmful. It would be good to clearly explain this.\n\nThe authors compute the sparsity level at each iteration as s = 1 - 0.8^i where i is the iteration. Is there a reason for using 0.8? It will be interesting to investigate the impact of this parameter as it seems it might be important.\n\nIt would be useful if the actual number of parameters is also reported in Tables 1 and 2. This can be easily calculated by the percentages shown but reporting the actual numbers as well will make the comparison easier to understand.", "summary_of_the_paper": "The paper explores the lottery ticket hypothesis, i.e., the existence of highly sparse subnetworks that can be trained in isolation without\nsacrificing the performance of the full models within the context of ASR. The authors show that it is possible to find subnetworks which contain just a fraction of the number of parameters of the full models without sacrificing performance on ASR. Three different architectures are considered and results reported on 3 datasets. Additional studies investigating the transferability and robustness of the sparse subnetworks are also conducted.", "summary_of_the_review": "This is a paper with limited novelty and some weaknesses as explained above, but it is still very interesting.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a comprehensive analysis of lottery tickets hypothesis (LTH) on automatic speech recognition. The authors verified the existence of highly sparse “winning tickets” in ASR task, and analyzed its robustness to noise, transferable to other datasets, and supported with structured sparsity.

As agreed with the reviewers, the paper is well-written, the justification is thorough, and the finding that LTH does perform well on ASR is interesting. Though the novelty is marginal as it's a direct application of the LTH, this is the first investigation of LTH and brings new insights to the community. 

The decision is mainly based on the thorough justification of the LTH to ASR and new insights it brings to the community.
Decision: Accept (Poster)