Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, authors propose to use lottery ticket hypothesis (LTH) for ASR model pruning. The whole idea generally inherits from the original LTH paper. The model is first trained from scratch. Then you collect all the model weights and sort them. A proportion of the smallest weights are set and fixed to 0. Given this sparsified subnetwork, one further repeats the whole process and prune more parameters. After several rounds, if the pruned subnetwork still gives at least the same performance of the original full network , it is called a "winning ticket". The "winning ticket" subnetwork should be lightweight, transferrable, and noise-robust.
This work applies LTH to 3 models (CNN-RNN CTC, RNN-T, Conformer) on 3 datasets (TED-LIUM, Common Voice, LibriSpeech). The authors empirically show 
1. The winning ticket exists for each method. The subnetwork could be only 4.4% size of the original full model, but still matches the performance.
2. Compared to other pruning methods, LTH performs best.
3. LTH with pretrained model performs better than LTH with random initialization.

Furthermore, the paper shows
1. One can enforce block sparsity for pruning to facilitate chip design.
2. The winning ticket can transfer to a new dataset and work well.
3. The pruned network is noise-robust.

main_review: Pros:
1. This paper is clearly written. The idea is simple and straightforward. Experiments are extensively performed, making the paper solid. 

Cons:
1. The LTH basically follows the original LTH work and validates the claims from original LTH on ASR models. Though the experimental results are solid, the technical novelty of the paper is a bit thin.
2. The description of IMP could be elaborated. An algorithm or pseudo-code would make the pipeline more clear. Given that the overall framework does not change much, you should more focus on the ASR-specific details, e.g. how the pruning is different for 3 models? Besides, how does block sparsity work exactly?
3. In section 2, you mention pruning RNN-based models may be more challenging than pruning CNN-base models. But the method used in this paper is indeed similar to prior works. This claim is a bit inconsistent.
4. For the random tickets, do you randomly initialize a new $\theta_0'$ every time, or just use a $\theta_0'$ different from $\theta_0$?
5. As an empirical paper, it is important to give thorough experiments. For example, it's better for you to provide "existence" tables on CommonVoice and TED-LIUM as well, as Table 1.
6. "Table 4 shows qualititive results of ...". I think you mean "quantitative"?
7. It would be better if you can show some run time comparison tables.
8. Given that many modern ASR models now involves quantization (wav2vec2), would this affect how LTH works?
summary_of_the_review: The paper is well-written with a clear and simple idea inherited from prior works. LTH does perform well on ASR and this is a very nice finding. Though the paper claims pruning RNN-based models is different from pruning CNN in CV, in the end the method is still quite similar to the original LTH, which makes the paper's novelty a bit thin. But the work does have a pretty thorough experimental section including all kinds of verification tasks, and proves the effectiveness of LTH on ASR models. The paper would be better if you can give more details on how the smallest weights (or block sparsity) are pruned for different models.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper investigates using the lottery tickets hypothesis (LTH) strategy for pruning neural network weights for speech recognition. The method first explains the general LTH framework and extensions with transfer learning scenarios. The paper shows the effectiveness of the proposed method in the standard ASR tasks, pre-training with other models, and transfer learning, especially in noisy speech recognition tasks.

Some comments:
- In the introduction "End-to-end automatic speech recognition (ASR) (Wang et al., 2019)": (Wang et al., 2019) is not a representative paper of end-to-end ASR
- In the introduction, "For example, the recognition of speech recorded by distant microphones is challenged by acoustic interference such as noise, reverberation and interference speakers (Kinoshita et al., 2020)": Again not sure (Kinoshita et al., 2020) is a correct citation. I think the following papers are more appropriate
  - Haeb-Umbach, Reinhold, et al. "Speech processing for digital home assistants: Combining signal processing with deep-learning techniques." IEEE Signal processing magazine 36.6 (2019): 111-124.
  - Kinoshita, Keisuke, et al. "The REVERB challenge: A common evaluation framework for dereverberation and recognition of reverberant speech." 2013 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics. IEEE, 2013.
- Currently, data augmentation techniques (artificially contaminate the clean speech with various noisy data) are more standard than transfer learning for noisy speech recognition. Please discuss it.
- Section 2 "On the model level, compared to CV models, speech models are mostly based on RNN backbones": I'm confused about this description. As you mentioned in the previous paragraphs, many speech models now use transformer, conformer, and CNN.
- Section 2 "with 10ms shift is 1,000": this is not exactly true as we usually use downsampling at the beginning of the encoder layer.
- Section 3, RQ3: Frankly, I don't think this is an essential question for me. This sounds very trivial that the use of pre-trained models would improve the performance.
- Section 4, What is the main message of Figure 1? I don't find the particular effectiveness of the proposed method from this figure/example.
- Section 4, Table 3: Why are the other methods, especially KD-based methods, so worse?
- Section 5 "A common way to address this issue is through speaker adaptation (Zhao, 1994;": Again, I don't think (Zhao, 1994) is a representative paper for speaker adaptation. MAP/MLLR adaptation techniques exist in this era, and they were standard techniques in the GMM era.
- Section 5: I recommend the authors use the real noisy speech data like the CHiME data



main_review: strengths:
- straightforward application of the LTH strategy to speech recognition
- showing the significant performance (very high compression without losing the performance so much). This is especially important for on-device scenarios.
- extensions for transfer learning scenarios

weaknesses:
- the novelty of the method itself is incremental from the original LTH and related work.
- some references are not appropriate —> (correction) the reference part was improved through the discussions
summary_of_the_review: Although the paper shows significant performance achieving high model compression rates, the algorithmic novelty of this paper is not strong. The application is only ASR and it may not attract so many AI/ML researchers.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper explores the lottery ticket hypothesis, i.e., the existence of highly sparse subnetworks that can be trained in isolation without
sacrificing the performance of the full models within the context of ASR. The authors show that it is possible to find subnetworks which contain just a fraction of the number of parameters of the full models without sacrificing performance on ASR. Three different architectures are considered and results reported on 3 datasets. Additional studies investigating the transferability and robustness of the sparse subnetworks are also conducted.
main_review: This is a well written paper presenting several interesting results. The novelty might not be high, it's more like an application of the lottery ticket hypothesis to the ASR problem but the conclusions are interesting. 

The main weakness in the paper is that it seems the authors train and test the models (in the robustness study) with the same type of noise.
"Namely, we re-trained the winning tickets identified from TED-LIUM, CommonVoice, and LibriSpeech on resynthesized TEDLIUM dataset with background noise. We used DESED dataset (Serizel et al., 2020) as the noise source, which consists of various background sounds..."
My interpretation of the above statement is that the winning tickets are re-trained with noise. A much more interesting experiment would be that noise is added only on the test data, this will demonstrate how robust the models are on out-of-distribution samples. 

It would also be more helpful if the authors presented the performance (Table 6) on different SNR levels, eg., -10 to 10 dB with a step of 5 dB, rather that showing 2 values for N_max which are not really interpretable.

It is not clear which parameters are masked in each architecture (A1). My understanding is that all parameters, no matter if they belong to the CNNs, RNNs or Conformer, can be masked. This is not clear in the paper and authors also mention (p. 3) that pruning RNNs might be harmful. It would be good to clearly explain this.

The authors compute the sparsity level at each iteration as s = 1 - 0.8^i where i is the iteration. Is there a reason for using 0.8? It will be interesting to investigate the impact of this parameter as it seems it might be important.

It would be useful if the actual number of parameters is also reported in Tables 1 and 2. This can be easily calculated by the percentages shown but reporting the actual numbers as well will make the comparison easier to understand.
summary_of_the_review: This is a paper with limited novelty and some weaknesses as explained above, but it is still very interesting.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper conducts comprehensive analyses of lottery tickets hypothesis on automatic speech recognition. The authors verified the existence of highly sparse “winning tickets” in ASR task, and analyze its robustness to noise, transferable to other datasets, and supports with structured sparsity. They conduct experiments on different model structures (CTC, RNN-T, Transfomer) and different datasets (LibriSpeech, TED-LIUM, Common Voice) with extensive analyses. 
main_review: Strengths:
1. The first investigation of lottery tickets hypothesis on automatic speech recognition. No previous works have studied this task.
2. Paper is well-structured, well-written, and easy to follow.
3. They conduct comprehensive analyses on several research questions and the verification on effectiveness of the two components (initial weight, and mask generated from IMP) in iterative weight magnitude pruning (IMP). 
4. They conduct advanced analyses on structured sparsity, transferability, and noise robustness, which echo the story told in the introduction part.

Weaknesses:
1. The WER of full models using the three model structures (CTC, RNN-T, Transformer) are not state-of-the-art. For example, the SOTA WER on LibriSpeech test-clean for the three models should be nearly (4.0, 3.0, 2.0) respectively according to my experiences, while the numbers the paper is (8.02, 5.90, 2.53). The authors should explain why the WERs have large gap to SOTA.
2. The method seems straightforward, and the paper just do some analyses to verify the lottery tickets hypothesis on this task. The authors should explain why it is challenging for LTH to generalize to ASR, so the authors need study it separately. Otherwise, I would assume these analysis are somewhat trivial, since it does not provide something special. 
3. It is straight-forward that random masks are worse than masks generated from IMP, and initializing from pre-trained model is better than random initialization. Do we find some something specific and different? 
4. In the structured sparsity analysis, I am not sure only using 1×4 block can cover the practical scenario? I assume that the block should be larger to be GPU-friendly.

summary_of_the_review: Given the above comments, I think this paper do have some strengths such as the paper is well-written and self-contained, and is the first to study on ASR. However, I have concerns on the soundness of the results since the WER is not close to SOTA and the novelty of the method is limited. Overall, I recommend weak reject of this paper.
review_title: null

Meta-review: The paper presents a comprehensive analysis of lottery tickets hypothesis (LTH) on automatic speech recognition. The authors verified the existence of highly sparse “winning tickets” in ASR task, and analyzed its robustness to noise, transferable to other datasets, and supported with structured sparsity.

As agreed with the reviewers, the paper is well-written, the justification is thorough, and the finding that LTH does perform well on ASR is interesting. Though the novelty is marginal as it's a direct application of the LTH, this is the first investigation of LTH and brings new insights to the community. 

The decision is mainly based on the thorough justification of the LTH to ASR and new insights it brings to the community.
Decision: Accept (Poster)