Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper applies two approaches for improving the speed of ASR models: learning to identify salient tokens and skip those that are identified not salient, and weight pruning. The results show some improvements on the SUPERB set and LibriSpeech-100h. The authors also benchmark the inference speed on Pixel 3 phone and demonstrate good speedup with the technique.
Two questions: 
(1) in section 4.5, when comparing different skipping methods, are the models also fine-tuned with those skipping methods? Or are they only applied during inference?
(2) On the SUPERB set, why would the quality be better than the uncompressed model?

A more recent related work on self-supervised learning for speech is missing:
Self-supervised learning with random-projection quantizer for speech recognition. ICML 2022
strength_and_weaknesses: Strength: the results in the paper show significant improvement compared to the previous work.
Weakness:
(1) The comparisons with the previous approach are limited, making it less convincing.
(2) The comparisons with other pruning algorithms are also lacking.
(3) SALAD is only slightly better than a simple uniform skipping.
(4) The proposed pruning algorithm is a minor extension of PARP, making its novelty limited.

clarity,_quality,_novelty_and_reproducibility: The article is clear. The contribution mostly is in the performance improvement.
summary_of_the_review: The proposed approach shows good improvement compared with PARP, but the empirical studies are a bit limited. There should also be more comparisons with other algorithms. The technical novelty is generally limited.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes two methods for efficient ASR modeling, the token/frame skipping and structured pruning. The token skipping is achieved by applying a binary classifier to detect whether a token is salient. The structured pruning is an iterative process of sparsifying, fine-tuning and recovering, until reaching a certain sparsity level. Experimental results demonstrates the effectiveness in terms of FLOPs savings against WER increases, in comparison with other popular pruning methods.

strength_and_weaknesses: Pros:
1. The paper is clearly written and easy to follow. The illustrative figures and examples are helpful to understand the ideas.
2. Examples are conducted on different backbone models including wav2vec2 and data2vec, with different sizes.
3. The ablation study compared with random removing frames or adaptive removing, are very useful to establish a sense of how the proposed method performs compared with baselines.
4. The pruning performance is impressive compared with other methods. I guess it is due to the token skipping

Cons:
1. The idea of both methods are incremental, though combining them and show they works is the value of this paper.
2. In figure 3 (a), why the S6-DAMON does not have values below 40% FLOPs? Is it that the WER is worse than the others?
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written. Novelty is not enough. No code or trained-model released for reproduction.
summary_of_the_review: Overall, though the papers' core ideas are not novel, the combination and comprehensive experimental evaluation show that it is working well. One concern is that no source code nor trained model provided for reproduction.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper tackles the problem of on-device ASR self-supervised learning. In this scenario, there is a large unlabeled dataset is available for unsupervised pre-training, and a much smaller supervised set for fine-tuning. Previous models such as HuBERT, data2vec use a large transformer model for the feature pre-training. Such a transformer is resource hungry for the scenario of running on an mobile phone or an embedded device.

This work proposes 3 different components. First, there is a method to prune the input tokens. The paper uses a teacher ASR model to train a small classifier which predicts which input token is salient. Examples of salient tokens are the first of the repeating letters, or the first of several silences.

Second proposed component is the method to sparcify the model. The paper builds upon the PARP and proposes an iterative procedure of sparcification and finetuning steps.

Finally, the third proposed model combines both previous models to use them for semi-supervised distillation from a teacher model.

The paper reports the results on Librispeech and Superb corpora and then evaluates the latency on a real device.
strength_and_weaknesses: Strengths:
  - The paper tackles a challenging problem.
  - The proposed models are sound. The design makes sense and motivated well
  - The paper provides evaluations on several datasets: Librispeech and Suberb
  - The paper evaluates the latency on a real device

Weaknesses:
  - There is some lack of focus in this paper: there are three models introduced. Each one is complex enough to have a small paper. This leads to a very condensed writing and each model is not described well.
  - The results are hard to compare to previous works. For example, the Fig. 3 models are trained in a different setup than wave2vec2 or HuBERT, therefore not comparable. The PARP paper's setup is also a bit different than presented here.
clarity,_quality,_novelty_and_reproducibility: # Clarity

The paper is mostly clearly written. I found that the section 3.4 could be extended with more details about the center model of the paper. For example, I did not understand how the argmax is computed (examples are sampled or really summed over the whole set?). There are a number of small issues with writing such as repeating expressions. I also found it very distracting to use such abbreviations for model names -- this is just over the top.

# Quality

Generally, the paper has above average quality. The methods are sound and and there is an attempt to experimentally test all the claims. One of the highlights of the paper I liked, and I don't see frequently, is testing on a real device. As mentioned before, the paper lacks focus by introducing several disconnected methods. Then, the presentation of the results needs alignment with the prior works as the faulty baseline might diminish the support for the claims.

# Novelty

Each proposed method has average novelty, but the combination of them seems to be novel.

# Reproducibility

The paper introduces several methods with many hyperparameters. The hyperparameters are given in the paper, but it might be challenging to reproduce all the implementation detals. The paper does not provide error bars and doesn't test the models for stability and robustness.

# Other

There is a recurring mistake in the paper: difference in percentages is not a percentage https://en.wikipedia.org/wiki/Percentage_point
summary_of_the_review: Given the previous assessment, now I recommend reject.

But this paper has a potential, and here are the ways to improve it:
  - Align the experimental results to the prior literature: HuBERT, data2vec, wav2vec, PARP. If it is not possible, provide a disclaimer (for example, that the baseline was trained on a larger corpus and therefore had better performance).
  - Tie the different components together.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes two improvements, fine-tuning self-supervised models for automatic speech recognition. One is an additional classifier to remove frames, and the other is an modified iterative fine-tuning procedure. No hypotheses are given in the paper. Experiments show that the proposed approach can achieve better word error rates at lower sparsity level.

Two reviewers comment that the novelty is limited. Two reviewers mention that there are no comparison to prior work. One reviewer even points out that the improvements are limited.

Overall, the reviewers are lukewarm about the paper. Though all reviewers think the paper is clear, I think the paper would benefit from better writing. For example, there are too many subsubsections, and the paper feels fragmented. The related work is a laundry list of papers, and does not provide contrast between the paper and the prior work. The main message is buried in the experimental results. One reviewer mention a lack of focus, and that's the typical sign that the writing can be improved. The paper would be stronger if the writing can be improved.


justification_for_why_not_higher_score: All reviewers do not think the paper has sufficient novelty and think the experimental design can be better.
Decision: Reject