Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: **Summary**
This paper considers continuous self-training for ASR. This paper build on the observation that a previous method slimIPL performance degrades as the number of pretraining step M increase. The authors hypothesize that pretraining would cause overfitting when the supervised data is limited. To address this mentioned concerns, the authors propose self-training from the beginning of process. The authors introduces a series of tricks to improve the robustness and convergence of slimIPL.
- regenerating PL when returning a sample to cache
- dynamically compute the returning probability with Levenshtein edit-distance
- sampling PL (with temperature scheduling) instead of using 1-best

The optimal recipe achieve similar result with the original paper on 100h dataset but much better on 10h dataset.
strength_and_weaknesses: **Strength**
- The experiment design is very well motivated and executed.
- The resulting method is both simple in recipe (1 stage training), also achieve significant WER reduction on 10h dataset.
- The paper writing is extremely clear.

**Wakenesses**
- Limited novelty.
- Absence of hyperparameter table in the appendix.
clarity,_quality,_novelty_and_reproducibility: **Clarity**
Extremely clear.

**Quality**
Very high.

**Novelty**
OK.

**Reproducibility**
Should be reproducible.
summary_of_the_review: accept
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a method to self-train ASR systems directly, without requiring an initial pre-training stage, which is especially useful in low-resource setups. Similarly to an existing method, slimIPL, the proposed approach generates pseudo-labels as training progresses and maintains a cache of pseudo-labels, regularly updated. The main differences are that the proposed method does not require a first pre-training step, that the pseudo-labels are updated when they are used before putting them back in the cache, and that the probability of removing a pseudo-label from the cache is computed from the difference between the older and updated labels. Additionally, the pseudo-labels are sampled from the predicted token distributions, whereas slimIPL generates 1-best labels. 

The method is evaluated on LibriSpeech with two simulated setups corresponding to 10 and 100h hours of labeled data. The impact of the different hyper-parameters and proposed techniques and tricks are evaluated and the results show a more stable training and improved word error rates.
strength_and_weaknesses: Strength: The idea is interesting and the results are good. Self-training from the start is indeed useful. The proposed improvements of slimIPL are relevant and well motivated and analysed.

Weaknesses: The proposed method is only evaluated on setups simulated from LibriSpeech. There is no comparison with training with all the labeled data of LibriSpeech. Moreover, the 10h scenario where the proposed approach beats other methods does not look like the one usually explored for this problem: this could be motivated more in the text. The approach described in this paper is not as good as prior work on usual 10h used.  It would also have been interesting to see some results for actual low-resource scenarios.
clarity,_quality,_novelty_and_reproducibility: The paper is clear, easy to follow and to understand. The proposed method is well described and illustrated. The evaluation is good but limited to one dataset only.

The analysis of the evolution of the Levenshtein distance between successive pseudo-label generation is interesting, as is the further analysis in the appendix. 

The proposed method is mostly improvements of existing techniques, but they are relevant and address a relevant problem, the choices are well motivated and seem novel enough. 

The paper looks clear enough for reproduction, and the authors state that the code will be made available.

Minor question:
  Do tables 5.1 and 5.2 contain results using only one of the methods (only cache strategy / only alignment sampling)? I think this could be clearer in the text.
summary_of_the_review: The paper is quite good, nice to read and well explained. The motivation for this idea is interesting and solves a real issue. I think this is worth sharing with the community, but I would like to see more results for different scenarios than just LibriSpeech.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper extends previous work on pseudo labeling for ASR (slimIPL to be specific) and achieve similar performance while much stable training process without even using the initial supervised training.
Generally, pseudo-labeling has been proved to be very valuable to industry applications of ASR because usually industries have labeled data for bootsrapping, and also have huge amount of unlabelled data for generation of pseudo labels. So the work in this study could be of great value to real applications.
strength_and_weaknesses: Strength:
1. Analyzed problems of previous methods (training unstable, easy to diverge) and proposed corresponding solutions (PL evolution and alignment sampling). The solutions are proved to be effective through experiments on Librispeech dataset.

Weakness
1. The writing style is kind of confusing: could be better at explaining the main process of the proposed method and the difference with and improvements over slimIPL.
2. The results in table 5 and 6 use different training hyperparameters compared to table 7. Not sure why this inconsistency exist but it could be clearer comparison under the same exp settings.
3. This line of research (PL for ASR) all uses Librispeech for evaluation. I think this limits the potential value for real scenario. The most suitable scenario of this technique is where we have some labeled data to train an initial usable model, then we utilize unlabeled data collected before hand or even periodically collect unlabeled data from the product, to which the ASR system is applied. In this case, the problem could be much more complex, e.g. the distribution of the initial labeled data could be different from the unlabeled data collected later, or even the unlabeled data have different distribution at different time steps. The conclusion drew from this paper (or other PL related studies) could not be suitable for this scenario. I believe if the authors add exps on more real data, the conclusion could be more convincing and thus more contribution to the community (like studies in https://www.isca-speech.org/archive/interspeech_2022/baby22_interspeech.html, https://arxiv.org/abs/2207.09078)
4. The alignment sampling methods assume a CTC decoder, which may not be applicable to RNN-T and LAS ASR models.
clarity,_quality,_novelty_and_reproducibility: As mentioned in the strength and weakness.
I believe the reproducibility is good.
summary_of_the_review: Overall, I recognize the value of value of this paper as further improvement in the research about PL for ASR, though marginal. I hope more (realistic) evaluations could be done to bring more benefits to the community.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes novel changes to an existing self training algorithm for ASR such that performance is improved without a pretraining step. Specifically, the proposed model generates pseudo labels (PL) for unlabelled data from the very start of supervised training and uses it to augment the dataset. This is done by dynamically evolving a cache of data by using a simple yet effective filtering criteria for unlabelled examples. Furthermore, to avoid instability in the proposed PL generation from the start an alignment sampling is used which samples tokens from the predicted distribution instead of an argmax. Results show improvement over previous work.

strength_and_weaknesses: Strengths:

1) The overall presentation and flow of ideas is good. The paper does a good job in setting up the motivation for the work by presenting some early results.
2) Extensive ablation studies over each component of the proposed methodology helps the paper.
3) Results are compared with comprehensive baselines.
4) Details about the model architecture and other results included for reproducibility.

Weaknesses:

1) The major part of this paper builds over the previously proposed slimIPL. Yet, not enough explanation is given about slimIPL in the main text. This might be inconvenient for a reader not conversant in the algorithm. Having a full algorithm in the main text will help the paper.
2) The algorithm for slimIPL (appendix A) confused me. In the 3rd step, when the cache is being initialized, why is there an update step using (x,y) in L? Hasn’t that already been done prior to generating the cache? Or is this a typo?
3) It is not very clear why alignment sampling should work. The authors say that it provides effective stabilization by adding noise to the targets. But the targets are noisy anyway during the start of training as the TER is close to 100% initially. So why would sampling produce targets that are any different? There is not enough explanation for this.
4) All results are shown on CTC based models. Do these techniques translate to encoder-decoder based ASR models like LAS and RNN-T which are very popular? The authors do not comment on this.
5) In the final results, compared to w2v 2.0 results, the proposed techniques work better only on dev-other and test-other but not the clean subsets using 10h of speech. Why should the proposed method be chosen over w2v2.0 ?
6) Minor issues: The caption for figure 1 can be more informative. What does “CAPE” and “sinpos” mean in table 7?

clarity,_quality,_novelty_and_reproducibility: 1) The paper is clear overall for a reader who knows about the field. But for a new reader, it can be made clearer by having an algorithm for the proposed methods.
2) The paper doesn’t have significant quality issues.
3) The paper has moderate novelty to the best of my knowledge.
4) The authors provide details for model architecture and are committed to releasing their code.

summary_of_the_review: This paper contributes to the field of self training for ASR models. The overall presentation of ideas is good and the proposed techniques are simple yet effective. The authors conduct extensive ablation studies to back their claims. However, the paper could benefit by explaining some intuition behind the alignment sampling criteria. The fact that the authors do not comment on non-CTC based models and that w2v2.0 still works better on 2 out of 4 test sets, does weaken the paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: 1. This paper presents a continuous self-training method that can train an ASR model from the start without any supervised pre-training. The experimental results have confirmed the effectiveness of the proposed approach. 
2. The present work has analyzed problems of previous methods (training unstable, easy to diverge) and proposed corresponding. The work in this study could be of great value to real applications.
3. Similar ideas have been proposed [1]. Thus, the proposed algorithm is not completely new. 

[1] Likhomanenko, Tatiana, et al. "slimipl: Language-model-free iterative pseudo-labeling." arXiv preprint arXiv:2010.11524 (2020).
justification_for_why_not_higher_score: 1. Similar ideas have been proposed [1]. Thus, the proposed algorithm is not completely new. 

[1] Likhomanenko, Tatiana, et al. "slimipl: Language-model-free iterative pseudo-labeling." arXiv preprint arXiv:2010.11524 (2020).
justification_for_why_not_lower_score: 1. This paper presents a continuous self-training method that can train an ASR model from the start without any supervised pre-training. The experimental results have confirmed the effectiveness of the proposed approach. 
2. The present work has analyzed problems of previous methods (training unstable, easy to diverge) and proposed corresponding. The work in this study could be of great value to real applications.
Decision: Accept: poster