Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper describes a self supervision training approach to pretrain speech encoders.  This is related to recent work on wav2vec 2.0, and SimCLR.  The contrastive loss is similar to wav2vec 2.0, while the invariance to perturbation (i.e. augmentation) is akin to SimCLR.
Performance is competitive with wav2vec 2 on Librispeech.
main_review: The goal of perturbation or augmentation invariance is well motivated in speech recognition, and also ML/ representation learning more broadly.  The Librispeech 960 numbers are quite strong.  

* The results using synthetic noisy data are less convincing than naturally occurring noise.  This is especially important since the synthetic noise is used during training as well.
* In figure 1 it appears that the positional padding is applied equally to both the input and output of the teacher.  This is not as clear from the description in Section 3.1 If this is the case it would be helpful to expand on the description.
* Section 3.3: SpecAugment typically masks both time and frequency bands.  It is not clear if this is done here, or if the masking is performed on either time or frequency.  Assuming both time and frequency are masked, is the area masked by both time and frequency replaced with zeros or gaussian noise?
* Section 3.5: What is the rationale for using CTC ASR decoding rather than say an attention decoder, or RNN-T model?
* Section 4.2 it would be useful to have more insight into why SPIRAL trains faster than wav2vec 2.0.  Is this due to implementation, or is there something fundamental to these algorithms that makes SPIRAL faster?
* Section 4.2: It might be useful to compare these results to SpeechStew 
* Section 5.3.1: it is interesting that other approaches are more robust to perturbing both teacher and student inputs, while SPIRAL requires clean teacher inputs. More thorough discussion of this would be helpful, though space is tight.

Shorter notes: 
* Section 2: the description of Liang et al (2018) is notably brief compared to others.  The comparison of perturbation invariance in ASR training seems like a worthwhile comparison to invariance during pretraining.
* Table 3: only row 2 includes 3 significant figures, the rest includes 2. it would be better to be consistent here.  (Same comment in tables 4. and 5)
* Table 5: it would be better to organize the columns in by increasing or decreasing SNR.
* Typo in Section 5.3.2 where "When the predictor is removed, We observe" the "We" shouldn't be capitalized.
summary_of_the_review: This is a compelling paper. it's a well motivated and technically sound perspective on self-supervised training.  The performance on Librispeech 960 is quite strong.  The performance in noise conditions is strong, but would be more convincing if shown on more actual rather than synthetic noise.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a new speech pre-training approach named SPIRAL.  The proposed method is trained by learning representation denoising of perturbed data with the teacher-student framework in a self-supervised manner.  The motivation of the method is to learn the representation that is invariant to perturbation so that the learnt representation is a high-level one (e.g. carrying content information) that can enhance the downstream speech applications.  Compared to the state-of-the-art speech pre-training method wav2vec 2.0, the proposed method can achieve competitive or better results but with a significant reduction of the training cost.  The proposed method is also able to deal with the noise-robustness problem.
main_review: The paper is well written with the following strengths.
(1) The proposed method is self-supervised learning with the teacher-student framework that is an extension to Mean Teacher (MT) and Boostrap Your Own Latent (BYOL).
(2) The proposed method is a novel one that aims to learn the representation denoising of perturbed data with the teacher-student framework.  In addition, the proposed method can also be combined with multi-condition training to improve the noise-robustness.
(3) The motivation of the method sounds quite reasonable which aims to learn the representation that is invariant to perturbation so that the learnt representation is a high-level one (e.g. carrying content information) to enhance the downstream speech applications.
(4) An in-utterance contrastive loss (proposed by Chopra et al. (2005) is adopted to avoid the model collapse problem. 
(5) Position randomization technique is further introduced to prevent the positional collapse problem.
(6) A gradual down-sampling strategy has been adopted to train the SPIRAL model to reduce the computation cost.
(7) Plenty of experiments have been conducted to evaluate the performance of the proposed method. 
(8) The proposed method can achieve competitive or better results than wav2vec 2.0 whereas with significantly less training cost.


summary_of_the_review: Based on the above main review (especially the strengths of the paper), although some of the ideas have been borrowed from previous work (e.g. MT, BYOL, contrastive loss, etc), the paper has proposed extensions to these work by considering the sequential applications in speech processing.  And the relations with the previous work have been clearly discussed in section 2.  Hence, I think the paper could be accepted for publication on ICLR.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presents a novel approach to self-supervised speech representation learning, which promises to be simpler than existing methods such as wav2vec 2.0. The approach is inspired by the BYOL approach from CV, and is shown to be indeed largely as effective as wav2vec 2.0, while being significantly more efficient during training.
main_review: Strengths:

The paper is overall well written, and presents a good set of experimental results. The baselines are strong (for what I can tell), the approach is well motivated and explained well. The authors promise to release code upon acceptance of the paper, which should allow readers to verify results and build on top of it. Pre-trained self-supervised audio feature extractors have the potential to improve speech recognition in many domains (e.g. low resource multi-lingual speech recognition) and reducing compute is a critical step in that direction.

Weaknesses:

Some of the experimental results are a bit early? I would like to see results with LM rescoring after the hyper parameters have been optimized, so that results are more comparable (and hopefully consistent)? Similarly with mixed precision training. Would it be possible to get rid of the convolutional layers and build a model that is based entirely on self-attention? This should be even more efficient on GPU?

Update: the additional tests on Chime data, and the updated decoding results address these concerns. I am updating my assessment.
summary_of_the_review: Making self-supervised speech representations easier to train is a significant contribution, and this paper presents a viable approach to doing so. Some results feel a bit preliminary, but the paper is well written and the authors may have updated results but the time of the conference, and they will release code (plus presumably models?). My recommendation is thus for accept.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper describes teacher-student self-supervised training, with a denoising advantage by perturbing the student. It elaborates on techniques to avoid collapse and compares results with wav2vec 2.0.
main_review: The paper describes teacher-student self-supervised training. The teacher generates a representation on clean data. The student matches  the teacher's representation with a perturbed version of the utterance. In-utterance contrastive loss is used to avoid learning a trivial representation. Positional collapse is avoided by random padding before/after the utterance fed to the teacher. The teacher is updated as the moving average of student checkpoints. Results are presented for low-resource and high-resource settings using librispeech and librilight.

The strengths are:
- The paper has an interesting premise, combining ideas from noisy-student training and self-supervised training.
- The results are somewhat comparable to wav2vec 2.0, but at lower training cost.
- The method is robust to unseen noise situations.
- As far as I understand, it does not require separately pre-training the teacher, unlike noisy student training. This could be clarified however.

The weaknesses include:
- The results are better or on par with wav2vec 2.0 in some cases, but not on the whole. The authors mention that the settings are not fully tuned -- it would be interesting to know what the best results are after tuning.
- While there are some ablation studies, some unanswered questions remain. In particular, I am curious how much is gained from additive noise perturbation and from positional randomization.
- In looking at the noisy test results (table 5), there is no additive noise in either pre-training or fine-tuning for wav2vec 2.0. Hence this comparison seems incomplete.

Other questions and notes:
- Related work could include contrastive semi-supervised learning work from Facebook, as another way to combine these two aspects.
- How is the teacher initialized?
- In table 2, I assume training step refers to pre-training steps. How is the optimal number of pre-training steps determined for each method?
- What perturbation settings are included in the wav2vec setup?
- Table 3: noisy student unlabeled data = "LS-860" -- is this a typo?
- Page 7: the-clean -> test-clean (typo)
summary_of_the_review: The paper has an interesting premise and some promising experiments to justify it. On the whole, the method is somewhat comparable to wav2vec 2.0, although it falls short in several cases. This can be made clearer in the abstract and introduction, while highlighting the reduced training cost. I am not convinced that the comparison to wav2vec 2.0 is complete in terms of denoising uses, as there is no discussion of specaug and additive noise in wav2vec 2.0 during pre-training or fine-tuning. Also, some additional ablation studies to understand the benefit of additive noise and positional randomization would strengthen the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces SPIRAL, a new method for self-supervised pre-training for speech. SPIRAL is based on the teacher-student framework similar to Mean Teacher (Tarvainen and Valpola, 2017) and BYOL (Grill et al., 2020) where the teacher’s weights are updated as a moving average of the student’s weights, but makes additional modifications for sequence tasks like speech:

* An in-utterance contrastive loss is used as the pre-training objective.
* Position randomization of the teacher’s input is used to avoid representation collapse.
* Ablation experiments are done to show that the predictor (which was essential in earlier works like BYOL and SimSiam to avoid collapse) can be replaced with a convolutional projection head without performance degradation.

Empirically, the main contributions of the paper are:
1. Achieving similar/better WER on LibriSpeech compared to wav2vec 2.0 with 35% of the training cost; and
2. Incorporating multi-condition training, which has been used in supervised training in the past, for noise-robust ASR.

main_review: Strengths:
1. While the paper adapts the teacher-student self-supervised pre-training framework that has been studied for image representation learning (Grill et al., Chen & He, Tian et al.), the modifications for sequential learning --- in-utterance contrastive loss, position randomization, and convolutional subsampling --- are essential for speech.
2. The authors perform ablations to show the relevance of the projection head vis-a-vis the predictor, and demonstrate they are complementary. This result raises important questions about conclusions drawn about this framework in previous works (like Tian et al. 2021), and their applicability to sequence representation learning.
3. Experimental setup (e.g. hyperparameters) is described in detail. SPIRAL obtains strong WER performance on LibriSpeech with a fraction of the training of wav2vec 2.0. Multi-condition training is shown to be effective at pre-training (versus only at fine-tuning)  to improve noise robustness (~10% relative WER reduction) while maintaining performance on clean speech.
4. The authors have mentioned that they will release the code at the time of publication. This would be useful for the community to extend this direction of research.
5. The paper is clear and easy-to-follow, with sufficient discussion of related work.

Weaknesses:
1. While the proposed model does well on clean LibriSpeech, performance under noisy conditions may be concerning (Section 5.2). In Table 5, when the synthetic noise at test time is matched with training noise (both in noise type and SNR), the WERs are good; but mismatched SNR significantly degrades WER (8.0% to 26.1% for “test-clean”). This suggests that the model may be overfitting to the range of SNRs during training. Second, there are no evaluations on mismatched noise types during train and test, so it is hard to predict the model’s generalizability to other types of noise. Furthermore, the absence of evaluations on real noisy data (such as CHiME-4) raises some questions about the benefits of multi-condition training. These questions are of particular interest since the model is named “perturbation invariant”.
2. There have been several recent works analyzing representation collapse of non-contrastive SSL models (e.g. Tian et al., 2021) which suggest that a predictor on the student branch, and weight decay during training are essential to prevent collapse. The authors mention briefly in Section 2 that the predictor was not sufficient and they had to use the in-utterance contrastive loss with position randomization. Can they suggest reasons why this might be the case?
3. The ablation in Section 5.3.2  indicates that performance degrades significantly when perturbations are applied to the teacher’s input. This again deviates from standard practice in image representation learning where augmentation is applied on the inputs for both the teacher and the student, and (as astutely observed by the authors) is closer in principle to standard self-training. Could the authors suggest why input noise is harmful but computation noise (dropout) works by, for instance, elucidating the link to Gal and Ghahremani (2016)?

Other comments/questions:
1. For multi-condition pre-training, is the noisy input used for both the student and the teacher? If yes, what would happen if the clean input is used for the teacher?
2. For LibriSpeech evaluation under the “low resource” setting, perhaps it would be fairer (although not comparable to Baevski et al., 2020) to exclude the train-clean-100 subset during pre-training, as done in Park et al., 2020.
3. In the first paragraph of Page 2, the authors comment that “SPIRAL also allows to combine with multi-condition training.” This is slightly misleading as there is no inherent limitation in the other models (wav2vec 2.0, HuBERT) mentioned in the previous line that would prevent multi-condition training with those models.
4. Have the authors tried learning from raw waveforms instead of log-mel filterbanks?

Some typos:
* Section 1, para 3: “...learning representation denoising of perturbed data…” →  “...learning denoising representation of perturbed data…”
* Section 3.2, para 1: “predicotr” → “predictor”
* Section 3.5, para 3: “...due limited receptive field.” → “...due to limited receptive field.” 


summary_of_the_review: While there are some clear limitations in the empirical sense, particularly in the claim that the model learns denoised high-level representations, the paper advances self-supervised learning for speech tasks by adapting the popular teacher-student framework popular in image representation learning. The model achieves WERs similar/better than the popular wav2vec 2.0 architecture while reducing training time and model size. The ablation experiments raise questions about whether the modeling/training strategies deemed essential in that modality are also needed for sequential tasks, although the authors have not addressed these questions directly in this work. Overall, this work (and the promised code release) is likely to lead to new explorations in this direction for speech self-supervised learning.
review_title: null

Meta-review: This paper proposed a self-supervised speech pre-training approach, by the name of SPIRAL, to learning perturbation-invariant representations in a teacher-student setting.  The authors introduced a variety of techniques to improve the performance and stabilize the training.  Compared to the popular unsupervised learning model wav2vec 2.0, better WERs were reported using SPIRAL with a reduced training cost.  All reviewers considered the work solid with sufficient novelty but also raised concerns regarding the generalization under unseen real-world noisy conditions and missing decoding details.  The authors responded with new Chime-3 results  and updated LM decoding results.  The new results show that, after a bug fix, SPIRAL can outperform wav2vec 2.0 when no external LM is used.  

Overall the proposed approach is technically novel.  The experiments are extensive and the results are compelling. In addition, the training time can be significantly reduced compared to wav2vec 2.0. All reviewers are supportive.  So I would recommend accept.
Decision: Accept (Poster)