Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents a non-contrastive pre-training method based on Barlow-Twins as an alternative to the wav2vec model which uses contrastive pre-training.  This work aims to improve the computational efficiency (memory and pre-training time) of the wav2vec model. The paper introduces two extensions to Barlow-Twins loss to account for the time dimension of an acoustic feature sequence.   Given a batch with $B$ input sequences each with time steps $T$ in embedding space of size $F$, first, time unrolling stacks all time steps from all inputs to get embeddings of shape $(B * T, F)$. The cross-correlation matrix of shape $F \times F$ is encouraged to be identity to ensure feature diversity.  Next, time merging concatenates features from all time steps to obtain embeddings of shape $(B, T * F)$. Similarly encouraging the cross-correlation of shape $B \times B$ to be identity ensures that the model does not collapse. Experiments are conducted on Librispeech dataset to evaluate the efficacy of the proposed approach.
strength_and_weaknesses: **Strengths**:
  - The paper is clear and the proposed approach is easy to follow. Performance on different subsets of the Librispeech dataset shows that non-contrastive SSL can achieve similar or even better WER than the contrastive models. The models also require smaller batch sizes resulting in fewer memory and GPU resources to train.

  - I appreciate the figure that explains clearly the difference between standard non-contrastive SSL and the proposed extension to speech. All details are presented to be able to reproduce the work.

**Questions and Weakness**:
   
  - Prior work: Non-contrastive methods for SSL in speech are not well discussed. There are quite a few generative/predictive models which have been investigated for SSL with speech. Some examples include MockingJay, Audio ALBERT, TERA, etc.  More exhaustive list can be found in [1]. data2vec [3] is a very close and relevant non-contrastive baseline that requires discussion.

  - I do not think that the claim "Non-contrastive SSL ASR yields a $2.3\times$ training speed up compared with contrastive SSL" is completely accurate.  It seems that the GPU hours were reported from the [2].  This may not be comparable because (a) the GPU hours can vary significantly depending on the setup. [4] reports the W2V base model training to 1764 hours when trained on 8 GPUs with gradient accumulation. It would be better to re-estimate the training times for W2V architectures on the same number of GPUs (8) used by non-contrastive models and by using the most recent w2v code for a fair comparison.  This could be done by running pre-training for a reduced number of steps.
    
  - While it is shown that non-autoregressive models require smaller batch size, the claim "Non-contrastive SSL requires fewer GPUs" is not accurate as gradient accumulation can be used for training using the same number of GPUs.  This also leads to better GPU hours.
    
- Issues with experimentation
  - For the case of iteratively combined pre-training, the paper only evaluates the setting of non-contrastive pre-training starting from a pre-trained contrastive model. This model has effectively been pre-trained for $650$K iterations. This needs to be compared against the contrastive and non-contrastive models trained for $650$K iterations to understand the performance gains/losses.

  - Confounding factors and lack of proper ablation make it hard to understand the source of the performance gains. Some examples are listed below:
    - Pre-training time is reported from the wav2vec paper which was trained on $64$ GPUs. The configuration of $8$ GPUs with gradient accumulation would have been a comparable configuration.
    - Audio is cropped to 5 seconds: This would lead to shorter sequences to the Transformer encoder than the wav2vec model. Given the quadratic complexity of the transformer, the extent of performance gains from (a) shorter sequence and (b) smaller batches is not clear. An ablation with fewer pre-training steps but isolating these factors would have made it easier to understand the impact.
    - Models are fine-tuned for $400$K steps. This is substantially more than the wav2vec models which are fine-tuned for 80K and 20K steps on LS-100 and LL-10  datasets respectively. It is not clear if the gains are only due to non-contrastive pre-training or also due to longer fine-tuning. Furthermore, this could result in slower fine-tuning for the non-contrastive model.
  - Baselines: While the paper compares against the wav2vec model, a relevant baseline would have been the data2vec [3] model which follows a non-contrastive setup as well.

**References:**

[1] Self-Supervised Speech Representation Learning: A Review. IEEE Journal of Selected Topics in Signal Processing 

[2] wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations. NeurIPS 2020

[3] data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language. ICML 2022

[4] Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognition. ICASSP 2022
clarity,_quality,_novelty_and_reproducibility: The paper is generally clear and the extension of Barlow-Twins for SSL on speech is novel. Enough details are presented to be able to reproduce the work. 
summary_of_the_review: This work introduces a non-contrastive method to improve the computational efficiency of SSL on speech sequences. I have recommended rejection primarily due to the experiments with confounding factors and lack of proper ablation as explained above. This makes it hard to understand the source and extent of performance gains.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Wav2vec 2.0 is widespread among speech community and used extensively to pre-train speech models for later fin-tuning on the speech recognition task. The downsides of wav2vec 2.0 is necessity of huge resources to get state-of-the-art on Librispeech, hard to tune as there are a lot of hyper-parameters and also contrastive loss which is often hard to deal. To democratize pre-training pipeline to people, authors in this paper propose to use non-contrastive loss and adapt Barlow-Twins from vision domain. They propose simple trick where reshaping is done of 3d tensor to 2d: time is stacked either with batch dim or with feature dim and then standard Barlow-Twins loss from vision is computed. These two operations, time unrolling and merging, ensure that all frames are different and that all sequence are different. With experiments on Librispeech and different amount of supervision (10h, 100h, 960) authors demonstrate that proposed method is in the same ballpark or better than wav2vec 2.0 and more GPU-efficient (can be trained with less resources, smaller batches, 2.3x faster in GPU hours). Also non-contrasitve training can be done after contrastive training and improve further results.
strength_and_weaknesses: **Strength**
- adoption of vision idea on Barlow-Twins to sequence domain
- experiments with different amount of supervision (10h, 100h, 960h) of Librispeech
- ablation on usage non-contrastive pre-training after contrastive one

**Weaknesses**
- Beam search is very shallow with not proper hyper-parameters. Yes, authors say about more practical use case, but beam search of CTC is still cheap, and it is research paper where we need to understand all limits and boundaries, especially in low-resource where contribution of language model is huge. I am ok of having several points of decoding for restricted setup and full setup. There are a lot of offline applications where we can use better / larger language models and more expensive search. With current setting it looks more artificial than practical and results obtained with language model are very-very bad, so hard to compare what exactly we are learning with SSL and how language is incorporated there. Because of transformer architecture we could learn some phoneme-level LM during SSL (and this is happening). There could be different effects of LM in the current paper in wav2vec as authors are doing 5s crop for input audio in SSL.
- "Nevertheless, trained on LibriSpeech dataset, our baselines are less than 1% absolute WER difference only when compared with SOTA solutions, e.g., Wav2Vec-2.0 (Baevski et al., 2020) which are the result of extensive HPO." - when I compare results for 10h, 100h, and 960 (Tables 1, 3, 4) of supervision with wav2vec 2.0 paper I don't see less than 1% absolute gap, it is much larger. I see a problem right now exactly with hyper-parameter search, reimplementation (or rerunning) wav2vec 2.0 baseline which is very weak and not consistent with what people reported (especially in the context that pretrained model is used and only fine-tuning is done). Also official recipe is 20k/80k fine-tuning steps only, not 400k as authors suggested to do in the paper. 
- This also raises questions on the efficiency: how do authors computed numbers for training time/gpu time for wav2vec 2.0? from 400k fine-tuning? from pre-training + fine-tuning? 
- I think the main weakness is reproduction of wav2vec 2.0 baseline which is far away from what is reported in prior work. Then comparison with proposed new methods looks very weird with these numbers. Next, results in Table 1, 3 for decoding with LM looks in some places very weird as not improvement is obtained even. Especially in 10h regime LM should help by a lot as we learn the phonemes / letters but very bad in word reconstruction and wav2vec 2.0 should gain huge boost with LM which is not the case authors reported. Overall results looks very weird and strange for me.
- absence of baselines on non-contrastive training, e.g. HUBERT, data2vec. (maybe at least comparison on speed in case authors can reproduce results from the paper)

Overall I feel this paper needs more work on experimental side to prove usefulness of the method. Right now it is more like proof of concept that Barlow-Twins loss can be trained with speech. No extra ablations on the parameters are given and how they are important or not (e.g. weak/strong augmentation, loss weights, EMA decay, etc.)
clarity,_quality,_novelty_and_reproducibility: **Clarity**
The paper is written well, however it could benefit with revision to simplify reading and remove repeated things in several places in the paper.

**Quality**
The idea itself is clear and well presented, however experimental setup and evaluation raise a bunch of questions which you can find below. Right now I have doubts about proper experimentation and efficiency metrics.

**Novelty**
The paper adapts known from vision Barlow-Twins loss while reducing the time dim in reshaping procedure. This reshaping is new component, the rest are known parts. However I am not sure why we should do time reduction and not some another dim.

**Reproducibility**
There are specified parameters in the paper on the training. Paper can benefit from revision where some details are specified too, e.g. how wav2vec 2.0 fine-tuning is exactly done to reproduce the prior paper.

**Detailed questions and suggestion to improve the text**
- Introduction: reference to wav2vec 2.0 in the context of absence of high quality large labeled data for many languages - it is strange usage of reference, maybe authors meant wav2vec multilingual paper then? Large curated unlabeled data - we have any more appeared since which actually gave huge speed up in development for SSL and semi-sup learning, e.g. LibriLight, VoxPopuli, Common Voice. For SSL in speech there are missing references to data2vec (Baevski, A., et al. Data2vec: A general framework for self-supervised learning in speech, vision and language, 2022, arXiv preprint arXiv:2202.03555), wav2vec-BERT (Chung, Y.A., et al. w2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training, ASRU, 2021), autoregressive/non-autoregressive predictive coding e.g. from James Glass team, etc.
- Figure 1. It will be very clear if we show properly the matrices in time merging and unrolling same way as used in the text (e.g. transpose the matrix for merging) and also add dimensions to all matrices on the plot to see reshape operation right away there. This can simplify reading the image and allow clear following it.
- do authors do only SpecAugment on time domain and not on frequency domain? Overall SpecAugment can be applied to the raw wave (using low-pass filter for frequency domain) but there could be other reasons to apply augmentation only after feature extraction (e.g. to avoid some boundary effects as we emulate FFT in feature extractor). Maybe it is better to reformulate this aspect in the text.
- usage of seq2seq is confusing. Often it is referred to a specific architecture and loss - encoder-decoder with cross entropy, e.g. machine translation, not to a general task itself of transforming one sequence to another sequence. Worth to say definition / change notation to properly refer to sequence modeling.
- time unrolling and time merging operations: how padding can affect the training here (say I have a bunch of very short audio and very long audio in the data, even if I cut to 5s I still can have 1s audio in the batch).
- why does time unrolling ensure that all frames cannot be different? why does this not happen for time merging too as soon as time dim again is reduced? why not having the cross correlation matrix of size TxT where we reshape B and F together?
- introduce HPO notation before its usage in the text (page 4), and also be consistent with "finetuning", "fine-tuning"
- "ensures similarly size of gradients" -> it is not true as it ensures only loss scale, while gradient can be different. More careful formulation in the text is needed.
- about augmenting the target branch - maybe it is worth to reference to fixmatch paper too as this is what they are doing too.
- projection head: it is not clear if authors use it in pre-training or not (I guess not).
- why do authors crop all audio to 5 sec during pre-training? how was this value established?
- about combination contrastive and then non-contrastive training there could be combination of non-contrastive and later contrastive or even jointly too losses. Any idea why not to try it or any expectations how it will work?
- "we fine-tune models for 400k" - why so long??? in wav2vec fine-tuning is done for 20k (10h) and 80k (100h) e.g.
- beam search decoding - I would recommend from my practice is to use random search instead of grid search, next try larger language model weights as 2 could be small especially for 10h where LM contribution is huge. And also use [-10, 10] range for word insertion penalty. At least worth to check best params from wav2vec 2.0 paper and include those in the search ranges.
- General comment on improving readability: there are a lot of repetitions of the content, e.g. description of evaluation/training and then repeating the same thing in every section on results. This can be simplified.
- page 7 usage of 6/12% relative improvement: when I computed it I got way less percentage of improvement. Could authors recheck here claims and specify exactly how they got these numbers?
- Table 2 "significant difference" - looking at numbers I would say they are similar, not significant different. Also in all results we can expect at least 0.1 WER std from my experience and from many papers where people report results. So in many cases improvements which are reported are not significant and it is more "in the same ballpark" which is still valid in the sense of simplifying the SSL training and speeding it up. No need to oversell results if they give same results but with way less resources. This is already good point and justifies the need of new method.
- What do authors want to show with reporting CER? Especially after decoding?
- Batch in wav2vec 2.0 is not needed for contrastive training itself as in vision, as we sample negatives from the same sample in future/past, not from other samples. Mostly large batch is needed to efficiently train transformer architecture itself. I would suggest to reformulate a bit the sentences on this topic in the text.

summary_of_the_review: There are a lot of methods proposed in computer vision which can not readily be applied to other domains, e.g. sequences in speech and text. This works aims to adapt Barlow-Twins SSL technique to sequences and show that non-contrastive training can be on pair with contrastive training, e.g. wav2vec 2.0. Authors proposed to pack time either with batch dim or feature dim to compute then cross correlation matrices in Barlow-Twins. I like that there is direction developed in adapting some methods from computer vision and also developing non-contrastive methods to simplify and speed up the training. However, empirical comparison done in the paper raises a bunch of questions I specified in the main review (baselines are significantly worse than reported in original paper, beam search is very weak, etc.). Also there is no comparison with other developed speech specific non-constrastive methods like HUBERT, or more general framework as data2vec which is the most closed to considered Barlow-Twins in terms of architecture and EMA (but not loss).
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper describes non-contrastive learning based self-supervised pre-training approach for the speech recognition task. As opposed to the popular methods such as Wav2vec2, Hubert, etc. which use contrastive learning for pre-training, this paper describes how non-contrastive methods such Barlow-Twins for the audio pre-training. 

The main contribution of the paper is:
(i) Extend Barlow-Twins (Siamese Network) training for audio modality (with additional time axis) with new loss functions for that additional axis.
strength_and_weaknesses: Strengths:

(i) Paper clearly shows the extension of Barlow-Twins methods for audio modality with intuitive losses. 
(ii) The empirical results show the method's efficiency (compute resources) compared to the Wav2vec2-like pre-training.

Weakness:

(i) In the section (augmentation in latent space), it says that masks are applied with a probability of 0.05 and mask length of 10 for the target model and 0.1 and 20 for the online model. 
- What is the fraction of input unchanged in the output of both the target model and online model representations? 
- Time unrolling loss compares similarity across time-frames for the same frames and if most of the temporal frames are unchanged then the solution would be trivial. I am curious in understanding the effect of augmentation on the overall objective, which is not described in the paper.
- In images, the augmentations(cropping, resizing, flipping, etc.) are more meaningful, which is not a luxury in speech. Due to the fixed audio duration for each batch (5 ms), perturbations in the frequency domain cannot be carried out, which further limits the augmentation space.

(ii) I do not see the comparison of static and dynamic scaling for the loss functions. Is the idea of using a stop gradient for dynamic loss novel? If not the citation for that is missing. 

(iii) Paper states that for the pre-training the audio utterances are cropped into 5 seconds. (Wav2vec2 uses 15.6 seconds). 
- Was this design choice considering the GPU size? 
- What is the effect of having longer utterances on the overall pre-training objective?
- Because Non-contrastive learning methods don't require larger batch sizes, does having longer utterances help or deteriorate the pre-training performance?

(iv) I find *Iteratively Combined* baseline overtly powerful because of the number of customization steps (400K for wav2vec2 + 250k for the Non-contrastive method), as compared to W2V2 and Non-Contrastive method. 
- I would suggest running each of the methods to a fixed number of steps and then comparing. Since the number of parameters are the same, this would give a more accurate picture of the efficacy of each of these methods.

(v) Results of other self-supervised methods such Hubert, Decoar, etc. should be placed to understand the relative strength of the non-contrastive method. 

(vi) Ablation of both the Time Unrolling and Time Merging loss is missing. Very curious to know what is the impact of time merging loss on the overall objective.

Minor:
(i) A reference missing in line 3 of the `Evaluation with High-Resource Labeled Data` paragraph.




clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to follow. The paper extends the non-contrastive self-supervised training to the audio modality which is novel. 
summary_of_the_review: Although the paper has some interesting ideas for improving the efficiency of self-supervised training for speech recognition, it lacks an extensive understanding of the various design choices made in the paper. While the paper is interesting, by adding certain key details it would be easy to understand the positioning of the non-contrastive methods in the self-supervised pre-training landscape.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: While wav2vec-style contrastive learning has shown to be very successful for ASR, it requires a lot of resources and time for training. In the vision domain, Barlow Twins, a solution that naturally avoids collapse, has shown to be able to achieve better (or competitive) performance compared with contrastive learning (e.g., SimCLR) while using much smaller batch size. However, Barlow Twins style training is under-explored in the ASR/audio domain.

The authors applied revised Barlow Twins training to the wav2vec-2 framework (CNN feature extractor + Transformer), and achieved competitive performance compared to wav2vec 2.0, while reducing training time, GPU usage and improving convergence. 

strength_and_weaknesses: The strength:
— To my knowledge, this is the first work that applies Barlow-Twins methodology to learn representation for ASR; The adoption of BT into sequential representation learning is not trivial. Previous, I only find BT is applied for speaker recognition.

— The authors show that the proposed methods are more resource efficient and achieve comparative performance. A) It improves convergence, b) it reduces training time, c) it significantly reduces GPU training times, d) it requires smaller batch size thus reducing memory requirements.

— The authors also found that combining the proposed method with a contrastive learning approach is helpful.


The draft can be improved with some clarification and extra ablation studies:
— Same as wav2vec 2.0, the proposed methods apply masks to the CNN-extracted features; Here, the mask is only used to distort the input (to my understanding), thus the authors can explore ways of distortions other than applying masks.

— Regarding Time Unrolling and time merging losses: The proposed method chunks the audio into 5-seconds. It seems that for longer input or when using larger batch size/projection dimension, the calculation of the two losses can be much time-consuming (i.e., slower). Also, what’s the relative importance of the two losses? Maybe an ablation study with different w_U and w_M (static loss) could be interesting.
 
— The authors show that further fine-tune wav2vec 2.0 with the proposed methods achieves better performance than both wav2vec 2.0 and the proposed non-contrastive training. What if the authors first train with the non-contrastive loss and then fine-tune with contrastive loss?

— The “iteratively combined” solution is only applied to LS-960; It would be good to see the "iteratively combined" solution also applied on low-resource scenarios, and see its relative improvement.

clarity,_quality,_novelty_and_reproducibility: — Page 7 – “discussed in Section ??”

— In the abstract, the authors say “a combination of both masked based SSL and non-contrastive SLL” improves the ASR performance. Here, ‘masking-based SSL’ actually means contrastive learning solution (e.g., wav2vec 2.0)? I think masking is also used for non-contrastive solutions.

— In table 1&2, the ‘Iteratively combined’ solution seems to be a misleading name. Do authors really try to iteratively apply contrastive and non-contrastive training? I think it probably means fine-tuning a contrastive model with non-contrastive loss.

— In Table 5, the resource consumption of “iteratively combined” should be the sum of both wav-2 and non-contrastive.

— In page 5, the authors claim that they do not perform any pre-processing; It seems in wav2vec, they do use mean/std normalization.

summary_of_the_review: At this moment, I’m inclined to recommend this paper, though the paper could be in a better shape with more clarification and ablation study.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper looks at an alternative to wav2vec by discussing a non-constrastive pre-training method

Strengths:
* well written and clear
* novelty with Barlow-Twins to speech domain
Weaknesses
* prior work is lacking
* issues with some of the claims
* experiments are lacking
justification_for_why_not_higher_score: * better experimental rigor and explanations
justification_for_why_not_lower_score: * interesting idea
Decision: Reject