Resource Efficient Self-Supervised Learning for Speech Recognition
Authors: Abhinav Mehrotra
Alberto Gil Couto Pimentel Ramos
Nicholas Donald Lane
Sourav Bhattacharya
Venue: Submitted to ICLR 2023 (2023)
Keywords: SSL
ASR

Abstract: Representation learning from sequential data using self-supervised learning (SSL) has proven to be a powerful technique and improved state-of-the-art (SOTA) results when fine tuned for various downstream tasks, including Automatic Speech Recognition (ASR). So far the success of SSL frameworks, e.g., Wav2Vec-2.0, for sequence-to-sequence (seq2seq) modeling is primarily carried out by masking intermediate features and then solving a contrastive task in an end-to-end manner. Although very successful, the overall training time (for example, days or weeks) and demanding resource requirements for achieving SOTA performance remain a significant barrier to further improving ASR solutions using such approaches. In this work we show that non-contrastive learning, such as an extension of the Barlow–Twins methodology, when applied to seq2seq SSL modeling improves convergence, while reducing training time. Our results show that Wav2Vec-2.0 architecture pre-training with a non-contrastive SSL approach reduces the GPU training hours by 2.3 times, compared to masking based SSL approaches, while achieving a significant improvement (i.e., up to 6% relative WER decrease) in the model performance for the ASR task. We further demonstrate that a combination of both masking based SSL and non-contrastive SSL improves the ASR performance, e.g., up to 12% relative WER decrease, for all splits of LibriSpeech evaluation dataset.


Under review as a conference paper at ICLR 2023
RESOURCE EFFICIENT SELF-SUPERVISED LEARNING FOR SPEECH RECOGNITION
Anonymous authors Paper under double-blind review
ABSTRACT
Representation learning from sequential data using self-supervised learning (SSL) has proven to be a powerful technique and improved state-of-the-an (SOTA) re- sults when fine-tuned for various downstream tasks, including Automatic Speech Recognition (ASR). So far the success of SSL frameworks, e.g., Wav2Vec-2.00 for speech-to-text modeling is primarily carried out by masking intermediate fea- tures and then solving a contrastive task in an end-to-end manner. Although very successful, the overall training time (for example, days or weeks) and demanding resource requirements for achieving SOTA performance remain a significant bar- rier further improving ASR solutions using such approaches. In this work we show that non-contrastive learning, such as extension of the Barlow-Twins methodology, when applied speech-to-text SSL. modeling improves conver- gence, while reducing training time. Our results show that Wec- architec- ture pre-training with non-contrastive SSL approach reduces the GPU training hours by 2.23 times, compared to masking based SSL approaches, while achiev- ing a significant improvement (i.e., up to 6% relative WER decrease) in the model performance for the ASR task. We further demonstrate that combination of both masking-based contrastive and non-contrastive SSL improves the ASR per- formance, e.g., up 12% relative WER decrease, for all splits of LibriSpeech evaluation dataset.
INTRODUCTION
Modern industry-sceu speech recognition systems often require cens-of-thousand hours of high quality labeled speech data achieve acceptable deployment performance (Baevski et al.] [020]] Ramos et al., 2022). However, large-scale data collection remains an extremely time consuming and costly procedure and does not scale as the number of languages to support increases. Furthermore, for a majority of the vast number of spoken languages, a large and high-quality training dataset is often unavailable (Babu et al. 2022). Thus, effective learning using primarily unlabeled data has been an important and long-standing research topic within the machine learning community, where the main emphasis is on learning good representations from unlabeled data and then fine-tuning using task dependent limited amount of labeled data.
Recent progress in self-supervised learning (SSL) has been highly successful in utilizing unlabeled data and demonstrated superior performance in the domains of computer vision (CV) (Chen al. 2020 He et al.) 2020; Chen & He 2021). natural language processing (NLP) (Devlin et al.) 2019
Lewis al. 2019), and speech recognition (SR) (Liu et al., 20200a] Chung et al. 2021 Baevski et al. 2022; Schneider et al. 2019: Baevski et al. 2020). In particular, SSL-based approaches ex- ploit abundance of unlabeled data to learn underlying representations, while using both contrastive
and non-contrastive approaches (Jaiswal et al., 2020 Balestriero & LeCun 2022). Especially, in the domain of ASR, masking based contrastive methods have emerged as the leading SSL approach and yielding current state-of-the-art solutions, e.g., Wav2 (Baevski et al. 2020). and Hu- BERT (Hsu al.) (02T). The success of these approaches mainly due to easy availability of large curated unlabeled open source datasets (Kearns) 2014] Panayotov et al.) 2015) Kahn et al. 2020) Wang et al. 2021 Ardila et al. 2020), availability of industry-scale GPU infrastructures, improve- ments in the data training pipeline and scaling (e.g., data-, pipeline-, model- parallelaallll)smm) of deep learning frameworks. However, the overall training time for achieving SOTA performance remains significant barrier further improving ASR solutions using contrastive SSL.
In this paper we present a technique for decreasing the training time of SSL based ASR systems by using non-contrastive SSL method, rather than contrastive method, for speech representation

---

Under review as a conference paper at ICLR 2023
learning. In general, non-contrastive SSL is heavily under represented in audio research, with few notable exceptions (Liu et al. 2022). This is primarily due to the fact that SSL methods are more es- tablished in the domain of CV than audio applications and often require novel extensions. Towards bridging this gap, we consider Barlow-Twing (Zbontar et al. 2021) as representative example of non-contrastive SSL and expand its scope from vision to audio by inventing the following exten- sions: (i) we incorporate a number of new loss functions via purposefully designed time-merging and time-unrolling, and (ii) applying static (hyper-parameter optimization) and dynamic (stop gradi- ent) methodologies to balance the different scales in individual losses. We further explore the effect of sequential use of con-contraetive and contrastive training and observe improved performance, i.e., decreased word error rate (WER) when compared to solely non-contrastive or contrastive training, which is in line with recent work on SSL based speech representation learning for speaker verifica- tion rather than ASR (Zhang & Yu 2022).
summary of our main findings regarding the benefits of our non-contrastive SSL approach for speech representation learning are as follows: (i) non-contrastive SSL ASR yields 2.23x training speed up compared with contrastive SSL ASR in our experiments, while simultaneously improving up to 6% relative WER (c.f., first and second rows of Tables and 6), (ii), fewer required GPUs and smaller batch size reducing memory requirements in con-contrat as opposed to contrastive methods (c.f., Table 3 and (iii) lowest WER achieved by sequentially combined approach followed by non-contrastive training (c.f., third rows of Tables 6).
APPROACH
The most common SSL methods in speech considered are masking- based contrastive learning and autoregressive prediction based learning. In this work, we explore the potential of a non-contrastive SSL method for learning speech representations and its effectiveness on the downstream ASR.
2.1
MOTIVATION
Recent work in the area of non-contrastive SSL (e.g., BYOL Grill et al. 2020), SimSiam (Chen & (021n) have shown remarkable capacity to learn powerful representations from only positive pairs, i.c., two augmented views of the same data point. Unlike contrastive SSL approaches that use negative pairs prevent representa- tional collapse, non-contrastive SSL approaches employ a dual pair of Siamese networks to process two augmented views of data point and minimize their representational differences.
He 202T). Barlow-Twins (Zbontar al.] 202I). DINO (Caron
In general, contrastive SSL methods require large batch sizes, e.g. SimCLR (Chen et al. 2020 and MoCo (He et al., 2020), to achieve good performance. On the contrary, non-contrastive SSL. approaches are comparatively more efficient and easy to train with smaller batches and reduced memory. As shown in (Zbontar et al. 202I). a non-contrastive SSL method such as Barlow-Twins could learn effectively with up to 16x smaller batch size.
2.2 METHON
In this subsection, we present an overview of our approach for learning speech embeddings with first of its kind non-contrastive SSL method designed for time series speech modeling (c.f., Figure I (b)) and its comparison with a standard non-contrastive SSL method for non-time series data such as images (see Figure (a)).
Similar to all non-contrasttve SSL methods used in vision, our approach for learning speech embed- dings has a dual pair of Siamese networks referred to as online (O) and target (T)) networks. Only the online network is trained via gradient descent and the target network employs a momentum encoder HH et al.) 2020) that slowly follows the online network in a delayed fashion through an ex- ponential moving average (EMA). The outputs of the online and target networks are then encouraged to learn good representations via a self-supervised loss function.
However, there are two key differences in approach for learning speech embeddings compared to image embeddings, which can be categorized as modeling and learning differences. These differ- ences summarized below.
First, instead of performing augmentations in the input space (c.f. Figure D (a)). our solution op- erates in a latent space (see Figure [ (b)). Specifically, we apply augmentation not directly on the
2

---

Under review as a conference paper at ICLR 2023
Online Network
b=0
Y4
Encoder
Projection
Z
b=1
X Augment
EMA
EMA
loss
Y8
Projection
4
Z
b=0
0
b=1
Target Network
(a) Standard non non-contrastive SSL approach with optional exponential moving average (EMA) and stop gradient (sg) to learn representation of non-time series data. Such approach takes image data X (depicted with batch size b=2), augments into two distorted views Y and Y and feeds it via online and target networks to produce representational embeddings Z and z An appropriate loss function is then created from these embeddings for self-supervised learning, particularly relevant to work, the existing Barlow-Twins loss function cross-correlates Z and 2 and promotes the correlation matrix to be close to identity.
[B"T,
Z Unsling
BB
T, F
[T"F,
B]
Online Network
Extractor
Augment
Encoder
Projection
Z
b=0
b=1
2 3
Time Mashiu
0
t=0
Z
2
Lu
M
3
X
EMA
EMA
loss
[B"T,
FI
ZB
[B, T, FJ
Y8
Feature
Augment
Encoder
Projaction
Z
b=0
b=1
0
Unseiin
Time
6
Target Network
t=0
t=1
T F,F
0
Meser
ZB
(b) Our approach for learning time series speech representations via non-contrastive SSL method. In the audio domain, a larger encoder is often composed of two submodules namely feature extractor and a smaller encoder as depicted in the figure, and data augmentation employed between. This motivates the changes to the online and target networks in our approach. Furthermore, traditional loss functions such as Barlow-Twins cannot be readily used given the dimensionality of the embed- dings Z is larger in the audio than the image domain. For this reason introduce time unrolling and merging approaches leading each to a different loss function which we statically or dynamically combine for improved training.
Figure 1: Comparison of non-contrastive SSL approaches for (a) image and (b) audio data.
input X, but rather to the outputs of the feature extractor to generate Y and Y The motivation behind this is that the most common form of data augmentation in audio is SpecAugment (Park et al. 2019) which acts on the audio spectogram. However the Wav2Vec-2.0 (Baevski et al. 2020 architecture the spectogram replaced by learnable sub-network referred to the feature extractor. Nevertheless we have found that the same principles underlying Spec are used this point in the processing pipeline in order avoid overfitting and increase generalization.
3

---

Under review as a conference paper at ICLR 2023
The second key difference stems from the way the loss is computed from the produced embeddings Z A and z Depending on if the scenario is non- time series or time series there is a fundamental difference between the tensor representation Z and z Specifically, shown in Figure (a) Zd and 8 are two dimensional tensors, whereas in Figure (b) 7 and 7 are three dimensional tensors due to extra time dimension.
In order to address this difference, in this work we extend the application of Barlow-Twins loss for speech-to-text SSL modeling, in particular for speech (and more generally time varying) representa- tion learning. As demonstrated in (Zbontar et al. [202T) for non time varying signals, one computes the cross-correlation matrix between the embeddings of online and target networks, and then op- timizes the model such that this correlatrrl matrix comes close to the identity. For learning speech embeddings, we appropriately generalize the original Barlow-Twins loss by computing from different views of the embedding data, namely as time unrolling and time merging approaches as depicted in Figure [ (b), as well as the text below.
unroll the sequence time axis and stack each frame together. Specifically, a batch of outputs with dimensions [B, T, F] is transformed to [B * T, F]. This transformation is performed for the outputs of both the online and target models. A cross correlation matrix C of size F x F' is then created and encouraged to be close to the identity via a loss function Ly. This enables the model enforce diversity in each sequence, such that all frames cannot be the same. In other words, the time unrolling loss on the cross correlation matrix based on the reshaped tensor [ T, F) promotes that the th feature across all the B samples and T timesteps is different from the j th feature.
Time Unrolling. In this approach,
ii) Time Merging. We merge together the features of each sequence on the time axis, so that the output of each utterance becomes one long tensor of features. We then compute transpose of this as the Barlow-Twins loss is computed on the second dimension. Specifically, a batch of outputs with dimensions [B, T, F] is transformed to [I' F, B]. Finally, we create B x B correlation matrix CM and feed it to the loss function LM. Through this loss we want the model to learn inter-sequence diversity such that the i th sample different from the j - th sample within the batch. This ensures that all sequences are not the same and thus SSL training does not collapse.
Formally, the loss function (L) can be defined as follows:
(1-0))
0
2(C NN - 1) where CtU CrossCorrelation (Z)) zg)
LU Σ
NU
(1
CA
2(C
LM +-
Nas (NM - 1)' where C. <- CrossCorrelation (Zt, 2
NM
wi
Lu + WM LM
= static
loss
Ly sx(Ler)
+
sg(Lae)
Ly
= dynamic
Here, No and NM denote the first dimension of the square matrices C and C. respectively. In practice we observe that the unrolling (U) loss Tu and the merging (M) loss Las often have differ- ent scales, and for this reason we explore combining these loss functions statically or dynamically into an overall loss L. More specifically, for static scaling we simply multiply each loss function with a hyper-parameter parameter weight wy for Lu and wM for Ly on which we conduct optimisations (HPO). For dynamic scaling, partial loss is divided by stop gradient applied to itself. Given that a scaling of the loss propagates to the size of the gradient update, this leads overall better gradient update behavior without the need for HPO on the weight of each loss.
pprem-errretter the stop gradient (sg) operation to ensure each
3 EVALUATION SETTINGS
3.1 ONLINE AND TARGET MODELS
We create the online and target networks via surgery on the Wav2Vec-2.0 Base architecture (Baevski et al. 2020). This is accomplished by taking the feature extractor and the encoder components of Wav2 2.0 as well as additional augment and projection layers, which we organize in a sequential manner from feature extractor, augment layer, encoder and projection layer described below.
4

---

Under review as a conference paper at ICLR 2023
Feature Extractor. The feature extractor consists of seven convolution blocks. Each block has a temporal convolution layer followed by layer normalization and a GELU activation func- tion (Hendrycks & Gimpel 2016). The number of channels, strides and kernel widths are the same as in the Wav2Vec-2.0 architecture (Baevski et al. 2020). This results in the feature extractor's hop length of 20ms and the receptive field of 25ms of audio.
Augmentation in Latent Space. An augmentation block is used to generate and apply masks on the feature extractor outputs. We a masking function similar SpecAugment (Park al.) 2019) and mask time-steps during training which avoid model collapse. We use the probability of 0.05
and mask length of 10 for the target model, and we use 0.1 and 20 for the online model. We use more masking for online network to bring highly distorted view close to its corresponding less distorted view, which inspired from the local/global crop strategy presented (Caron al. 202T) and weak/strong augmentation strategy used in (Sohn et al. (020). Note that, in order to avoids boundary effects, we apply augmentation only to the outputs of the feature extractor that mimics FFT style transformation with receptive field of 25ms of audio and a 20ms hop length.
Encoder. We use the same encoder architecture as Wav2Vec-2.0 Base model which consists of 12 transformer blocks.
Projection. We project the output a lower dimensional latent space. Specifically, we use dense layer to transform the output of the encoder to 29 dimensional space. This helps to efficiently compute the loss on smaller output dimensions. Here, we selected 29 dimensional output as it is also the same dimension of final vocabulary (discussed later in 3.4)
3.2 DATASET
During the SSL phase, pre-train the models using the LibriSpeech corpus (Panayotov et al., 2013) of 960 hours of audio (LS-960) without the transcriptions. We crop the audio utterances into 5 seconds of speech and batch them together. We do not perform any pre-processing since the feature extractor directly processes the raw audio data.
When fine-tuning the models, consider 3 labeled data settings:
i) LS-960: 960 hours of LibriSpeech data with transcriptions (Panayotov et al.) 2015).
ii) LS-100: the train-clean-100 subset of LibriSpeech comprising 100 hours of labeled data,
iii) 1.L-10: Libri-light limited resource training subset of 10 hours labeled dataset, which was originally extracted from LibriSpeech (Kahn et al.) (0200).
We evaluate models on the standard LibriSpeech dev-cleam/other and fest-clean/other sets.
3.3 PRE-TRAINING
For pre-training the representation learning model, that includes the feature extractor and encoder networks, consider three settings described below.
W2V-2: for this setting, we the pre-trained weights of Wec2 model, which is trained with masking based contrastive SSL method (Baevski et al.) 2020). We use the checkpoint released in (Baevski al.) (2200) for Wev2Vec-2.0 Base model which was pre-trained on 960 hours of LibriSpeech corpus.
ii) Non-Contrastive: this setting is based on our proposed non-contrastive SSL method for speech representation learning as described in Section 2.2
ii Sequentially Combined: in this setting, we consider pre-training the model with the W2V-2 approach followed by our proposed Non-Contrastive setting. Specifically, we start with pre- trained weights from Wav2Vec-2.0 Base model (as described for W2V-2 pre-training setting) and further pre-train with con-contrastive SSL method (i.e., Non-Contrastive setting). Note that there could be multiple ways to combine the contrastive and non-contrastive training such as non-contrastive followed by contrastive, vice versa, or in an iterative sequence with each contrastive non-contrastive being applied fixed number of steps or perhaps with scheduler to switch between the two approaches in a more elaborate fashion. However, we leave these for further exploration they are outside of the scope of the present work.
We use the ADAM optimizer KKingma & Ba 2015) with a learning rate of 10-5 The learning rate was selected between 10-5 and 5 10- after training the models for 10K steps.
Optimization.
5

---

Under review as a conference paper at ICLR 2023
We exponential-decae with decay rate of 0.99 learning rate scheduler. For Von-Contrastive and Sequentially Combined approaches, we train the models for 400K and 250K steps. For both of these methods we use the batch size corresponding to 16 minutes of audio (details are described in Table 4). We observed that the convergence with dynamic scaling of the loss (discussed in (2.2) was better than the static option, which motivated us
it throughout our experiments. We note however that it is possible that the static option could work with other values of WM and wy than the ones we tried. Specifically values in [0.1, 0.5, 1.0], so more extensive HPO could perhaps improve on the dynamic scaling.
experimented with nine combinations with wM and wy taking
3.4 FINE-TUNING
For the fine-tuning phase, we take the target network without the projection layer and add randomly initialized linear projection on top. This layer's outputs have the same dimensionality V as the vocabulary of the downstream task. Specifically, for the ASR task we select V as 29, which includes 26 characters A-Z, the white space, the apostrophe (') as well as a blank symbol required for the connectionist temporal classification (CTC) deseribed below.
We use CTC loss GGaves et al. 2006) and train the network by minimizing it. We keep the fea- ture extractor block frozen during the fine-tuning and train the parameters of only the encoder and projection layer parts of the network (c.f., Figure (h)). During the fine-tuning phase we keep the augmentation parameters the same as for the target model. However, during fine-tuning with LS-100 and LL-10 datasets, we double the number of masks to avoid overfitting.
Optimization. During the fine-tuning phase, we use the ADAM optimizer (Kingma & Ba 2015) with learning rate of 5 10-5 The learning rate was selected between 5 10-3 and 7 10 after training the models for 5K steps. We use the exponential 0.99 a learning rate scheduler. For LL-10, LS-100 and LS-960, we fine-tune the models for maximum of 30K, 100K and 400K steps respectively. We fine-tune all models with the batch size of 24 accounting for step gradient accumulation used with a batch size of 8. We use the same parameters while fine-tuning with LS-960, LS- 100 and LL-10 datasets.
function with the decay rate of
3.5 DECODING AND LANGUAGE MODEL
For decoding, we CTC beam search decoder with and without language model (LM) to evaluate our models. A 4-gram LM trained on the LibriSpeech corpus is considered for decoding with the LM. We use beam width of 50 to measure the model performance with and without LM. We note that this beam width is smaller than higher performing systems. but we consider is a good trade-off' with latency deployment time. We tune the weights of the LM for the range of 10.1, 2.0] with an interval of 0.1 and word insertion penalty for the range of [-0.5, 0.5] with an interval of 0.05.
3.6 METRICS
For ASR quality we consider the word error rate (WER) evaluation across various datasets, and for resource efficiency metrics we consider GPU hours and wall clock hours, as well as factors that influence these such as number of GPUs and batch size.
3.7 BASELINE
Past studies have proposed variations around the idea of learning speech representations by pre- dicting the content of unseen regions (Ling et al. 2020 Ling & Liu| 2020 Liu et al. [020b] Chi al. 2021]. On the other hand, recent studies have shown the potential of SSL pre-training by contrasting the target unseen frame with randomly sampled ones (Baevski et al. 2019). Similarly, studies in (Baevski al., 2020 Hsu al. | (021) have proposed predict discrete targets of masked regions as the SSL training objective. As shown in (Hsu et al.) 2021), Wav2Vec-2.0 and HuBERT other reconstruction or contrastive alternative approaches. However, we only add Wav2Vec-2.0 as baseline in our ex- periments since Wav2Vec-2.0 BASE model's performance (2.7 / 7.9, 3.4 / 8.0 on dev-clean/other, test-clean/other) is very similar to HuBERT BASE model's performance (2.7 / 7.8, 3.4 / 8.1 on dev-clean/other, teattcrran/ohher..
approaches have demonstrated great performance by outperforming
3.8 SCOPE
is worth noting that given our focus is on the relative merits between contrastive and non- contrastive methods, we restricted the HPO to make this exploration feasible. In particular, did

---

Under review as a conference paper at ICLR 2023
not do extensive HPO on: (i) data augmentation layers, namely spec augment, (ii) batch size scaling which gives better results on contrastive rather than non con-contrastive approaches, and (ii) static loss scaling parameters (i.e., WM and wy)). Nevertheless, trained LibriSpeech dataset, have less than 1.5% absolute WER (on dev/test-clean splits) difference only when compared with SOTA solutions, e-g., Wev2 (Baevski et al. 2020) which are the result of extensive HPO.
baselines
RESULTS
4.1 QUALITY COMPARISON OF NON-CONTRASTIVE AND MASK BASED CONTRASTIVE SSL
In order to examine the potential of our non-contrastive SSL method for learning effective speech embeddings, we first compare its performance, for the downstream ASR task, with the SOTA con- trastive SSL. method Wav2Vee-2.0 (Baevski et al. 2020). We perform this evaluation by fine-tuning the pre-trained models in two settings. Firstly, with high-resource setting where large quantities of labeled speech are available. This enables us to compare the effectiveness of speech embeddings learned with contrastive and non-contrastive SSL methods. Secondly, with low-resource setting where the amount of labeled data is limited. This helps us to compare the usefulness of speech embeddings, learned with the two methods, for improving low resource settings.
Evaluation with High-Resource Labeled Data. We fine- tune the embedding models, pre-trained using W2V-2 and Kon-Contrastive methods (discussed Section (.3), with LS-960 dataset. Ad- ditionally, we also fine-tune the pre-trained W2V-2 model with explore the performance of models when trained sequentially with both approaches (i.e., Sequen- tially Combined method). Though this approach is computationally inefficient, we investigate it to examine the potential of sequentially combined pre-training when resources are not constraint or when a pre-trained model is available and we would like to further improve it.
SSL approach to
Table presents the ASR model performance in terms of WER (c.f. Appendix for character error SSL method
rate performance). The results for these evaluations show our proposed outperforms the SOTA W2V-2 approach for speech representation learning by achieving up to 6% relative WER improvement dev/test-clean splits LibriSpeech dataset.
At the same time, our results show the Sequentially Combined SSL approach significantly boosts the performance of downstream ASR task and achieves up 12% relative WER improvement dev/test-clean splits. Conclusively, if the priority is to decrease WER rather than training time, Sequentially Combined SSL method observed to yield the best results.
Evaluation with Low-Resource Labeled Data. To evaluate the performance in low resource set- tings, we considered fine-tuning the W2V-2 and Non-Contrastive pre-trained models with L.S-00 and 1.0.10 datasets (discussed in Section 3.2
Table presents the ASR model performance on low resource settings in terms of WER on Lib- riSpeech dev and test splits (additional character error rate performance reported in Appendix). Our results show that even on the low resource settings, the speech embeddings learned on unlabeled data with our non-contrastive SSL method are significantly more effective than the ones learned with contrastive SSL method. Specifically, we observe up to 12% relative WER improvement on dev/test-clead sets both LS-100 and LL-10 datasets. Note that we did not perform HPO for the evaluation with L.S1000 and LL,-10, instead we used the optimal ones found for LS-960 evaluations. This is reflected in the quality performance compared to that reported in (Baevski al., 2020). Nevertheless, we do see an improvement in terms of WER performance of Non-Contrastive SSL compared to contrastive W2V-2 method.
4.2 COMPUTATIONAL AND RESOURCE EFFICIENCY
There has been great progress in the improvement of ASR systems which are founded through mod- eling advancements (e.g.. Conformer (Gulati et al. 2020)), pre-training and learning methodologies Ses.WayyVVec2200 (Bevsski et al., 2020)), data augmentations (e.g.. SpecAugment (Park et al. ). and on. Nevertheless, training time and the requirements for huge number of resources remain a significant barrier to scale up these improvements for a wider number of practitioners, supported languages, latency requirements and deployment scenarios.
7

---

Under review as a conference paper at ICLR 2023
Table I: Model performance in terms WER on the Librispeech dev/test sets when fine-tuned on LS-960. Our Won-Contrastive SSL method outperforms the SOTA W2V-2 approach (Baevski et al., Sequentially Combined SSL approach further boosts the performance of W2V-2 model.
2020
Model
LM
dev
test
clean
other
clean
other
W2V-2
None
4.33
11.21
4.47
11.36
Non- Non-Contin Contrastive Sequentially Combined
None
4.01
11.26
4.29
11.15
None
3.81
11,11
4.01
11.00
W2V-2
4-gram 2.99
7.97
3.30
8.10
Non- Contrastive
4-gram
2.88
7.87
3.22
8.07
Sequentially Combined
4-gram
2.83
7.90
3.20
8.10
Table 2: Model performance in terms of WER on the Librispeech dev/test sets when fine-tuned on LS-100 and LL-10. Our Non-Conton Con-Contrasttve SSL method achieves marginal improvement in WER as compared to the SOTA W2V-2 approach.
Model
Fine-tuning Data
LM
dev
test
clean
other
clean
other
W2V-2
10
None
12.18
24.95
12.58
25.63
Non-Contrastive
None
12.17
24.17
12.19
24.81
W2V-2
10
4 g-gram
6.65
17.22
7.04
17.76
Non- Contrastivee
10
4-gram
6.38
16.45
6.64
17.01
W2V-2
100
None
6.32
19.20
6.65
19.15
Con-Contrastive
100
None
6.42
18.47
6.84
18.30
W2V-2 Non-Contrastive
100
4-gram
4.03
13.64
4.51
13.71
100
4-gram 4.04
13.25
4.45
13.00
Due to this reason, in this section we investigate the computational and resource efficiency of non- contrastive
contrastive SSL approaches for speech representation learning. Table shows the quantification of the GPU hours required for pre-training with these SSL methods, and Table presents the batch size requirements which is key enabler of allowing for better embedding models at the cost of increased memory requirement.
Table 3 shows that indeed in SSL for ASR, training time can be decreased via non-contrastive SSL (as proposed in this paper) rather than contrastive W2V-2 by as much as 2376 / 1064 2.23 times. Furthermore, Table 3 also highlights that non-contrastive SSL method requires fewer GPUs when compared with contrastive W2V-2, which can be seen consequence of the required batch size in each approach as we discuss next.
Inline with the previous study (Zbontar et al. 2021). where contrastive approaches have been found to require much larger batch sizes than non-contrastive approaches. Our experimental setup, as described in Table also highlights fewer required GPUs and smaller batch size in non-contrastive (960 secs) as opposed to contrastive (5376 secs) approaches, allowing for training on memory constrained GPUs.
Finally, it is worth noting that the priority is to decrease WER rather than training time or GPU resources, lowest WER can be achieved by sequentially combined contrastive followed by non- contrastive training. Although, we have already discussed WER improvements from this approach in the previous subsection, would like to highlight that the increase in computation time and batch size cost is only of a small fraction as can be seen in Tables 3nd
4.3 IMPACT OF BATCH SIZE AND SEQUENCE LENGTH ON COMPUTATION TIME
In this subsection, we perform an ablation study to understand the impact of the batch size and sequence length training time. In particular, having reference the setting considered in
8

---

Under review as a conference paper at ICLR 2023
Table 3: Computation time for pre-training with different SSL approaches. Compared to W2V-2, our
approach requires 2.23x less GPU hours. With fraction of extra computation time, we can further boost the performance of W2V-2 using Sequentially Combined method.
Model
GPU Hours
No. of GPUs
Wall Clock Hours
W2V-2
2457
64
38
W2V-2
2376
8
297
Con-Contrrastive
1064
8
133
Sequentially Combined 3040 (2376+664)
8
380 (297+83)
Table 4: Batch size requirements for pre-training with different SSL approaches. Our results show
that Con-Contrastive SSL method requires smaller batch sizes as compared to W2V-2 approach. For non-contrastive method we use the batch size per GPU of 120 secs by leveraging the gradient accumulation of 3 steps each with batch size of 40 sees.
Model
No. of GPUs (V100's)
Batch size/GPU (secs)
Batch size (secs)
W2V-2
64
84
5376
W2V-2
8
84 (x8)
5376
Non-Contrastivve
8
40 (x3)
960
Sequentially Combined
8
40 (x3)
960
Table 5: Training cost (in terms of GPU hours) for Non-Contrastive SSL method with different batch
sizes and sequence lengths. Results are computed when training V100 GPUs with gradient ac- cumulation of 3 steps. We observe that increasing the batch size results in higher computation time than increasing the sequence lengths. Similarly, the impact on WER is reported in the Appendix.
Batch Size
GPU Hours
Seq Len 3s
Seq Len 5s
Seq Len 7s
24
704
1064
1483 OOM
48
1216
OOM
72
1696
OOM
OOM
Way2Vec-2.0, namely batch size 6 with approx. 15s in each sample resulting in lower amount of audio per batch of 1.4min. We expand our measurements from a single setting of batch size 24 with 5s in each sample with yields 2min of audio per batch different configurations of batch sizes 24, 48 and 72 as well as sequence lengths of 3s, 5s and 7s. We limit the exploration to maximum batch size of 72 and sequence length of 7s as allowed by our infrastructure (with NVIDIA V100 GPUs) without increasing the gradient accumulation steps (fixed at three as in other experiments).
As shown in T we observe that increasing the batch size results in a higher computation time than increasing the sequence lengths. This may be due to GPU contention at higher batch sizes, or that depending on the configuration of batch size and sequence length different kernels for the different operations are being selected by the backend. Moreover, given that the non-contrastive approach is based on siamese network format, with exponential moving average, there are twice the weights in memory as well as twice the number of forward passes which limits how much we can increase the batch size and the sequence lengths. Therefore, even with three step gradient accumulation (as used in all other experiments in the paper) we can in the table that the more demanding settings result in out of memory (OOM) on V100 GPUs having 32 GB memory.
5 CONCLUSION
This work made two contributions towards resource efficient speech representation learning for ASR. First, we investigated the potential of replacing established contrastive SSL approaches (e.g.. Wav2Vec-2.0 (Baevski et al., 2020)) with non-contrastive SSL method to reduce training time and GPU resources needed for training ASR systems. Second, we extended existing image based non-contrastive Barlow-Twins (Zbontar et al.) (021) method for speech representation modeling and more generally to arbitrary time varying data. Numerical results demonstrated a training speed up of 2.23 times as well as improved model performance (i.e., lower WER) via the proposed non- contrastive as opposed to contrastive approaches. Finally, our results demonstrated the benefits of sequentially combining contrastive and non-contrastive methods to further boost ASR performance.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. Common voice: A muasively-mullllingua speech corpus. In LREC'200 2020.
Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kri- tika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, al. XIs-r: Self-supervised cross- lingual speech representation learning at scale. In Interspeec" 22, 2022.
Alexei Baevski, Michael Auli, and Abdelrahman Mohamed. Effectiveness of self-supervised pre- training for speech recognition. arXiv preprint
2019.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. Wav2vec 2.0: A frame- work for self-supervised learning of speech representations. In Advances in Neural Information Processing Systems, 2020.
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning speech, vision and language. arXiv preprint arXiv:2202.03555. 2022.
Randall Balestriero and Yann LeCun. Contrastive and Non-Contrre Con-Contra Self-Supervised Learning Recover Global and Local Spectral Embedding Methods. arXiv e-prints, art. arXiv:2205. May 2022.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In International Conference on Computer Vision, 2021.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Computer Vision and Pattern Recognition, 2021.
Po-Han Chi, Pei-Hung Chung, Tsung-Han Wu, Chun-Cheng Hsieh, Yen-Hao Chen, Shang- Li, and Hung-yi yi Lee. Audio albert: A lite bert for self-supervised learning of audio representation. In S1 2021.
Yu-An Chung, Yu Zhang, Wei Han, Chung- Coang-Cheng&ooo Chiu, James Qin, Ruoming Pang, and Yonghui Wu. W2v-bert: Combining contrastive learning and masked language modeling for self- supervised speech pre-training. In ASRU'21, 2021.
Jacob Devlin, Ming-Was Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, 2019.
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist tempo- ral classification: labelling unsegmented sequence data with recurrent neural networks. In ICML. 2006.
Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukeuoglu, Remi Munos, and Michal Valko. Bootstrap your own latent new approach to self-supervised learning. In Advances in Neural Information Processing Systems, 2020.
Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang. Conformer: cavvolution-augmente transformer for speech recognition. Interspeech, 2020.
Kaiming He, Haoqi Fan, Yuxin WU, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Computer Vision and Pattern Recognition, 2020.
10

---

Under review as a conference paper at ICLR 2023
Dan Hendrycks and Kevin Gimpel. Gaussian arXiv:1606.08415, 2016.
enror
linear units (gelus).
arXiv preprint
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEBE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451-3466 2021.
Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee, and Fil- lia Makedon. A Survey on Contrastive Self-supervised Learning. arXiv e-prints, art. arXiv:201 1.00662, October 2020.
Jacob Kahn, Morgane Rivière, Weiyi Zheng, Evgeny Kharitonov, Qiantong Xu, Mazaré, Julien Karadayi, Vitaliy Liptchinsky, Ronan Collobert, Christian Fuegen, al. Libri- light: benchmark for asr with limited or no supervision. In ICASSP. 2020.
Jodi Kearns. Librivox: Free public domain audiobooks. Reference Reviews, 2014.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International
Conference on Learning Representations, 2015.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: denoising sequence-fo-sequueec pre- training for natural language generation, translation, and comprehension. CoRR, abs/1910.13461, 2019.
Shaoshi Ling and Yuzong Liu. Decoar 2.0: Deep contextualized acoustic representations with vector quantization. arXiv preprint arXiv:2012.0665) 2020.
Shaoshi Ling, Yuzong Liu, Julian Salazar, and Katrin Kirchhoff. Deep contextualized acoustic
representations for semi-supervised speech recognition. In ICASSP' 20, 2020.
Alexander Liu, Yu-An Chung, and James Glass. Non-autoregrressive predictive coding for learning speech representations from local dependencies. arXiv preprint (00.i:::011100044:0 2020a.
Andy T Liu, Shu-wen Yang, Po-Han Chi, Po-chun Hsu, and Hung-yi Lee. Mockingjay: Unsuper- vised speech representation learning with deep bidirectional transformer encoders. In ICASSP 20, 2020b.
Shuo Liu, Adria Mallol-Ragolta, Emilia Parada-Cabeleiro, Kun Qian, Xin Jing, Alexander Kathan, Bin Hu, and Bjoern W. Schuller. Audio Self-supervised Learning: A Survey. arXiv e-prints, art. arXiv:2203.01205, March 2022.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an ASR corpus based on public domain audio books. In ICASSP. 2015.
Daniel Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin Cubuk, and Quoc Le. Specaugment: A simple data augmentation method for automatic speech recognition. In Interspeech, 2019.
Alberto Gil Couto Pimentel Ramos, Abhinav Mehrotra, Nicholas Donald Lane, and Sourav Bhat- tacharya. Conditioning sequence-to-sequenee networks with learned activations. In International Conference on Learning Representations, 2022.
Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. Wav2vec: Unsupervised pre-training for speech recognition. In Interspeech, 2019.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In NeuRIPS, 2020.
Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. Voxpopuli: A large-scale multilingual speech corpus for representation leaming, semi-supervised learning and interpretation. In ACL'21, 2021.

---

Under review as a conference paper at ICLR 2023
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning. 2021.
Chunlei Zhang and Dong Yu. C3-DINO: Joint Contrastive and Non-contrastive Self-Supervised Learning for Speaker Verification. arXiv e-prints, art, arXiv:2208.0744 August 2022.
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX
MODEL PERFORMANCE IN TERMS OF CER
To examine the ASR model performance in terms of label prediction on low and high resource settings, we compute character error rate (CER) on LibriSpeech dev and splits. The results for these evaluations are presented in Tables and
Table 6: Model performance in terms of CER on the Librispeech dev/test sets when fine-tuned on LS-960. Both Con-Contr and Sequentially Combined SSL methods outperform the SOTA W2V-2 approach with marginal difference.
Model
LM
dev
test
clean
other
clean
other
W2V-2
None
1.30
4.38
1.31
4.30
Con-Contrustive
None
1.23
4.40
1.26
4.24
Sequentially Combined
None
1.16
4.41
1.19
4.20
W2V-2
4-gram 1.02
3.64
1.08
3.53
Con-Contrastive
4-gram
0.99
3.67
1.05
3.59
Sequentially Combined
4-gram
0.97
3.77
1.05
3.65
Table 7: Model performance in terms of CER for more fine-grained analysis in terms of label pre- diction on the Librispeech dev/test sets when fine-tuned on LS-100 and LL-10. Our Non-Contrast SSL method achieves marginal improvement in CER compared to the SOTA W2V-2 approach.
Model
Fine-tuning Data
LM
dev
test
clean
other
clean
other
W2V-2
10
None
3.90
10.15
3.99
10.12
Non-Contrastive
10
None
3.83
9.80
3.85
9.72
W2V-2
10
4-gram
2.70
8.61
2.82
8.59
Non-Contrastive
10
4-gram
2.62
8.5
2.63
8.40
W2V-2
100
None
1.96
7.98
2.09
7.66
Con-Contrastive
100
None
1.99
7.57
2.12
7.29
W2V-2
100
4-gram
1.47
6.80
1.68
6.60
Non-Contrastive
100
4-gram
1.51
6.61
1.64
6.28
IMPACT AUDIO SEQUENCE LENGTH DURING PRE-TRAINING ON CER AND WER
We focused our numerical experiments on investigating the advantages of the proposed method in terms of WER, batch size and GPU hours required with batches formed of 5 seconds audio samples. In addition to the GPU hours ablation study with 24, 48 and 72 batch sizes and 3 and 7 seconds audio clips, below we report the effect sequence lengths on CER and WER for and seconds. In particular, Table shows that shorter sequences such 3 seconds do have impact the quality of the learned representations, whereas longer sequences of 5 and 7 seconds led to similar and much better learned representation with small variations in the results.
13

---

Under review as a conference paper at ICLR 2023
Table Effect of sequence lengths on model performance. The results show that shorter sequences
such as 3 seconds negatively affect the quality of the learned representations. Whereas, longer sequences of and seconds led similar and much better learned representation with small variations in the results.
Sequence Length
Metric
dev
clean
other
clean
other
3
CER
1.68
4.74
1.68
4.61
5
CER
1.23
4.40
1.26
4.24
7
CER
1.28
4.34
1.31
4.21
3
WER
5.63
12.47
5.77
12.63
5
WER
4.01
11.26
4.29
11.15
7
WER
4.12
11.18
4.33
11.12
SEQUENTIALLY COMBINED SSL MODEL PERFORMANCE ON LOW-RESOURCE SETTINGS
To complement the results for low-resource settings presented in Table 2 we present the results of ASR model performance when pre-trained with Sequentially Combined method compared to other approaches in Table 9 The models were fine-tuned on LL-10 10 and LS-100 datasets. The evaluation performed the dev and test splits. The results show that sequentially combined SSL method achieves marginal WER improvement as compared to the SOTA W2V-2 and Non-contrastive SSL approaches.
Table 9: Sequentially combined pre-trained model's performance in terms of WER on the Lib- rispeech dev/test sets when fine-tuned on LS-100 and LL-10. Sequentially combined SSL method achieves marginal improvement in WER as compared to the SOTA W2V-2 and Non-contrastive SSL approaches.
Model
Fine-tuning Data
LM
dev
test
clean
other
clean
other
W2V-2
10
None
12.18
24.95
12.58
25.63
Non-Contrastive
10
None
12.17
24.17
12.19
24.81
Sequentially Combined
10
None
11.97
24.18
12.27
24.91
W2V-2
100
None
6.32
19.20
6.65
19.15
Non-C Contrastive
100
None
6.42
18.47
6.84
18.30
Sequentially Combined
100
None
6.48
18.03
6.81
18.01
14


Reviews:
[{"review_id": "zqey3k2AvB", "reviewer": "Reviewer_t7qA", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "While wav2vec-style contrastive learning has shown to be very successful for ASR, it requires a lot of resources and time for training. In the vision domain, Barlow Twins, a solution that naturally avoids collapse, has shown to be able to achieve better (or competitive) performance compared with contrastive learning (e.g., SimCLR) while using much smaller batch size. However, Barlow Twins style training is under-explored in the ASR/audio domain.\n\nThe authors applied revised Barlow Twins training to the wav2vec-2 framework (CNN feature extractor + Transformer), and achieved competitive performance compared to wav2vec 2.0, while reducing training time, GPU usage and improving convergence. \n", "summary_of_the_review": "At this moment, I’m inclined to recommend this paper, though the paper could be in a better shape with more clarification and ablation study. ", "strength_and_weaknesses": "The strength:\n— To my knowledge, this is the first work that applies Barlow-Twins methodology to learn representation for ASR; The adoption of BT into sequential representation learning is not trivial. Previous, I only find BT is applied for speaker recognition.\n\n— The authors show that the proposed methods are more resource efficient and achieve comparative performance. A) It improves convergence, b) it reduces training time, c) it significantly reduces GPU training times, d) it requires smaller batch size thus reducing memory requirements.\n\n— The authors also found that combining the proposed method with a contrastive learning approach is helpful.\n\n\nThe draft can be improved with some clarification and extra ablation studies:\n— Same as wav2vec 2.0, the proposed methods apply masks to the CNN-extracted features; Here, the mask is only used to distort the input (to my understanding), thus the authors can explore ways of distortions other than applying masks.\n\n— Regarding Time Unrolling and time merging losses: The proposed method chunks the audio into 5-seconds. It seems that for longer input or when using larger batch size/projection dimension, the calculation of the two losses can be much time-consuming (i.e., slower). Also, what’s the relative importance of the two losses? Maybe an ablation study with different w_U and w_M (static loss) could be interesting.\n \n— The authors show that further fine-tune wav2vec 2.0 with the proposed methods achieves better performance than both wav2vec 2.0 and the proposed non-contrastive training. What if the authors first train with the non-contrastive loss and then fine-tune with contrastive loss?\n\n— The “iteratively combined” solution is only applied to LS-960; It would be good to see the \"iteratively combined\" solution also applied on low-resource scenarios, and see its relative improvement.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "l8iRXUqYEzp", "reviewer": "Reviewer_hCQK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Wav2vec 2.0 is widespread among speech community and used extensively to pre-train speech models for later fin-tuning on the speech recognition task. The downsides of wav2vec 2.0 is necessity of huge resources to get state-of-the-art on Librispeech, hard to tune as there are a lot of hyper-parameters and also contrastive loss which is often hard to deal. To democratize pre-training pipeline to people, authors in this paper propose to use non-contrastive loss and adapt Barlow-Twins from vision domain. They propose simple trick where reshaping is done of 3d tensor to 2d: time is stacked either with batch dim or with feature dim and then standard Barlow-Twins loss from vision is computed. These two operations, time unrolling and merging, ensure that all frames are different and that all sequence are different. With experiments on Librispeech and different amount of supervision (10h, 100h, 960) authors demonstrate that proposed method is in the same ballpark or better than wav2vec 2.0 and more GPU-efficient (can be trained with less resources, smaller batches, 2.3x faster in GPU hours). Also non-contrasitve training can be done after contrastive training and improve further results.", "summary_of_the_review": "There are a lot of methods proposed in computer vision which can not readily be applied to other domains, e.g. sequences in speech and text. This works aims to adapt Barlow-Twins SSL technique to sequences and show that non-contrastive training can be on pair with contrastive training, e.g. wav2vec 2.0. Authors proposed to pack time either with batch dim or feature dim to compute then cross correlation matrices in Barlow-Twins. I like that there is direction developed in adapting some methods from computer vision and also developing non-contrastive methods to simplify and speed up the training. However, empirical comparison done in the paper raises a bunch of questions I specified in the main review (baselines are significantly worse than reported in original paper, beam search is very weak, etc.). Also there is no comparison with other developed speech specific non-constrastive methods like HUBERT, or more general framework as data2vec which is the most closed to considered Barlow-Twins in terms of architecture and EMA (but not loss).", "strength_and_weaknesses": "**Strength**\n- adoption of vision idea on Barlow-Twins to sequence domain\n- experiments with different amount of supervision (10h, 100h, 960h) of Librispeech\n- ablation on usage non-contrastive pre-training after contrastive one\n\n**Weaknesses**\n- Beam search is very shallow with not proper hyper-parameters. Yes, authors say about more practical use case, but beam search of CTC is still cheap, and it is research paper where we need to understand all limits and boundaries, especially in low-resource where contribution of language model is huge. I am ok of having several points of decoding for restricted setup and full setup. There are a lot of offline applications where we can use better / larger language models and more expensive search. With current setting it looks more artificial than practical and results obtained with language model are very-very bad, so hard to compare what exactly we are learning with SSL and how language is incorporated there. Because of transformer architecture we could learn some phoneme-level LM during SSL (and this is happening). There could be different effects of LM in the current paper in wav2vec as authors are doing 5s crop for input audio in SSL.\n- \"Nevertheless, trained on LibriSpeech dataset, our baselines are less than 1% absolute WER difference only when compared with SOTA solutions, e.g., Wav2Vec-2.0 (Baevski et al., 2020) which are the result of extensive HPO.\" - when I compare results for 10h, 100h, and 960 (Tables 1, 3, 4) of supervision with wav2vec 2.0 paper I don't see less than 1% absolute gap, it is much larger. I see a problem right now exactly with hyper-parameter search, reimplementation (or rerunning) wav2vec 2.0 baseline which is very weak and not consistent with what people reported (especially in the context that pretrained model is used and only fine-tuning is done). Also official recipe is 20k/80k fine-tuning steps only, not 400k as authors suggested to do in the paper. \n- This also raises questions on the efficiency: how do authors computed numbers for training time/gpu time for wav2vec 2.0? from 400k fine-tuning? from pre-training + fine-tuning? \n- I think the main weakness is reproduction of wav2vec 2.0 baseline which is far away from what is reported in prior work. Then comparison with proposed new methods looks very weird with these numbers. Next, results in Table 1, 3 for decoding with LM looks in some places very weird as not improvement is obtained even. Especially in 10h regime LM should help by a lot as we learn the phonemes / letters but very bad in word reconstruction and wav2vec 2.0 should gain huge boost with LM which is not the case authors reported. Overall results looks very weird and strange for me.\n- absence of baselines on non-contrastive training, e.g. HUBERT, data2vec. (maybe at least comparison on speed in case authors can reproduce results from the paper)\n\nOverall I feel this paper needs more work on experimental side to prove usefulness of the method. Right now it is more like proof of concept that Barlow-Twins loss can be trained with speech. No extra ablations on the parameters are given and how they are important or not (e.g. weak/strong augmentation, loss weights, EMA decay, etc.)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "jdgqWuU2d39", "reviewer": "Reviewer_5kcU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a non-contrastive pre-training method based on Barlow-Twins as an alternative to the wav2vec model which uses contrastive pre-training.  This work aims to improve the computational efficiency (memory and pre-training time) of the wav2vec model. The paper introduces two extensions to Barlow-Twins loss to account for the time dimension of an acoustic feature sequence.   Given a batch with $B$ input sequences each with time steps $T$ in embedding space of size $F$, first, time unrolling stacks all time steps from all inputs to get embeddings of shape $(B * T, F)$. The cross-correlation matrix of shape $F \\times F$ is encouraged to be identity to ensure feature diversity.  Next, time merging concatenates features from all time steps to obtain embeddings of shape $(B, T * F)$. Similarly encouraging the cross-correlation of shape $B \\times B$ to be identity ensures that the model does not collapse. Experiments are conducted on Librispeech dataset to evaluate the efficacy of the proposed approach.", "summary_of_the_review": "This work introduces a non-contrastive method to improve the computational efficiency of SSL on speech sequences. I have recommended rejection primarily due to the experiments with confounding factors and lack of proper ablation as explained above. This makes it hard to understand the source and extent of performance gains.\n", "strength_and_weaknesses": "**Strengths**:\n  - The paper is clear and the proposed approach is easy to follow. Performance on different subsets of the Librispeech dataset shows that non-contrastive SSL can achieve similar or even better WER than the contrastive models. The models also require smaller batch sizes resulting in fewer memory and GPU resources to train.\n\n  - I appreciate the figure that explains clearly the difference between standard non-contrastive SSL and the proposed extension to speech. All details are presented to be able to reproduce the work.\n\n**Questions and Weakness**:\n   \n  - Prior work: Non-contrastive methods for SSL in speech are not well discussed. There are quite a few generative/predictive models which have been investigated for SSL with speech. Some examples include MockingJay, Audio ALBERT, TERA, etc.  More exhaustive list can be found in [1]. data2vec [3] is a very close and relevant non-contrastive baseline that requires discussion.\n\n  - I do not think that the claim \"Non-contrastive SSL ASR yields a $2.3\\times$ training speed up compared with contrastive SSL\" is completely accurate.  It seems that the GPU hours were reported from the [2].  This may not be comparable because (a) the GPU hours can vary significantly depending on the setup. [4] reports the W2V base model training to 1764 hours when trained on 8 GPUs with gradient accumulation. It would be better to re-estimate the training times for W2V architectures on the same number of GPUs (8) used by non-contrastive models and by using the most recent w2v code for a fair comparison.  This could be done by running pre-training for a reduced number of steps.\n    \n  - While it is shown that non-autoregressive models require smaller batch size, the claim \"Non-contrastive SSL requires fewer GPUs\" is not accurate as gradient accumulation can be used for training using the same number of GPUs.  This also leads to better GPU hours.\n    \n- Issues with experimentation\n  - For the case of iteratively combined pre-training, the paper only evaluates the setting of non-contrastive pre-training starting from a pre-trained contrastive model. This model has effectively been pre-trained for $650$K iterations. This needs to be compared against the contrastive and non-contrastive models trained for $650$K iterations to understand the performance gains/losses.\n\n  - Confounding factors and lack of proper ablation make it hard to understand the source of the performance gains. Some examples are listed below:\n    - Pre-training time is reported from the wav2vec paper which was trained on $64$ GPUs. The configuration of $8$ GPUs with gradient accumulation would have been a comparable configuration.\n    - Audio is cropped to 5 seconds: This would lead to shorter sequences to the Transformer encoder than the wav2vec model. Given the quadratic complexity of the transformer, the extent of performance gains from (a) shorter sequence and (b) smaller batches is not clear. An ablation with fewer pre-training steps but isolating these factors would have made it easier to understand the impact.\n    - Models are fine-tuned for $400$K steps. This is substantially more than the wav2vec models which are fine-tuned for 80K and 20K steps on LS-100 and LL-10  datasets respectively. It is not clear if the gains are only due to non-contrastive pre-training or also due to longer fine-tuning. Furthermore, this could result in slower fine-tuning for the non-contrastive model.\n  - Baselines: While the paper compares against the wav2vec model, a relevant baseline would have been the data2vec [3] model which follows a non-contrastive setup as well.\n\n**References:**\n\n[1] Self-Supervised Speech Representation Learning: A Review. IEEE Journal of Selected Topics in Signal Processing \n\n[2] wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations. NeurIPS 2020\n\n[3] data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language. ICML 2022\n\n[4] Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognition. ICASSP 2022", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_Dy21RMsytS", "reviewer": "Reviewer_5Dwq", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper describes non-contrastive learning based self-supervised pre-training approach for the speech recognition task. As opposed to the popular methods such as Wav2vec2, Hubert, etc. which use contrastive learning for pre-training, this paper describes how non-contrastive methods such Barlow-Twins for the audio pre-training. \n\nThe main contribution of the paper is:\n(i) Extend Barlow-Twins (Siamese Network) training for audio modality (with additional time axis) with new loss functions for that additional axis.", "summary_of_the_review": "Although the paper has some interesting ideas for improving the efficiency of self-supervised training for speech recognition, it lacks an extensive understanding of the various design choices made in the paper. While the paper is interesting, by adding certain key details it would be easy to understand the positioning of the non-contrastive methods in the self-supervised pre-training landscape.\n\n", "strength_and_weaknesses": "Strengths:\n\n(i) Paper clearly shows the extension of Barlow-Twins methods for audio modality with intuitive losses. \n(ii) The empirical results show the method's efficiency (compute resources) compared to the Wav2vec2-like pre-training.\n\nWeakness:\n\n(i) In the section (augmentation in latent space), it says that masks are applied with a probability of 0.05 and mask length of 10 for the target model and 0.1 and 20 for the online model. \n- What is the fraction of input unchanged in the output of both the target model and online model representations? \n- Time unrolling loss compares similarity across time-frames for the same frames and if most of the temporal frames are unchanged then the solution would be trivial. I am curious in understanding the effect of augmentation on the overall objective, which is not described in the paper.\n- In images, the augmentations(cropping, resizing, flipping, etc.) are more meaningful, which is not a luxury in speech. Due to the fixed audio duration for each batch (5 ms), perturbations in the frequency domain cannot be carried out, which further limits the augmentation space.\n\n(ii) I do not see the comparison of static and dynamic scaling for the loss functions. Is the idea of using a stop gradient for dynamic loss novel? If not the citation for that is missing. \n\n(iii) Paper states that for the pre-training the audio utterances are cropped into 5 seconds. (Wav2vec2 uses 15.6 seconds). \n- Was this design choice considering the GPU size? \n- What is the effect of having longer utterances on the overall pre-training objective?\n- Because Non-contrastive learning methods don't require larger batch sizes, does having longer utterances help or deteriorate the pre-training performance?\n\n(iv) I find *Iteratively Combined* baseline overtly powerful because of the number of customization steps (400K for wav2vec2 + 250k for the Non-contrastive method), as compared to W2V2 and Non-Contrastive method. \n- I would suggest running each of the methods to a fixed number of steps and then comparing. Since the number of parameters are the same, this would give a more accurate picture of the efficacy of each of these methods.\n\n(v) Results of other self-supervised methods such Hubert, Decoar, etc. should be placed to understand the relative strength of the non-contrastive method. \n\n(vi) Ablation of both the Time Unrolling and Time Merging loss is missing. Very curious to know what is the impact of time merging loss on the overall objective.\n\nMinor:\n(i) A reference missing in line 3 of the `Evaluation with High-Resource Labeled Data` paragraph.\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject