Year: 2021

Abstract:
We introduce a new approach for speech pre-training named SPIRAL which works by learning denoising representation of perturbed data in a teacher-student framework. Specifically, given a speech utterance, we first feed the utterance to a teacher network to obtain corresponding representation. Then the same utterance is perturbed and fed to a student network. The student network is trained to output representation resembling that of the teacher. At the same time, the teacher network is updated as moving average of student’s weights over training steps. In order to prevent representation collapse, we apply an in-utterance contrastive loss as pre-training objective and impose position randomization on the input to the teacher. SPIRAL achieves competitive or better results compared to state-of-theart speech pre-training method wav2vec 2.0, with significant reduction of training cost (80% for BASE model, 65% for LARGE model). Furthermore, we address the problem of noise-robustness that is critical to real-world speech applications. We propose multi-condition pre-training by perturbing the student’s input with various types of additive noise. We demonstrate that multi-condition pre-trained SPIRAL models are more robust to noisy speech (9.0% 13.3% relative word error rate reduction on real noisy test data), compared to applying multi-condition training solely in the fine-tuning stage. The code will be released after publication.

heading: 1 INTRODUCTION
text: Industrial-scale automatic speech recognition (ASR) systems are usually trained with ten-thousands of hours of hand-transcribed speech data (Galvez et al., 2021). However, labeling speech data is expensive and time-consuming, especially for languages with small speaker populations, or for specific domains (e.g., legal, financial, scientific).
Recently, methods of utilizing unlabeled speech data to improve speech recognition system have achieved remarkable progress. Amongst them, self-training (Manohar et al., 2015; Kahn et al., 2020a; Synnaeve et al., 2020a; Chen et al., 2020b; Xu et al., 2020; Park et al., 2020b; Xiao et al., 2021), also known as pseudo-labeling, starts by training an ASR model with labeled speech data, which is referred to as teacher model. Then the teacher model, usually combined with a language model (LM), is used to produce pseudo-labels for unlabeled speech data. Finally, the labeled data and the pseudo-labeled data are combined to train a new model, which is referred to as student model. The process is repeated by taking the student model as the teacher in next iteration. Another line of work is speech pre-training (van den Oord et al., 2019; Chung & Glass, 2020; Wang et al., 2020; Baevski et al., 2020b; Liu et al., 2020). Pre-training learns speech representation from unlabeled data in a self-supervised way. The pre-trained model is then fine-tuned on the labeled data. Self-training and pre-training are complementary as shown in recent work (Xu et al., 2021; Zhang et al., 2020).
In this paper, we introduce a new speech pre-training method which works by learning denoising representation of perturbed data with the teacher-student framework, named as Self-supervised Perturbation-Invariant Representation Learning (SPIRAL). Compared to state-of-the-art speech pretraining methods such as wav2vec 2.0 (Baevski et al., 2020b) and HuBERT (Hsu et al., 2021), our method allows end-to-end training with a single contrastive loss, and without relying on discrete unit
discovery techniques such as vector quantization (Jégou et al., 2011; Baevski et al., 2020a;b) or iterative clustering process (Hsu et al., 2021). We apply multi-condition training with SPIRAL (Seltzer et al., 2013; Ko et al., 2015) to improve noise-robustness for the downstream speech tasks.
SPIRAL is motivated by the observation that human tolerates speech perturbations or distortions fairly well. For example, people can communicate effectively in a noisy environment, or over a distorted telephone channel. Therefore, we hypothesize that by learning representation invariant to perturbation, the model will learn high-level representation which can enhance speech applications.
To learn perturbation-invariant representation in a self-supervised way, we employ a teacher-student framework similar to Tarvainen & Valpola (2017). During pre-training, given a speech utterance, we guide the student network which consumes the perturbed utterance to learn from the teacher network which consumes the clean utterance. The student is trained to produce denoised representation of the perturbed utterance similar to teacher’s representation of the clean utterance. Meanwhile, the teacher, which shares the same model architecture with student, is updated as moving average of the student’s weights over past training steps.
We apply the in-utterance contrastive loss to avoid model collapse to trivial constant representation (Chopra et al., 2005). As speech utterance are sequential data, there is another possible trivial solution which we call positional collapse. Positional collapse occurs when the student “cheats” by exploiting position correlation in teacher’s representation to minimize the loss, while ignoring the content of the input utterance. To prevent positional collapse, we propose position randomization by adding random number of paddings on both sides of input utterance to the teacher.
Large-scale speech pre-training is computationally demanding. To reduce computation cost, we adopt a gradual down-sampling strategy in SPIRAL model, which has been verified effective in speech recognition literatures with negligible performance degradation (Peddinti et al., 2018; Han et al., 2020b; Huang et al., 2020). We also speculate that aggressive down-sampling helps to remove redundancy in speech.
To evaluate the effectiveness of SPIRAL, we conduct experiments on LibriSpeech and Libri-Light datasets. By training a small convolutional classifier on the representation of a frozen SPIRAL model, we can achieve WER of 3.5% and 6.4% on Librispeech test-clean and test-other respectively. SPIRAL achieves competitive or better results compared to state-of-the-art speech pre-training methods, while being much more training-efficient. We also demonstrate that multi-condition pretrained SPIRAL are more robust to noisy speech with 9.0% - 13.3% relative word error rate (WER) reduction on real noisy test data from ChiME-3 (Barker et al., 2015), compared to the model applying multi-condition training solely in fine-tuning stage.

heading: 2 RELATED WORK
text: Mean Teacher (MT) (Tarvainen & Valpola, 2017) proposes using a student network to learn from a teacher network which is the moving average version of the student in the semi-supervised learning setting. The authors apply a supervised loss for labeled data and a consistency loss between teacher and student predictions for unlabeled data. However, direct application of MT to self-supervised learning leads to representation collapse (Grill et al., 2020).
Noisy student training (NST) (Xie et al., 2020; Park et al., 2020b) is a self-training method. NST demonstrates the importance of the aggressive injection of noise into the student. Although not emphasized, no noise is injected into pseudo-labeling process of the teacher. We consider our work as an extension of self-training approach to the self-supervised learning regime. Instead of using the teacher to provide pseudo-labels, we utilize the teacher for pseudo-reference representation.
Denoising autoencoders (Vincent et al., 2008) learn to recover a clean input from a corrupted version. However, speech data contain redundancy which is irrelevant to some speech applications such as speech recognition. Previous work (Baevski et al., 2019) shows that speech pre-training by recovering masked input speech features is not effective. In SPIRAL, we instead enforce latent representation of a corrupted input to resemble that of the corresponding clean input.
Bootstrap Your Own Latent (BYOL) (Grill et al., 2020) is a self-supervised image representation learning method. The method is based on a teacher-student framework similar to MT. The authors refer to student network as online network and teacher network as target network. They observe
that naive application of MT to self-supervised learning leads to trivial constant representation. They prevent the representation collapse by appending a predictor to the student network. The theory behind is under investigation (Chen & He, 2021; Tian et al., 2021). Our method draws inspirations from BYOL and shares the similar architecture, but there are crucial differences. Instead of learning a single global representation for an image as in BYOL, SPIRAL learns a sequence of representation for an utterance. We aim for sequence applications such as speech recognition. In our preliminary experiments, we observe that appending a predictor to student network is not sufficient to prevent trivial constant representation for sequential representation learning. We use in-utterance contrastive loss (Baevski et al., 2020b) combined with input position randomization to successfully avoid representation collapse. We still keep the predictor in SPIRAL, but only for the sake of performance improvement from our observation. Another difference is that BYOL does not perform representation denoising. BYOL applies perturbation, which they call augmentation, to both the inputs of the teacher and the student. We demonstrate that representation denoising is crucial for speech pre-training. When perturbation is applied to the teacher’s input, the effectiveness of speech pre-training degrades drastically.
Wav2vec 2.0 (Baevski et al., 2020b) is a self-supervised speech representation learning method which belongs to the masked prediction family. Masked prediction methods are effective for text pre-training (Devlin et al., 2019), but not for speech pre-training when naively applied (Baevski et al., 2019). The reason is that speech data contains redundancy such as speaker information, pronunciation variations, which are irrelevant to the semantic meaning of the utterance. To overcome this problem, wav2vec 2.0 perform masking in intermediate latent space and performs target discretization with a differentiable quantization scheme. However, quantization leads to a more complex model by introducing additional hyper-parameters and an additional diversity loss. SPIRAL does not utilize quantization, and still achieves competitive performance compared to wav2vec 2.0. We hypothesize that aggressive down-sampling and learning by matching output representation may help to remove redundancy from the learned representation. We leave the investigation of whether target discretization could further improve SPIRAL for future work.
Liang et al. (2018) demonstrates that under the supervised learning setting, enforcing noise-invariant representation by penalizing difference between clean and noisy data improves ASR model accuracy.

heading: 3 METHOD

heading: 3.1 SELF-SUPERVISED PERTURBATION-INVARIANT REPRESENTATION LEARNING
text: (SPIRAL)
Figure 1 shows the diagram of SPIRAL in the pre-training stage, where we use two neural networks, a student Fθ and a teacher Fθ′ . The weights of the teacher θ′ is the moving average of the weights of the student θ. At step t, the weights of the teacher θ′t are updated as θ′t ← αtθ′t−1 + (1− αt)θt, (1) where αt determines the rate of weight updates. Given a speech utterance X = (x1, . . . ,xT ) of length T , the student takes a perturbed version X̃ = s(X) = (x̃1, . . . , x̃T ) as input where s(·) is a perturbation function. The output of the student is a representation sequence Z = F (X̃; θ) = (z1, . . . ,zT ). The teacher takes the same utterance without perturbation as input and output another representation sequence Z ′ = F (X; θ′) = (z′1, . . . ,z ′ T ). For each representation zi ∈ Z, the student is trained to match the teacher’s representation z′i at the same position amongst k distracting samples. The distracting samples are randomly drawn from other positions of the same utterance in Z ′, which is found to be more effective than samples drawn from an entire batch of utterances (Baevski et al., 2020b). The in-utterance contrastive loss is defined following Sohn (2016); Wu et al. (2018) as,
L = − T∑ i=1 log exp(φ(zi, z ′ i)/κ)∑ j∈Di exp(φ(zi, z ′ j)/κ) , (2)
where φ(a,b) = aTb/‖a‖‖b‖ is cosine similarity, Di is the set of indices of distractors for the i-th position, and κ is the temperature parameter.
However, applying in-utterance contrastive loss could cause a kind of representation collapse which we refer to as positional collapse. Contrastive candidates are sampled based on their positions in
utterances. When a teacher’s representation z′i is correlated with its position i (e.g., correlation introduced by positional encoding in Transformer), the student could exploit this correlation to generate its representation zi solely based on the position index i, while ignoring content of the input. In this case, the model does not learn meaningful representation of the input content. Therefore, we prevent positional collapse by randomizing positions of teacher’s representation. In particular, we add random number of padding data at both ends of the input to the teacher to randomly shift the position information for each output representation z′i. The student thereby is unable to exploit the spurious position information to minimize the contrastive loss. Note that when calculating the contrastive loss, we exclude the corresponding representation of the padded data.

heading: 3.2 MODEL ARCHITECTURE
text: As illustrated in Figure 2, student Fθ is composed of an encoder f(·), a projection head g(·) (Chen et al., 2020a) and an optional predictor q(·) (Grill et al., 2020), i.e., Fθ = (f ◦g◦q)(·; θ). The teacher Fθ′ has the same structure expect that it has no predictor, Fθ′ = (f◦g)(·; θ′). The encoder consists of two blocks. In each block, we first apply temporal convolutions to perform down-sampling, followed by Transformer (Vaswani et al., 2017) with convolutional relative position encoding (Baevski et al., 2020b). Each convolution is followed by layer normalization (LN) (Ba et al., 2016) and ReLU. For the projection head, we apply a simple linear layer. The predictor consists of two layers of temporal convolution and a linear layer. The convolutions are followed by batch normalization (BN) (Ioffe & Szegedy, 2015) and ReLU.
During pre-training, we add computation noise to both the student and the teacher by applying dropout (Srivastava et al., 2014) and LayerDrop (Fan et al., 2020) in Transformer. We use the same dropout and LayerDrop rates for the student and the teacher.

heading: 3.3 ADAPTIVE SPECAUGMENT
text: We apply adaptive SpecAugment similar to Park et al. (2020a) as the primary perturbation method. Along either time or frequency dimension, we sample uniformly a certain proportion p of all timesteps to be start indices and mask the subsequent consecutive L time-steps. The masked time-steps is filled with zeros along frequency dimension. Along time dimension, we use Gaussian noise as masking values to avoid numerical problems for LN (Park et al., 2020a).

heading: 3.4 MULTI-CONDITION PRE-TRAINING
text: For noise-robust pre-training with SPIRAL, we perturb input of the student with various types of additive noise. We consider this technique as an implementation of multi-condition training (MCT) (Seltzer et al., 2013) in self-supervised setting. Specifically, for each input utterance to the student, we sample noise clips from a noise dataset, and mix the noise clips with the whole utterance by addition in time-domain. We first uniformly sample a signal-to-noise ratio (SNR) from a predefined range for each utterance. Then we scale the noise volume according to the required SNR. In our preliminary experiments, we found that applying additive noise alone as perturbation degrades performance. Therefore, we apply additive noise perturbation together with adaptive SpecAugment.

heading: 3.5 MODEL FINE-TUNING
text: After pre-training, we take the encoder from the student model in SPIRAL and add a randomly initialized convolutional classifier on top of it. The convolutional classifier is composed of two layers of convolution, followed by LN and ReLU, and a linear output projection. The convolution filters consist of 512 channels with kernel width of 5.
We fine-tune the model with connectionist temporal classification (CTC) (Graves et al., 2006) objective for speech recognition. We use 1024 sub-words as output units. The sub-words are generated from training transcripts of LibriSpeech with SentencePiece (Kudo & Richardson, 2018).
We further investigate SPIRAL’s ability to learn high-level representation of speech during pretraining. In addition to whole-model fine-tuning, we apply frozen fine-tuning. We freeze the pretrained parameters and only fine-tune the convolutional classifier which can only perform local classification due to limited receptive field.

heading: 4 EXPERIMENTAL SETUP

heading: 4.1 DATA
text: For pre-training, we use the 960-hour training data (ignoring the labels) from LibriSpeech (Panayotov et al., 2015)(LS-960), or 60k-hour unlabeled audio data from Libri-Light (Kahn et al., 2020b) (LL-60K). For Libri-Light, we segment the data using official tools with a threshold of 16s, resulting in 46.9k hours of data. The two datasets are both derived from English audiobooks from LibriVox project1. For ASR fine-tuning, we apply 100-hour subset (train-clean-100) as low-resource labeled data and entire LS-960 with labels as high-resource labeled data, both from LibriSpeech.
For multi-condition training, we use the noise dataset from Reddy et al. (2021). The dataset consists of 181 hours of noise data with about 150 noise types and 70,000 clips. We shuffle and split the noise data with a ratio of 8:1:1, which are used for training, synthesizing noisy dev-sets and synthetic noisy test-sets (results in Appendix A.1) respectively. SNRs of speech mixtures are set from 0 to 30 dB. We evaluate on real noisy data test set from CHiME-3 (Barker et al., 2015), which is comprised of speech data recorded in real noisy environments (bus, cafe, pedestrian area, and street junction). The data are recorded with a microphone array composed of multiple microphone channels located at different positions of a tablet, and a close-talking microphone.
1https://librivox.org/

heading: 4.2 TRAINING SETUPS
text: We apply 128-dimensional log-mel filterbank extracted with 20 ms window and 10 ms stride as the input acoustic feature. We experiment with BASE model and LARGE model configurations as shown in Table 1. The numbers of parameters are comparable to wav2vec 2.0 BASE and LARGE models correspondingly. For SpecAugment, we set p = 0.025 and L = 20 for time-dimension mask, and p = 0.02 and L = 20 for frequency-dimension mask.
In pre-training, we optimize with Adam (Kingma & Ba, 2015) optimizer, warming up the learning rate for the first 8% of updates to a peak of 3e-3. Then the learning rate decays to 0 with a cosine schedule. The moving average update rate αt of teacher’s weight also follows a cosine schedule (Grill et al., 2020). We increase αt from 0.995 to 1.0 and from 0.990 to 0.999 for BASE and LARGE models respectively. We train the BASE model with batch size of 24 per GPU for 200k steps on 16 V100 GPUs, which takes about 1.3 days. For the LARGE model, we train with batch size of 20 per GPU for 500k steps on 32 V100 GPUs, which takes about 7.25 days. As shown in Table 2, there is a significant reduction of training cost (GPU days) compared to wav2vec 2.0 (Baevski et al., 2020b). SPIRAL requires 80% and 65% less training cost for BASE and LARGE respectively. Note that mix-precision training is not applied for SPIRAL yet.
For fine-tuning, we optimize with Adam and a tri-state rate schedule where the learning rate is warmed up for the first 10% of updates to 3e-5, held constant for the next 40% and then linearly decayed to zero following Baevski et al. (2020b). We fine-tune BASE and LARGE with batch size of 14 and 18 per GPU respectively on 8 GPUs for 80k steps on train-clean-100. We fine-tune LARGE with batch size of 10 per GPU on 16 GPUs for 320k steps on LS-960. We apply SpecAugment for whole-model fine-tuning but not for frozen fine-tuning. For multi-condition pre-training and fine-tuning, we randomly perturb each utterance with additive noise with 50% probability before applying SpecAugment. SNR is uniformly sampled from 0-30 dB.

heading: 4.3 LANGUAGE MODEL AND DECODING
text: We use a word-level Transformer LM (Baevski & Auli, 2019) trained on the Librispeech LM corpus which is identical to Synnaeve et al. (2020b). We perform grid search for decoding parameters and choose the best parameters according to performance on dev-other with beam 50. The final test performance is measured with beam 500. We use the beam search decoder of Pratap et al. (2019).

heading: 5 RESULTS

heading: 5.1 EVALUATION UNDER LOW-RESOURCE AND HIGH-RESOURCE LABELED DATA SETTINGS
text: We first evaluate our method under a low-resource ASR setting in which we fine-tune the models with 100-hour LibriSpeech data (train-clean-100). The results are shown in Table 3. We evaluate a BASE model pre-trained with 960-hour LibriSpeech (LS-960) and a LARGE model pre-trained with Libri-Light (LL-60K). The frozen BASE model performs well, achieving a WER of 13.0% on testother, which is on par with wav2vec 2.0 BASE. This suggests that SPIRAL indeed learns meaningful high-level representations in a self-supervised way. When we fine-tune the whole BASE model, the model achieves WER of 5.4% and 11.2% on test-clean and test-other respectively, outperforming wav2vec 2.0 BASE with 11.5% and 15.8% relative WER reduction.
However, when decoding with a Transformer LM, the gain of SPIRAL is smaller than that of wav2vec 2.0 and leads to inferior performance. The transformer LM is word based, but the SPIRAL models are operating in subword level. The mapping of words to subwords is not unique. This could hinder beam search performance. We expect that subword-level LM and more decoding parameter tuning can close the performance gap.
The SPIRAL LARGE model consists of more parameters and is pre-trained with more data. The model without LM achieves WER of 3.3% and 6.3% on test-clean and test-other respectively. The significant improvement of LARGE over BASE demonstrates the scalability of SPIRAL. The results of SPIRAL LARGE are competitive to wav2vec 2.0 LARGE. This is encouraging, as SPIRAL LARGE only takes 35% of training cost of wav2vec 2.0 LARGE.
We further evaluate SPIRAL LARGE pre-trained with Libri-Light (LL-60K) under a high-resource ASR setting with 960-hour LS-960 as fine-tuning data. As shown in Table 4, the LARGE model without LM achieves WER of 2.2% and 4.6% on test-clean and test-other respectively, which are
on par with the wav2vec 2.0 LARGE model. We again observe the performance improvement of SPIRAL with LM is smaller than wav2vec 2.0. We except to close the performance gap with the mentioned LM decoding refinement. We note that the supervised models and the noisy student model (Park et al., 2020b) in Table 4 are autoregressive models. Our models are fine-tuned with CTC objective which is non-autoregressive and generally inferior to autoregressive models. We use CTC objective for its simplicity and comparability to previous speech pre-training methods.
We consider SPIRAL as a preferred alternative to wav2vec 2.0 given that SPIRAL only requires 20%−35% computation cost of wav2vec 2.0. We expect further efficiency improvement when we implement mix-precision training for SPIRAL.

heading: 5.2 NOISE-ROBUST PRE-TRAINING
text: To evaluate noise-robustness of the pre-trained models, we compare the effects of applying multicondition training (MCT) in pre-training or fine-tuning stages of SPIRAL. The results are shown in Table 5. The vanilla SPIRAL BASE model and wav2vec 2.0 BASE model deteriorate with significantly higher WER on noisy test data.
On real noisy test speech data in CHiME-3 for different microphone channels (ch), SPIRAL with multi-condition pre-training significantly improves speech recognition performance. Compared to the model applying MCT solely in fine-tuning, applying MCT both in pre-training and fine-tuning achieves 12.4%, 13.3% and 9.0% relative WER reduction for ch 1, 5 and 2 respectively. There is smaller performance improvement of 3.8% relative WER reduction for ch 0, which is a close-talking microphone with the highest SNR. We note that ch 2 faces backwards to the speaker. SNR of the recordings from ch 2 is the lowest, leading to high WER.
We note that other pre-training methods including wav2vec 2.0 may benefit from multi-condition training, which are worth for further investigation.

heading: 5.3 ABLATIONS

heading: 5.3.1 INPUT PERTURBATION AND COMPUTATION NOISE OF TEACHER
text: SPIRAL learns denoised representation of perturbed data. By default, we only apply perturbation to the input of the student. An alternative method is to perturb both inputs of the teacher and the student, and optimize consistency between their representations (Grill et al., 2020; Chen & He, 2021). We conduct experiments to evaluate the effects of perturbing the input and adding computation noise (dropout and LayerDrop) to the teacher. The results are shown in Table 6. The results suggest that applying SpecAugment to teacher’s input degrades performance significantly. Performance degradation decreases but is still significant with lower ratio and width of the masks. This supports the necessity of representation denoising, and our view of SPIRAL as an extension of self-training in which teacher network are fed with clean input. The results also support applying computation noise to teacher during pre-training. There is a 15.9% relative WER reduction with computation noise. This may be linked to Gal & Ghahramani (2016).

heading: 5.3.2 EFFECTS OF PREDICTOR AND PROJECTION HEAD
text: We do ablation studies to understand the role of predictor and projection head in SPIRAL. The results are shown in Table 7. When removing the predictor from the student, we observe performance degradation, but representation collapse does not happen. In the architectures relying on predictor to prevent collapse (Grill et al., 2020; Chen & He, 2021), applying batch normalization (BN) in the predictor is essential. While in SPIRAL, we observe that BN in the predictor can be replaced by layer normalization (LN) with a smaller performance improvement. When the predictor is removed, we observe performance improvement by applying a convolutional projection head. The convolutional projection head is composed of a temporal convolution layer with LN and ReLU, and a linear layer. But applying convolutional projection head to the model with a predictor, there is no further performance improvement. This suggests that convolutional projection head and predictor play a similar role in SPIRAL, and they are not complementary.

heading: 6 CONCLUSION
text: We presented SPIRAL, a new approach to speech pre-training by learning denoising representation of perturbed data with a teacher-student framework. SPIRAL can learn high-level speech representation in self-supervised way. Training a small convolutional classifier on frozen representation of SPIRAL achieves WER of 3.5% and 6.4% on Librispeech test-clean and test-other respectively. We show that SPIRAL achieves competitive or better results compared to state-of-the-art speech pre-training methods, with significant reduction of training cost. We investigate multi-condition pre-training and demonstrates that multi-condition pre-training is more effective than solely applying multi-condition training in the fine-tuning stage. We presume SPIRAL as a general pre-training method, which can apply to other modalities such as images and text. We leave it for future work.

heading: A APPENDIX
text: A.1 PERFORMANCE ON SYNTHETIC NOISY DATASET
On the synthetic noisy dataset (NS-Librispeech) with matched SNR range (0-30 dB) of the training data, SPIRAL pre-trained and fine-tuned with MCT is more effective than applying MCT solely in fine-tuning. We observe 5.3% and 9.1% relative WER reduction on synthetic noisy test-clean and test-other sets respectively.

Review summaries:
This paper describes a self supervision training approach to pretrain speech encoders.  This is related to recent work on wav2vec 2.0, and SimCLR.  The contrastive loss is similar to wav2vec 2.0, while the invariance to perturbation (i.e. augmentation) is akin to SimCLR. Performance is competitive with wav2vec 2 on Librispeech.

The paper proposes a new speech pre-training approach named SPIRAL.  The proposed method is trained by learning representation denoising of perturbed data with the teacher-student framework in a self-supervised manner.  The motivation of the method is to learn the representation that is invariant to perturbation so that the learnt representation is a high-level one (e.g. carrying content information) that can enhance the downstream speech applications.  Compared to the state-of-the-art speech pre-training method wav2vec 2.0, the proposed method can achieve competitive or better results but with a significant reduction of the training cost.  The proposed method is also able to deal with the noise-robustness problem.

The paper introduces SPIRAL, a new method for self-supervised pre-training for speech. SPIRAL is based on the teacher-student framework similar to Mean Teacher (Tarvainen and Valpola, 2017) and BYOL (Grill et al., 2020) where the teacher’s weights are updated as a moving average of the student’s weights, but makes additional modifications for sequence tasks like speech:  * An in-utterance contrastive loss is used as the pre-training objective. * Position randomization of the teacher’s input is used to avoid representation collapse. * Ablation experiments are done to show that the predictor (which was essential in earlier works like BYOL and SimSiam to avoid collapse) can be replaced with a convolutional projection head without performance degradation.  Empirically, the main contributions of the paper are: 1. Achieving similar/better WER on LibriSpeech compared to wav2vec 2.0 with 35% of the training cost; and 2. Incorporating multi-condition training, which has been used in supervised training in the past, for noise-robust ASR.