SPIRAL: Self-supervised Perturbation-Invariant Representation Learning for Speech Pre-Training
Authors: Wenyong Huang
Zhenhe Zhang
Yu Ting Yeung
Xin Jiang
Qun Liu
Venue: ICLR 2022 Poster (2022)
Keywords: Speech Representation Learning
Speech Pre-training
Speech Recognition
Self-supervised Representation Learning

Abstract: We introduce a new approach for speech pre-training named SPIRAL which works by learning denoising representation of perturbed data in a teacher-student framework. 
Specifically, given a speech utterance, we first feed the utterance to a teacher network to obtain corresponding representation. Then the same utterance is perturbed and fed to a student network. The student network is trained to output representation resembling that of the teacher. At the same time, the teacher network is updated as moving average of student's weights over training steps. In order to prevent representation collapse, we apply an in-utterance contrastive loss as pre-training objective and impose position randomization on the input to the teacher. SPIRAL achieves competitive or better results compared to state-of-the-art speech pre-training method wav2vec 2.0, with significant reduction of training cost (80% for BASE model, 65% for LARGE model). 
Furthermore, we address the problem of noise-robustness that is critical to real-world speech applications. We propose multi-condition pre-training by perturbing the student's input with various types of additive noise. We demonstrate that multi-condition pre-trained SPIRAL models are more robust to noisy speech (9.0% - 13.3% relative word error rate reduction on real noisy test data), compared to applying multi-condition training solely in the fine-tuning stage. Source code is available at https://github.com/huawei-noah/Speech-Backbones/tree/main/SPIRAL.

Published as a conference paper at ICLR 2022
SPIRAL: SELF-SUPERVISED PERTURBATION- INVARIANT REPRESENTATION LEARNING FOR SPEECH PRE- TRAINING
Wenyong Huang, Zhenhe Zhang, Ting Yeung, Xin Jiang, Qun Liu Huawei Noah's Ark Lab [wenyong, huang, 2 zhangzhenhel, yeung. yu, t: whwignghhuwe...commmmmmm com (jiang. xin, qun. 1 1i @huawei..cmom
ABSTRACT
We introduce new approach for speech pre-training named SPIRAL which works by learning denoising representation of perturbed data teacher-student frame- work. Specifically, given speech utterance, we first feed the utterance to a teacher network to obtain corresponding representation. Then the same utter- ance is perturbed and fed to studen! network. The student network is trained to output representation resembling that of the teacher. At the same time, the teacher network is updated moving average of student's 's weights over training steps. In order to prevent representation collapse, we apply an in-utterance contrastive loss pre-training objective and impose position randomization teacher. SPIRAL achieves competitive or better results compared to state-of-the, speech pre-training method wav2vec 2.0, with significant reduction of training cost (80% for BASE model, 65% for LARGE model). Furthermore, we address the problem of robusterbss that is critical real-world speech applications. We propose multi multi-condition pre-training by perturbing the student's 's input with various types of additive noise. We demonstrate that multi-condition pre-trained SPIRAL models are more robust to noisy speech (9.0% 13.3% relative word er- ror rate reduction on real noisy test data), compared to applying multi-condition training solely in the fine-tuning stage. Source code is available
the input the
INTRODUCTION
Industrial-scet automatic speech recognition (ASR) systems are usually trained with ten- thousands of hours of hand-transcribed speech data (Galvez al., 202T). However, labeling speech data ex- pensive and time-consuming, especially for languages with small speaker populations, or for specific domains (e.g., legal, financial, scientific).
Recently, methods of utilizing unlabeled speech data to improve speech recognition system have achieved remarkable progress. Amongst them, self-training (Manohar al. 2015 Kahn et al,, Park 2020a Synnaeve et al. [2020a] Chen et al. 2020b; Xu et al.) 2020; Park et al. 2020b, Xiao et al.
2021), also known as pseudo-labeling, starts by training an ASR model with labeled speech data, which is referred to teacher model. Then the teacher model, usually combined with a language model (LM), is used to produce pseudo-labels for unlabeled speech data. Finally, the labeled data and the pseudo-labeled data combined to train new model, which is referred to student model. The process is repeated by taking the student model as the teacher in next iteration. Another line of work is speech pre-training (van den Oord al. 2019 Chung Glass 2020 Wang al.) 2020 Baevski et al.) 20205 Liu et al.) 2020). Pre-training learns speech representation from unla- beled data in a self self-supervised way. The pre-trained model is then fine-tuned on the labeled data. Self-training and pre-training are complementary as shown in recent work (Xu et al. 2021 Zhang
al. 2020).
In this paper, we introduce new speech pre-training method which works by learning denois- ing representation of perturbed data with the teacher-student framework, named as Self-supervised

---

Published as a conference paper at ICLR 2022
Perturbation- Invariant Representation Learning (SPIRAL). Compared to ofite-o--hh-arr speech pre- training methods such as wav2vec 2.0 (Baevski et al. 2020b) and HuBERT (Hsu et al. (021), our method allows end-to-end training with single contrastive loss and without withoutreyying
discrete unit discovery techniques such as vector quantization (Tégou et al. 2011 Baevski et al. 2020a b) or iter- ative clustering process (Hsu al., 202T). We apply multi-condition training with SPIRAL (Seltzer it al. 2013
Ko et al., 2015) to improve noise-robustness for the downstream speech tasks.
SPIRAL is motivated by the observation that human tolerates speech perturbations or distortions fairly well. For example, people can communicate effectively in a noisy environment, or over a distorted telephone channel. Therefore, we hypothesize that by learning representation invariant to perturbation, the model will learn high-level representation which can enhance speech applications.
To learn perturbation-invariie representation in self-supervised way, we employ teacher-student framework similar to Tarvainen & Valpola (2017). During pre-training, given speech ulterance, we guide the student network which consumes the perturbed utterance learn from the teacher network which consumes the clean utterance. The student is trained to produce denoised representation of the perturbed utterance similar to teacher's representation the clean utterance. Meanwhile, the teacher, which shares the same model architecture with student, is updated as moving average of the student's weights over past training steps.
We apply the in-utterance contrastive loss avoid model collapse to trivial constant representa- tion (Chopra et al. 2005). As speech utterance are sequential data, there is another possible trivial solution which we call positional collapse. Positional collapse occurs when the student "cheats" by exploiting position correlation in teacher's representation to minimize the loss, while ignoring the content of the input utterance. To prevent positional collapse, we propose position randomization by adding random number of paddings on both sides of input utterance to the teacher.
Large-scale speech pre-training is computationally demanding. To reduce computation cost, we adopt gradual down-sampling strategy in SPIRAL model, which has been verified effective in speech recognition literatures with negligible performance degradation (Peddinti al.) 2018 Han et al. 2020b Huang et al. 2020). We also speculate that aggressive down-sampling helps to remove redundancy speech.
To evaluate the effectiveness of SPIRAL, we conduct experiments on LibriSpeech and Libri-Light Light datasets. By training small convolutional classifier on the representation of a frozen SPIRAL model, we can achieve WER of 3.5% and 6.4% on Librispeech test-clean and test-other respectively. SPIRAL achieves competitive or better results compared to state-of-the-art speech pre-training methods, while being much more training-efficient. We also demonstrate that multi-condition pre- trained SPIRAL are more robust to noisy speech with 9.0% 13.3% relative word error rate (WER) reduction on real noisy test data from ChiME-3 (Barker et al., 2015), compared the model apply- ing multi-conditionn training solely in fine-tuning stage.
2 RELATED WORK
Mean Teacher (MT) Tarvainen & Valpola, 2017) proposes using a student network to learn from teacher network which the moving average version of the student in the semi-supervised learning setting. The authors apply supervised loss for labeled data and consistency loss between teacher and student predictions for unlabeled data. However, direct application of MT to self-supervised learning leads to representation collapse (Grill et al., 2020).
Noisy student training (NST) (Xie al. 2020 Park et al.. 2020b) is a self self-training method. NST demonstrates the importance of the aggressive injection of noise into the student. Although not emphasized, no noise is injected into pseudo-labeling process of the teacher. We consider our work as extension of self-training approach to the self-supervised learning regime. Instead of using the teacher to provide pseudo-labels, we utilize the teacher for pseudo-reference representation.
Denoising autoencoders (Vincent et al. 2008 learn to recover clean input from corrupted ver- sion. However, speech data contain redundancy which is irrelevant to some speech applications such speech recognition. Previous work (Baevski et al. 2019) shows that speech pre-training by recovering masked input speech features is not effective. In SPIRAL, we instead enforce latent representation of corrupted input to resemble that of the corresponding clean input.
2

---

Published as a conference paper at ICLR 2022
Bootstrap Your Own Latent (BYOL) Grril et al. 2020 is a self-supervised image representation learning method. The method is based on a teacher-student framework similar to MT. The authors refer to student network online network and teacher network as target network. They observe that naive application of MT to self-supervised learning leads to trivial constant representation. They prevent the representation collapse by appending predictor
the student network. The theory behind is under investigation (Chen & Hc] 2021] TTan et al., (02I). Our method draws inspirations from BYOL and shares the similar architecture, but there are crucial differences. Instead of learning single global representation for an image as in BYOL, SPIRAL learns sequence of representation for an utterance. We aim for sequence applications such as speech recognition. In our preliminary experiments, we observe that appending predictor to student network is not sufficient to prevent trivial constant representation for sequential representation learning. We use in-utterance contrastive loss (Baevski et al.] 2020b) combined with input position randomization to successfully avoid representation collapse. We still keep the predictor in SPIRAL, but only for the sake of performance improvement from our observation. Another difference is that BYOL does not perform representation denoising. BOL applies perturbation, which they call augmentation, to both the inputs of the teacher and the student. We demonstrate that representation denoising is crucial for speech pre-training. When perturbation is applied to the teacher's input, the effectiveness of speech pre-training degrades drastically.
Wav2vec 2.0 (Baevski et al. 2020b) is a self-supervised speech representation learning method which belongs the masked prediction family. Masked prediction methods are effective for text pre-training (Devlin et al.] 2019), but not for speech pre-training when naively applied (Baevski al. 2019). The reason is that speech data contains redundancy such as speaker information, pro- nunciation variations, which are irrelevant to the semantic meaning of the utterance. To overcome this problem, wav2vec 2.0 perform masking in intermediate latent space and performs target dis- cretization with differentiable quantization scheme. However, quantization leads to more com- plex model by introducing additional hyper-parameters and an additional diversity loss. SPIRAL does not utilize quantization, and still achieves competitive performance compared to wav2vee 2.0. We hypothesize that aggressive down-sampling and learning by matching output representation may help to remove redundancy from the learned representation. We leave the investigation of whether target discretization could further improve SPIRAL for future work.
Liang et al. 2018 demonstrates that under the supervised learning setting, enforcing noise-invarian representation by penalizing difference between clean and noisy data improves ASR model accuracy.
METHOD
3.1 SELF-SUPERVISED PERTURBATION-IINARIA REPRESENTATION LEARNING
(SPIRAL)
Figure[[ shows the diagram of SPIRAL in the pre-training stage, where we use two neural networks, student Fg and teacher Fe.. The weights of the teacher 0 is the moving average of the weights of the student f. At step f, the weights of the teacher 6% are updated as
0g - 0400 (1-0,)0
where an determines the rate of weight updates. Given a speech utterance X = (2). ØT) of length T. the student takes a perturbed version X = s X = (ă1..... .T) as input where s(.) is perturbation function. The output of the student is representation sequence Z F(X; 0) (21...., 27). The teacher takes the same utterance without perturbation as input and output an- other representation sequence Z' F(X;0") = (zi...., x) For each representation 2; Z, the student is trained to match the teacher's representation 2 at the same position amongst & dis- tracting samples. The distracting samples are randomly drawn from other positions of the same utterance in Z'. which is found to be more effective than samples drawn from an entire batch of ut- terances (Baevski et al., 20206). The in-utterance contrastive loss is defined following SShn (016)); Wu et al. 2018) as,
T
exp(o(z,, 2))/k) log Σ
C
(2)
DjED, exp(o(z,, /k))
3

---

Published as a conference paper at ICLR 2022
In-utterance contrastive loss
Zi m.,,
Z7
Gradient
Moving average
Student 0
Teacher
0'
81 m.. Z
Perturbation
Random positional
padding
X1, XT
Figure Illustration of SPIRAL architecture for speech pre-training.
where caa, b) = a babblll position, and λ the temperature parameter.
//||||||lll]] is cosine similarity, D; is the set of indices of distractors for the i-th
However, applying in-utterance contrastive loss could cause a kind of representation collapse which we refer as positional collapse. Contrastive candidates are sampled based on their positions in utterances. When a teacher's representation z/ is correlated with its position i (e.g., correlation introduced by positional encoding in Transformer), the student could exploit this correlation to gen- erate its representation z; solely based on the position index i, while ignoring content of the input. In this case, the model does not learn meaningful representation of the input content. Therefore, prevent positional collapse by randomizing positions of teacher's representation. In particular, we add random numher of padding data at both ends of the input to the teacher to randomly shift the position information for each output representation z!. The student thereby is unable exploit the spurious position information to minimize the contrastive loss. Note that when calculating the contrastive loss, we exclude the corresponding representation of the padded data.
3.2
MODEL ARCHITECTURE
Encoder f(-)
g(-)
(())
10ms rate
40ms rate
80ms rase
Subsampling Convolution
Transformer Layers
Subsampling Convolution
Transformer Layers
Projection Head
Predictor
Figure 2: The architecture of the student model in SPIRAL. The frame rate of input denoted as '10/40/80 ms'. The dashed line indicates the optional predictor which be removed with small performance degradation. The structure of the teacher model is the same but without the predictor.
As illustrated in Figure 2 student Fg is composed of an encoder f(-). projection head g(-) (Chen et al. (2200a) and an optional predictor q(-) (Grill et al. 2020) i.e., Fe = (:: 8). The teacher Fer has the same structure expect that has predictor, Fgr (f (oog) 6"). The encoder consists of two blocks. In each block, we first apply temporal convolutions to perform wwwpling followed
by Transformer (Vaswani et al. 2017) with convolutional relative position encoding (Baevski al.
4

---

Published as a conference paper at ICLR 2022
2020b). Each convolution is followed by layer normalization (LN) (Ba et al. 2016) and ReLU. For the projection head, we apply a simple linear layer. The predictor consists of two layers of temporal convolution and linear layer. The convolutions are followed by batch normalization (BN) (loffe & Szegedy 2015) and ReI.U.
During pre-training, add computation noise to both the student and the teacher by applying dropout (Srivastava et al.] 2014) and LayerDrop (Fan et al. (2200) in Transformer. We use the same dropout and LayerDrop rates for the student and the teacher.
3.3 ADAPTIVE SPECAUGMENT
We apply adaptive SpecAugment similar to Park et al. (2020a) the primary perturbation method. Along either time or frequency dimension, we sample uniformly a certain proportion p of all time- steps to be start indices and mask the subsequent consecutive L time-steps. The masked time-steps is filled with zeros along frequency dimension. Along time dimension, we use Gaussian noise as masking values avoid numerical problems for LN (Park al., 2020a).
3.4 MULTI-CONDITION PRE-TRAINING
For noise-robust pre-training with SPIRAL, we perturb input of the student with various types of additive noise. We consider this technique as an implementation (MCT) (Seltzer al. 2013) in supf-supe setting. Specifically, for each input utterance to the student, we sample noise clips from noise dataset, and mix the noise clips with the whole utterance by addition in time-domain. We first uniformly sample signal-to-noise ratio (SNR) from pre- defined range for each utterance. Then we scale the noise volume according to the required SNR. In our preliminary experiments, we found that applying additive noise alone as perturbation degrades performance, Therefore, we apply additive noise perturbation together with adaptive SpecAugment.
multi-condition training
3.5
MODEL FINE-TUNING
After pre-training, we take the encoder from the student model in SPIRAL and add a randomly initialized convolutional classifier top of it. The convolutional classifier is composed of two layers of convolution, followed by LN and ReLU, and a linear output projection. The convolution filters consist of 512 channels with kernel width 5.
We fine-tune the model with connectionist temporal classification (CTC) (Graves et al. 2006) ob- jective for speech recognition. We use 1024 subwords as output units. The sub-words are generated from training transcripts of LibriSpeech with SentencePiece (Kudo & Richardson 2018).
We further investigate SPIRAL's ability to learn high-level representation of speech during pre- training. In addition to whole-model fine-tuning, we apply frozen fine-tuning, We freeze the pre- trained parameters and only fine-tune the convolutional classifier which can only perform local clas- sification due to limited receptive field.
4 EXPERIMENTAL SETUP
4.1 DATA
For pre-training, we use the 960-hour training data (ignoring the labels) from LibriSpeech (Panay- otov et al.] (LS--66)))
60k-hour unlabeled audio data from Libri-Light (Kahn al., 2020b)
(LL-60K). ). For Libri-Light, we segment the data using official tools with a threshold of 16s, resulting in 46.9k hours of data. The datasets are both derived from English audiobooks from LibriVox project? 3 For ASR fine-tuning, we apply 100-hour subset (train-clean- 100) as low-resoures labeled data and entire LS-960 with labels as high-resource labeled data, both from LibriSpeech.
For multi-condition training, we use the noise dataset from Reddy et al. 2021). The dataset consists of 181 hours of noise data with about 150 noise types and 70,000 clips. We shuffle and split the noise data with ratio of 8:1:1, which are used for training, synthesizing noisy dev-sets and synthetic noisy
https: ibri wwwiiix.iixorr..
5

---

Published as a conference paper at ICLR 2022
Table 1: Detailed configurations of the SPIRAL BASE and LARGE models.
Modules
Conv.1
Transf.1
Conv.2
Transf.2
Proj. H.
Predictor
#Params
layer emb. dim. fin dim. layerdrop attn. heads
layer
kernel size channel stride
kernel size channel stride
emb. dim.
kernel size
Hyper -params
fin dim.
dim.
channel
layerdrop attn. heads
2 512
10
5,5,1
5,1
768
5,3,1 256,256,256
BASE model
384,512,512
2048
1536,768
3072
256
91.5M
2,2,1
0
2,1
0.05
12
4
20
5,5,1
512
5,1
1024
5,5,1 512.512.512
LARGE model
384,512,512
2048
2048,1022 1024 2,1
4096
512
287M
2,2,1
0.05 8
0.05
16
Table 2: Comparison of pre-training cost between wav2vec 2.0 and SPIRAL.
Model
Unlabeled data
Training steps
GPU days
Mixed precision
Wav2vec 2.0 BASE
Baevki al. 20200
I.S-960
500k
102.4
SPIRAL BASE
LS-960
200k
20,8
Wav2vec 2.0 LARGE SPIRAL LARGE
Bacvsi
2020b
L.L-60k
1000k
665.6
LL-60k
500k
232.0
test-sets (results in Appendix A.2 respectively. SNRs of speech mixtures are set from to 30 dB. We evaluate on real noisy data test set from CHiME-3 (Barker et al.] 2013). which is comprised of speech data recorded in real noisy environments (bus, cafe, pedestrian area, and street junction). The data are recorded with a microphone array composed of multiple microphone channels located different positions of a tablet, and close-talking mierophone.
4.2 TRAINING SETUPS
We apply 128-dimensional log-mel filterbank extracted with 20 ms window and 10 ms stride as the input acoustic feature. We experiment with BASE model and LARGE model configurations as shown in Table [ The numbers of parameters are comparable to wav2vec 2.0 BASE and LARGE models correspondingly. For SpecAugment, set p 0.025 and L 20 for time-dimension mask, and p 0.02 and = 20 for frequency-dimension mask.
In pre-training, optimize with Adam (Kingma & Ba 2015) optimizer, warming up the learn- ing rate for the first 8% of updates to a peak of 3e-3. Then the learning rate decays to with cosine schedule. The moving average update rate ar of teacher's weight also follows cosine schedule (Grill et al. 2200)). We increase a from 0.995 to 1.0 and from 0.990 to 0.999 for BASE and LARGE models respectively. We train the BASE model with batch size of 24 per GPU for 200k steps on 16 V100 GPUs, which takes about 1.3 days. For the LARGE model, we train with batch size of 20 per GPU for 500k steps on 32 V 1100 GPUs, which takes about 7.25 days. As shown in Table there is significant reduction of training cost (GPU days) compared to wav2vec 2.0 (Baevski et al. 2020b). SPIRAL requires 80% and 65% less training cost for BASE and LARGE respectively. Note that mix-precision training is not applied for SPIRAL yet.
For fine-tuning, we optimize with Adam and a tri-state rate schedule where the learning rate is warmed up for the first 10% of updates to 3e-5, held constant for the next 40% and then linearly decayed to zero following Baevski et al. (022b)). We fine-tune BASE and LARGE with batch size of 14 and 18 per GPU respectively with batch size of 10 per GPÚ on GPUs for 320k steps on LS-960. We apply SpecAugment for whole-model fine-tuning but not for frozen fine-tuning. For multi-condition pre-training and
GPUs for 80k steps train-clean- 100. We fine-tume LARGE

---

Published as a conference paper at ICLR 2022
Table 3: ASR results fine-tuned from low-resource train-clean-100) Language models used in de- coding are listed in LM. We compare SPIRAL BASE pre-trained on LS-960 and SPIRAL LaRGe pre-trained on LL-60k with previous methods. We report WER (%) on Librispeech dev/test sets.
Model
Unlabeled
LM
dev
test
dala
clean
other
clean
other
Supervised/Semi- Supeeviiiee
Hybrid DNN/HMM Lüscher et al 2019
4-gram
5.0
19.5
5.8
18.6
Iter. pseudo-lahelie (Xu 2020
L1.-60k LS-860
ggraa++rrasst
3.19
6.14
3.72
7.11
Noisy student (Park al. 2020b
LSTM
3.9
8.8
4.2
8.6
Selr-supervised
wav2vec 2.0 BASE (Bavskki et al. 2020h
LS-960
13.5
13.3
SPIRAL BASE frozen (ours)
LS-960
7.9
12.7
7.6
13.0
SPIRAL BASE (ours)
LS-960
5.5
11.1
5.4
11.2
wav2vec 2.0 BASE Baccski ot al. 2020b
LS-960
4-gram
2.7
7.9
3.4
8.0
SPIRAL BASE (ours)
LS LS-960
4-gram
2.7
7.0
3.3
7.5
wav2vec 2.0 BASE (Bevski et al. 2020h
LS-960
Transf.
6.3
2.6
SPIRAL BASE (ourS)
LS-960
Transf.
2.3
5.8
2.7
6.1
wav2vec 2.0 LARGE (Bavvkki et al. 2020b
LL-60k
6.5
3.1
SPIRAL LARGE frozen (ours)
LL-660k
7.1
9.2
6.6
9.7
SPIRAL LARGE (ours)
LL-60k
3.3
5.9
3.3
6.3
wav2vec 2.0 LARGE [Baevski al. 20206
LL-60k
Transf. Transf.
4.0
2.0
SPIRAL LARGE (ours)
LL-60k
1.9
3.9
2.2
4.3
fine-tuning, randomly perturb each utterance with additive noise with 50% probability before applying SpecAugment. SNR is uniformly sampled from 0-30 dB.
4.3 LANGUAGE MODEL. AND DECODING
We use word-level Transformer LM (Baevski & Auli 2019) trained on Librispeech LM corpus which is identical to Synnaeve et al. (2020b). For low-resource ASR setting, we also evaluate SPIRAL BASE with the official LibriSpeech 4-gram LM. We observe that models fine-tuned with subword units performs worse than models fine-tuned with character units when decoding with word-level LM. Therefore, we apply character-based models for LM decoding, which is the same setting wav2vec 2.0. The results of LM decoding with subword-base models available in Appendix A.. A.I
As output frame rate of pre-trained SPIRAL encoder is low (80ms), the output sequence may be too short for character units. To reuse the pre-trained encoder, we devise an upsampling strategy for the SPIRAL encoder output in fine-tuning stage. We apply 1-D convolution layer project the original encoder output dimension into a vector of dimension 4d. At each time-step, we reshape the projected output vector from (1, 4d) (4,d). The frame rate becomes 20ms. Then feed the upsampled outputs to convolutional classifier.
We perform random search for decoding parameters and choose the best parameters according performance on dev-other with beam 50. The final test performance is measured with beam 500. We use the beam search decoder of PPrtaap al. (201))
RESULTS
5.1 EVALUATION UNDER SETTINGS
AND
LABELED DATA
We first evaluate our method under a low-resource ASR setting in which we tin--tune the models with 100-hour LibriSpeech data (train-clean- 100). The results shown in Table 3 We evaluate a BASE model pre-trained with 960-hour LibriSpeech (LS-960) and a LARGE model pre-trained with Libri-Light (LLL-6KK.. The frozen BASE model performs well, achieving WER of 13.0%

---

Published as a conference paper at ICLR 2022
Table 4: ASR results fine-tuned from high-resource I.S-960. Language models used in decoding are listed in LM. We compare SPIRAL LARGE pre-trained on Libri-Light (LL-60k) with previous methods. We report WER (%) on Librispeech dev/test sets.
Model
Unlabeled
LM
dev
test
data
clean
other
Clean
other
Supervised
ContextNet (Han et al. 2020a
LSTM
1.9
3.9
1.9
4.1
Conformer Gulati et
2020
LSTM
2.1
4.3
1.9
3.9
Semi-supervised
CTC Transf. + $2S Transf. PL. Synnaeve et 20200
Synnaeve et al. 2020a
LL-60k
CLM+Transf.
2.10
4.79
2.33
4.54
L.L-60k
CLM+Transf.
2.00
3.65
2.09
4.11
Iter. pseudo-laheling, Xu 2020
LL-60k
4-gram+ Transf.
1.85
3.26
2.10
4.01
Noisy student (Park al. 2020b
LL-60k
LSTM
1.6
3.4
1.7
3.4
Self-supervised
wav2vec 2.0 LARGE (Bavski et al. 2020
L.L-60k
2.1
4.5
2.2
4.5
SPIRAL LARGE frozen (ours)
LL-60k
4.0
6.2
3.5
6.4
SPIRAL LARGE (ours)
LL-60k
2.1
4.3
2.2
4.6
wav2vec 2.0 LARGE Bacsski et al. 2020b
LL-60k
Transf. Transf.
1.6
3.0
1.8
3.3
SPIRAL LARGE (ours)
L.L-60k
1.5
3.1
1.8
3.5
on test-other, which is par with wav2vee 2.0 BASE. This suggests that SPIRAL indeed learns meaningful high-level representations in a self-supervised way. When we fine-tune the whole BASE model, the model achieves WER of 5.4% and 11.2% test-clean and test-other respectively, outper- forming wav2ved 2.0 BASE with 11.5% and 15.8% relative WER reduction. When decoding with Transformer LM, the BASE model achieves WER of 2.7% and 6.1% test-clean and test-other respectively. The results are on par with wav2vec 2.0 BASE
The SPIRAL LARGE model consists of more parameters and pre-trained with more data. The model achieves WER of 2.2% and 4.3% on test-clean and test-other respectively. The significant improvement of LARGE over BASE demonstrates the scalability SPIRAL. The results of SPIRAL LARGE are competitive to wav2vee 2.0 LARGE. This is encouraging, as SPIRAL LARGE only takes 35% of training cost of wav2vec 2.0 LARGE.
We further evaluate SPIRAL LARGE pre-trained with Libri-Light (LL-60K) under a high- highures ASR setting with 960-hour I.S-660 as fine-tuning data. As shown Table 4 the LARGE model achieves WER of 1.8% and 3.5% on test-clean and test-other respectively, which are on par with the wav2vec 2.0 LARGE model. We note that the supervised models and the noisy student model (Park) et al. 2020b) in Table 4 are autoregressive models. Our models are fine-tuned with CTC objective which is aonr-utoreeressive and generally inferior to autoregressive models. We use CTC objective for its simplicity and comparability to previous speech pre-training methods.
We consider SPIRAL as preferred alternative to wav2vec 2.0 given that SPIRAL only requires 20%-35% computation cost of wav2vec 2.0. We expect further efficiency improvement when implement mix-precision training for SPIRAL.
5.2 NOISE-ROBUST PRE-TRAINING
To evaluate noise-robustness of the pre-trained models, we compare the effects of applying multi- condition training (MCT) in pre-training or fine-tuning stages of SPIRAL. The results are shown in Table 5 The vanilla SPIRAL BASE model and wav2vec 2.0 BASE model deteriorate with signifi- cantly higher WER noisy test data.
On real noisy test speech data in CHiME-3 for different microphone channels (ch), SPIRAL with multi-condition pre-training significantly improves speech recognition performance. Compared to the model applying MCT solely in fine-tuning, applying MCT both in pre- pre-training and fine-tuning achieves 12.4%. 13.3% and 9.0% relative WER reduction for ch 1, 5 and 2 respectively. There smaller performance improvement of 3,8% relative WER reduction for ch O. which is a close- talking microphone with the highest SNR. We note that ch 2 faces backwards the speaker. SNR
8

---

Published as a conference paper at ICLR 2022
Table 5: Evaluation noise-robustness of the models. We wav2vec 2.0 BASE released by the authors as the baseline. The SPIRAL BASE models are pre-trained with LS-960 and fine-tuned with train-clean-100. We report WER (%) on Librispeech and CHiME-- real data test sets.
Pre-train w/ MCT
Fine-tune
Librispeech
CHiME-3
BASE model
w/ MCT
clean
other
chO
chS
chl
ch2
wav2vec 2.0
6,1
13.3
23.2
56.1
68.3
98.1
SPIRAL
5,4
11.2
24.1
52.1
58.9
92.6
SPIRAL
5.7
11.4
20.8
35.5
41.1
76.4
SPIRAL SPIRAL
5.7
11.5
20.8
33.6
38.5
74.0
5.9
11.4
20.0
31.1
35.6
69.5
of the recordings from ch 2 is the lowest, leading to high WER. We note that other pre-training methods including wav2vec 2.0 may benefit from multi-condition training, which are worth for further investigation.
5.3 ABLATIONS
5.3. INPUT PERTURBATION AND COMPUTATION NOISE OF TEACHER
SPIRAL learns denoised representation of perturbed data. By default, we only apply perturbation to the input of the student. An alternative method is to perturb both inputs of the teacher and the student, and optimize consistency between their representations (Grill et al.] 2020; Chen & He] 202I). We conduct experiments evaluate the effects of perturbing the input and adding computation noise (dropout and LayerDrop) to the teacher. The results are shown in Table 8 in Appendix A.3) The
results suggest that applying SpecAugment to teacher's input degrades performance significantly. Performance degradation decreases but is still significant with lower ratio and width of the masks. This supports the necessity of representation denoising, and our view of SPIRAL as an extension of self-training in which teacher network fed with clean input. The results also support applying computation noise to teacher during pre-training. There is 15.9% relative WER reduction with computation noise. This may be linked to Gal Ghahramani (016)).
5.3.2 EFFECTS OF PREDICTOR AND PROJECTION HEAD
We do ablation studies to understand the role of predictor and projection head in SPIRAL. The re- sults shown in Table Appendix A.3 When removing the predictor from the student, we observe performance degradation, but representation collapse does not happen. In the architectures relying on predictor prevent collapse (Grill et al. 2020 Chen & He 2021), applying batch nor- malization (BN) in the predictor is essential. While in SPIRAL, we observe that BN in the predictor can be replaced by layer normalization (LNN with a small performance degradation. When the pre- dictor is removed, we observe performance improvement by applying a convolutional projection head. The convolutional projection head is composed of a temporal convolution layer with LN and ReLU, and a linear layer. But applying convolutional projection head to the model with predictor, there is no further performance improvement. This suggests that convolutional projection head and predictor play similar role in SPIRAL, and they not complementary.
CONCLUSION
We presented SPIRAL, a new approach to speech pre-training by learning denoising representation of perturbed data with a teacher-student framework. SPIRAL can learn high-level speech repre- sentation in self-supervised way. Training small convolutional classifier frozen representation of SPIRAL achieves WER of 3.5% and 6.4% on Librispeech test-clean and test-other respectively. We show that SPIRAL achieves competitive better results compared to state-of-the-art speech pre- pre-training methods, with significant reduction of training cost. We investigate multi-condition pre- pre-training and demonstrates that multi-condition pre-training is more effective than solely apply- ing multi-condition training in the fine-tuning stage. We presume SPIRAL as a general pre-training method, which apply to other modalities such images and text. We leave it for future work.

---

Published as a conference paper at ICLR 2022
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. arXiv:1607.0645:
Layer normalization, 2016.
Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling. In 7th International Conference Learning Representations, ICLR 2019, 2019.
Alexei Baevski, Michael Auli, and Abdelrahman Mohamed. Effectiveness of self-supervised pre- training for speech recognition, 2019. #rXiv:1911.03912
Alexei Baevski, Steffen Schneider, and Michael Auli. vq-wav2vee: Self-supervised learning of discrete speech representations. In 8th International Conference on Learning Representations ICLR 2020, 2020a.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli, wav2vec 2.0: A frame- work for self-supervised learning of speech representations. In Advances in Neural Information Processing Systems, volume 33, 2020b.
Jon Barker, Ricard Marxer, Emmanuel Vincent, and Shinji Watanabe. The third 'CHiME' speech separation and recogmition challenge: Dataset, task and baselines. 2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), pp. 504-511, 2015.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. Proces Proceedings the 37th International Conference on Machine Learning (ICMI.), pp. 1597-1607, 2020a.
Xinlei Chen and Kaiming He. Exploring simple Siamese representation learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Paltern Recognition, pp. 15750-15758, 2021.
Yang Chen, Weiran Wang, and Chao Wang. Semi-supervised ASR by end-to-end self-training. In Proc. Interspeech 2020, pp. 2787-27911 2020b.
Chopra, R. Hadsell, and LeCun. Learning a similarity metric discriminatively, with application to face verification. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 05), pp. 339-546. 2005.
Yu-An Chung and James Glass. Generative pre-training for speech with autoregressive predictive coding. In 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3497-3501, 2020.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, pp. 4171-4186, 2019.
Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth demand with structured dropout. In 8th International Conference on Learning Representations ICLR 2020, 2020.
Yarin Gal and Zoubin Ghahramani. Dropout a Bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of The 33rd International Conference on Machine Learning (ICML), pp. 1050-1059. 2016.
Daniel Galvez, Greg Diamos, Juan Manuel Ciro Torres, Keith Achorn, Anjali Gopi, David Kanter, Max Lam, Mark Mazumder, and Vijay Janapa Reddi. The People's s Speech: A large-scale diverse english speech recognition dataset for commercial usage. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist tem- poral classification: Labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd International Conference on Machine Learning (ICML), pp. 369-376, 2006.
10

---

Published as a conference paper at ICLR 2022
Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Beraaroo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap your own latent new approach to self-supervised learning. In Advances in Neural Information Processing Systems, volume 33, pp. 21271-21284 2020.
Anmol Gulati, James Qin, Shung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang. Conformer: Convolution Convolution-augmentee augmeented transformer for speech recognition. Proc. Interspeeck 2020, pp. 5036-5040, 2020.
Wei Han, Zhengdong Zhang, Yu Zhang, Jiahui Yu, Chung-Chenn Chiu, James Qin, Anmol Gulati, Ruoming Pang, and Yonghui Wu. ContextNet: Improving convolutional neural networks for automatic speech recognition with global context. In Proc. Interspeech 2020, pp. 3610-3614, 2020a.
Wei Han, Zhengdong Zhang, Yu Zhang, Jiahui Yu, Ruoming Pang, and Yonghui Wu. ContextNet: Improving convolutional neural networks for automatic speech recognition with global context. In Proc. Interspeech 2020, pp. 3610-3614, 2020b.
Chiu, James Qin, Anmol Gulati,
Wei-Ning Hsu, Yao-Hung Hubert Tsai, Benjamin Bolte, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: How much can a bad teacher benefit ASR pre-training? In 2021 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 6533-6537, 2021.
Wenyong Huang, Wenchao Hu, Yu Ting Yeung, and Xiao Chen. Ton=-Transfomer Transducer: Low Latency, Low Frame Rate, Streamable End-to- no-End Speech Recognition. In Proc. Interspeech 2020, pp. 5001-5005, 2020.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by the reducing internal covariate shift. In Proceedings the 32nd International Conference on Machine Learning (ICML), pp. 448-456, 2015.
Herve Jégou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(1):117-128, 2011.
Jacob Kahn, Ann Lee, and Awni Hannun. Self-training for end-to-end speech recognition. In 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7084- 7088, 2020a.
Jacob Kahn, Morgane Rivière, Weiyi Zheng, Evgeny Kharitonov, Qiantong Xu, Pierre-Emmnn Emmanuel Mazaré, Julien Karadayi, Vitaliy Liptchinsky. Ronan Collobert, Christian Fuegen, et al. Libri- light: A benchmark for ASR with limited or supervision. In 2020 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP), pp. 7669-7673, 2020b.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd Interna- tional Conference on Learning Representations, ICLR 2015, 2015.
Tom Ko, Vijayaditya Peddinti, Daniel Povey, and Sanjeev Khudanpur. Audio augmentation for speech recognition. In Proc. Interspeech 2015, pp. 3586-3589, 2015.
Taku Kudo and John Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing: System Demonstrations, 2018
Davis Liang, Zhiheng Huang, and Zachary Lipton. Learning noise-invariant representations for robust speech recognition. In 2018 IEEE Spoken Language Technology Workshop (SLT), pp- 56-63, 2018.
Andy T. Liu, Shu-wen Yang, Po-Han Chi, Po-chun Hsu, and Hung-yi Lee. Mockingjay: Unsuper- vised speech representation learning with deep bidirectional transformer encoders. In 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6419-6423, 2020.
11

---

Published as a conference paper at ICLR 2022
Christoph Lüscher, Eugen Beck, Kazuki Irie, Markus Kitza, Wilfried Michel, Albert Zeyer, Ralf Schlüter, and Hermann Ney. RWTH ASR Systems for LibriSpeech: Hybrid vs Attention. In Proc. Interspeech 2019, pp. 231-235, 2019.
Vimal Manohar, Daniel Povey, and Sanjeev Khudanpur. Semi-supervised maximum mutual infor- mation training of deep neural network acoustic models. In Proc. Interspeech 2015, pp. 2630- 2634, 2015.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An ASR corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 5206-5210. IEEE, 2015.
Daniel S. Park, Yu Zhang. Chung-Chent Chiu, Youzheng Chen, Bo Li, William Chan, Quoe V. Le, and Yonghui Wu. Specaugment on large scale datasets. In 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6879-6883, 2020a.
Daniel S. Park, Yu Zhang, Ye Jia, Wei Han, Chung-Cheng Chiu, Bo Li, Yonghui Wu, and Quoc V. Le. Improved noisy student training for automatic speech recognition. In Proc. Interspeech 2020, pp. 2817-2821, 2020b.
Vijayaditya Peddinti, Yiming Wang, Daniel Povey, and Sanjeev Khudanpur. Low latency acoustic modeling using temporal convolution and LSTMs. IEEE Signal Processing Letters, 25(3):373- 377, 2018.
Vineel Pratap, Awni Hannun, Qiantong Xu, Jeff Cai, Jacob Kahn, Gabriel Synnaeve, Vitaliy Liptchinsky, and Ronan Collobert. Wav2Letter++: A fast open-source speech recognition system. In 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6460 6464, 2019.
Chandan KA Reddy, Harishchandra Dubey, Vishak Gopal, Ross Cutler, Sebastian Braun, Hannes Gamper, Robert Aichner, and Sriram Srinivasan. ICASSP 2021 deep noise suppression challenge. In 202/ IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6623-6621 IEEE, 2021.
Michael L.. Seltzer, Dong Yu, and Yongqiang Wang. An investigation of deep neural networks for noise robust speech recognition. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7398-7402, 2013.
Kihyuk Sohn. Improved deep metric learning with multi-class N-pair loss objective. In Advances in Neural Information Processing Systems, volume 29, 2016.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15():1192--1988 2014.
Gabriel Synnaeve, Qiantong Xu, Jacob Kahn, Tatiana Likhomanenko, Edouard Grave, Vineel Pratap, Anuroop Sriram, Vitaliy Liptchinsky, and Ronan Collobert. End-to-end ASR: from su- pervised semi-supervised learning with modern architectures. In workshop Self-supervision in Audio and Speech (SAS), 37th International Conference on Machine Learning (ICML 2020), 2020a.
Gabriel Synnaeve, Qiantong Xu, Jacob Kahn, Tatiana Likhomanenko, Edouard Grave, Vineel Pratap, Anuroop Sriram, Vitaliy Liptchinsky, and Ronan Collobert. End-to-end ASR: from su- pervised semi-supervised learning with modern architectures. In workshop Self-supervision in Audio and Speech (SAS), 37th International Conference on Machine Learning (ICML 2020), 2020b.
Antti Tarvainen and Harri Valpola. Mean teachers better role models: Weight-averaged consis- tency targets improve semi-supervised deep learning results. In Advances in Neural Information Processing Systems, volume 30, 2017.
Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self self-supervised learning dynam- ics without contrastive pairs. In Proceedings of the 38th International Conference on Machine Learning (ICMI). pp. 10268-10278. 2021.
12

---

Published as a conference paper at ICLR 2022
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding, 2019. MEXiv:880.0034480
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems, volume 30, 2017.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th International Conference on Machine Learning (ICML), pp. 1096-1103, 2008.
Weiran Wang. Qingming Tang, and Karen Livescu. Unsupervised pre-training of bidirectional speech encoders via masked reconstruction. In 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 6889 60999,883341 2020.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3733-3742, 2018.
Alex Xiao, Christian Fuegen, and Abdelrahman Mohamed. Contrastive semi-supervised learning for ast. In 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3870-3874, 2021.
Qizhe Xie, Minh-Thaan Thang Luong, Eduard Hovy, and Quoc V. Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
Qiantong Xu, Tatiana Likhomanenko, Jacob Kahn, Awni Hannun, Gabriel Synnaeve, and Ronan Collobert. Iterative pseudo-labeling for speech recognition. In Proc. Interspeech 2020, pp. 1006- 1010, 2020.
Qiantong Xu, Alexei Baevski, Tatiana Likhomanenko, Paden Tomasello, Alexis Conneau, Ronan Collobert, Gabriel Synnaeve, and Michael Auli. Self-training and pre- pre-training are complementary for speech recognition. In 2021 IEFE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 3030-3034, 2021.
Yu Zhang, James Qin, Daniel S. Park, Wei Han, Chung-Cheng Chiu, Ruoming Pang, Quoc V. Le, and Yonghui Wu. Pushing the limits of semi-supervised learning for automatic speech recogni- tion. In NewrlPS 2020 workshop: Self-Superviser Learning for Speech and Audio Processing, 2020.
13

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.1 OUTPUT UNITS COMPARISON FOR LANGUAGE MODEL DECODING
We evaluate decoding performance of different combinations of SPIRAL output units and Trans- former LM. The subword-level LM is trained on the same Librispeech LM corpus and shares the same 1024 subwords as the SPIRAL model fine-tuned with subword units.
Table ASR results fine-tuned from low-resource train-clean- 100 and high-resource train-960, The model units and language models for decoding listed in fine-tuning units and LM respectively. We compare SPIRAL BASE pre-trained on L.S-960 and SPIRAL LARGE pre-trained on LL.600k with previous methods. We report WER (%) Librispeech dev/test sets.
Unlabeled data
Fine-tuning units
dev
test
Model
LM
clean
other
clean
other
Low-resource
SPIRAL BASE
LS-960
subword
5.5
11.1
5.4
11.2
SPIRAL BASE
LS-960
char
5.3
11.0
5.4
11.1
SPIRAL BASE
LS-960
subword
word subword
2.9
6.8
3.2
7.2
SPIRAL BASE
LS-960
subword
2.7
6.3
2.9
6.7
SPIRAL BASE
LS-960
char
word
2.3
5.8
2.7
6.1
SPIRAL LARGE
LL-60k
subword
3.3
5.9
3.3
6.3
SPIRAL LARGE
LL-60k
char
3.5
6.3
3.5
6.7
SPIRAL LARGE
LL-60k
subword
word subword
2.4
4.5
2.5
4.8
SPIRAL LARGE
LL-60k
subword
2.3
4.5
2.4
4.8
SPIRAL LARGE
LL-60k
char
word
1.9
3.9
2.2
4.3
High-resource
SPIRAL LARGE
LL-60k
subword
2.1
4.3
2.2
4.6
SPIRAL LARGE
LL-60k
char
2.2
4.5
2.3
4.7
SPIRAL LARGE
LL-60k
subword
word subword
1.7
3.5
1.9
3.7
SPIRAL LARGE
LL-60k
subword
1.6
3.3
1.7
3.5
SPIRAL LARGE
LL-60k
char
word
1.5
3.1
1.8
3.5
A.2 PERFORMANCE ON SYNTHETIC NOISY DATASET
On the synthetic noisy dataset (NS-Librispeech) with matched SNR range (0-30 dB) of the training data, SPIRAL pre-trained and fine-tuned with MCT more effective than applying MCT solely in fine-tuning. We observe 5.3% and 9.1% relative WER reduction synthetic noisy test-clean and test-other sets respectively.
Table 7: Evaluation noise-robustness of the models. We use wav2vec 2.0 BASE released by the authors as the baseline. The SPIRAL BASE models are pre-trained with LS-960 and fine-tuned with 1000 We report WER (%) Librispeech test sets and synthetic noisy Librispeech test sets at 0 30 dB (NS-Librispeech)
Pre-train w/ MCT
Fine-tune w/ MCT
Librispeech clean
NS-Librispeech other
BASE model
other
clean
wav2vec 2.0
6.1
13.3
14.4
27. 4
SPIRAL
5.4
11.2
12.2
23.3
SPIRAL
/
5.7
11.4
7.6
16.5
SPIRAL
5.7
11.5
7.4
15.8
SPIRAL
/
5.9
11.4
7.2
15.0
14

---

Published as a conference paper at ICLR 2022
A.3 RESULTS OF ABLATION STUDIES
Here are the results of ablation studies discussed in Section 5.3
Table 8: Ablation studies of input perturbation with SpecAugment and computation noise teacher. We list the mask ratio and mask length as p. for time and frequency masks. The first row the default setting of SPIRAL. We apply SPIRAL BASE fine-tuned with train-clean-100 100, and report WER (%) on the Librispeech dev-other set.
Time mask Frequency mask
Computation noise
dev other
11.1
13.2
0.025, 20 0.0125,
0.02, 20
47.9
0.01, 20
42.8
0.025, 10
0.02, 10
39.4
Table 9: Ablation studies predictor and projection head in SPIRAL. We apply SPIRAL BASE fune-tuned with train-clean-100, and report WER (%) the Librispeech dev-other set.
Architecture
dev other
SPIRAL BASE
11.1
+ predictor use LN
11.6
+ conv proj. head
11.5
- predictor
13.7
conv proj. head
12.1
15


Reviews:
[{"review_id": "wtkHRFXppW", "reviewer": "Reviewer_J87m", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. While the paper adapts the teacher-student self-supervised pre-training framework that has been studied for image representation learning (Grill et al., Chen & He, Tian et al.), the modifications for sequential learning --- in-utterance contrastive loss, position randomization, and convolutional subsampling --- are essential for speech.\n2. The authors perform ablations to show the relevance of the projection head vis-a-vis the predictor, and demonstrate they are complementary. This result raises important questions about conclusions drawn about this framework in previous works (like Tian et al. 2021), and their applicability to sequence representation learning.\n3. Experimental setup (e.g. hyperparameters) is described in detail. SPIRAL obtains strong WER performance on LibriSpeech with a fraction of the training of wav2vec 2.0. Multi-condition training is shown to be effective at pre-training (versus only at fine-tuning)  to improve noise robustness (~10% relative WER reduction) while maintaining performance on clean speech.\n4. The authors have mentioned that they will release the code at the time of publication. This would be useful for the community to extend this direction of research.\n5. The paper is clear and easy-to-follow, with sufficient discussion of related work.\n\nWeaknesses:\n1. While the proposed model does well on clean LibriSpeech, performance under noisy conditions may be concerning (Section 5.2). In Table 5, when the synthetic noise at test time is matched with training noise (both in noise type and SNR), the WERs are good; but mismatched SNR significantly degrades WER (8.0% to 26.1% for “test-clean”). This suggests that the model may be overfitting to the range of SNRs during training. Second, there are no evaluations on mismatched noise types during train and test, so it is hard to predict the model’s generalizability to other types of noise. Furthermore, the absence of evaluations on real noisy data (such as CHiME-4) raises some questions about the benefits of multi-condition training. These questions are of particular interest since the model is named “perturbation invariant”.\n2. There have been several recent works analyzing representation collapse of non-contrastive SSL models (e.g. Tian et al., 2021) which suggest that a predictor on the student branch, and weight decay during training are essential to prevent collapse. The authors mention briefly in Section 2 that the predictor was not sufficient and they had to use the in-utterance contrastive loss with position randomization. Can they suggest reasons why this might be the case?\n3. The ablation in Section 5.3.2  indicates that performance degrades significantly when perturbations are applied to the teacher’s input. This again deviates from standard practice in image representation learning where augmentation is applied on the inputs for both the teacher and the student, and (as astutely observed by the authors) is closer in principle to standard self-training. Could the authors suggest why input noise is harmful but computation noise (dropout) works by, for instance, elucidating the link to Gal and Ghahremani (2016)?\n\nOther comments/questions:\n1. For multi-condition pre-training, is the noisy input used for both the student and the teacher? If yes, what would happen if the clean input is used for the teacher?\n2. For LibriSpeech evaluation under the “low resource” setting, perhaps it would be fairer (although not comparable to Baevski et al., 2020) to exclude the train-clean-100 subset during pre-training, as done in Park et al., 2020.\n3. In the first paragraph of Page 2, the authors comment that “SPIRAL also allows to combine with multi-condition training.” This is slightly misleading as there is no inherent limitation in the other models (wav2vec 2.0, HuBERT) mentioned in the previous line that would prevent multi-condition training with those models.\n4. Have the authors tried learning from raw waveforms instead of log-mel filterbanks?\n\nSome typos:\n* Section 1, para 3: “...learning representation denoising of perturbed data…” →  “...learning denoising representation of perturbed data…”\n* Section 3.2, para 1: “predicotr” → “predictor”\n* Section 3.5, para 3: “...due limited receptive field.” → “...due to limited receptive field.” \n\n", "summary_of_the_paper": "The paper introduces SPIRAL, a new method for self-supervised pre-training for speech. SPIRAL is based on the teacher-student framework similar to Mean Teacher (Tarvainen and Valpola, 2017) and BYOL (Grill et al., 2020) where the teacher’s weights are updated as a moving average of the student’s weights, but makes additional modifications for sequence tasks like speech:\n\n* An in-utterance contrastive loss is used as the pre-training objective.\n* Position randomization of the teacher’s input is used to avoid representation collapse.\n* Ablation experiments are done to show that the predictor (which was essential in earlier works like BYOL and SimSiam to avoid collapse) can be replaced with a convolutional projection head without performance degradation.\n\nEmpirically, the main contributions of the paper are:\n1. Achieving similar/better WER on LibriSpeech compared to wav2vec 2.0 with 35% of the training cost; and\n2. Incorporating multi-condition training, which has been used in supervised training in the past, for noise-robust ASR.\n", "summary_of_the_review": "While there are some clear limitations in the empirical sense, particularly in the claim that the model learns denoised high-level representations, the paper advances self-supervised learning for speech tasks by adapting the popular teacher-student framework popular in image representation learning. The model achieves WERs similar/better than the popular wav2vec 2.0 architecture while reducing training time and model size. The ablation experiments raise questions about whether the modeling/training strategies deemed essential in that modality are also needed for sequential tasks, although the authors have not addressed these questions directly in this work. Overall, this work (and the promised code release) is likely to lead to new explorations in this direction for speech self-supervised learning.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "TgcS6NUWAj", "reviewer": "Reviewer_H4es", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper describes teacher-student self-supervised training. The teacher generates a representation on clean data. The student matches  the teacher's representation with a perturbed version of the utterance. In-utterance contrastive loss is used to avoid learning a trivial representation. Positional collapse is avoided by random padding before/after the utterance fed to the teacher. The teacher is updated as the moving average of student checkpoints. Results are presented for low-resource and high-resource settings using librispeech and librilight.\n\nThe strengths are:\n- The paper has an interesting premise, combining ideas from noisy-student training and self-supervised training.\n- The results are somewhat comparable to wav2vec 2.0, but at lower training cost.\n- The method is robust to unseen noise situations.\n- As far as I understand, it does not require separately pre-training the teacher, unlike noisy student training. This could be clarified however.\n\nThe weaknesses include:\n- The results are better or on par with wav2vec 2.0 in some cases, but not on the whole. The authors mention that the settings are not fully tuned -- it would be interesting to know what the best results are after tuning.\n- While there are some ablation studies, some unanswered questions remain. In particular, I am curious how much is gained from additive noise perturbation and from positional randomization.\n- In looking at the noisy test results (table 5), there is no additive noise in either pre-training or fine-tuning for wav2vec 2.0. Hence this comparison seems incomplete.\n\nOther questions and notes:\n- Related work could include contrastive semi-supervised learning work from Facebook, as another way to combine these two aspects.\n- How is the teacher initialized?\n- In table 2, I assume training step refers to pre-training steps. How is the optimal number of pre-training steps determined for each method?\n- What perturbation settings are included in the wav2vec setup?\n- Table 3: noisy student unlabeled data = \"LS-860\" -- is this a typo?\n- Page 7: the-clean -> test-clean (typo)", "summary_of_the_paper": "The paper describes teacher-student self-supervised training, with a denoising advantage by perturbing the student. It elaborates on techniques to avoid collapse and compares results with wav2vec 2.0.", "summary_of_the_review": "The paper has an interesting premise and some promising experiments to justify it. On the whole, the method is somewhat comparable to wav2vec 2.0, although it falls short in several cases. This can be made clearer in the abstract and introduction, while highlighting the reduced training cost. I am not convinced that the comparison to wav2vec 2.0 is complete in terms of denoising uses, as there is no discussion of specaug and additive noise in wav2vec 2.0 during pre-training or fine-tuning. Also, some additional ablation studies to understand the benefit of additive noise and positional randomization would strengthen the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Q9081kyrY8B", "reviewer": "Reviewer_GA9R", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nThe paper is overall well written, and presents a good set of experimental results. The baselines are strong (for what I can tell), the approach is well motivated and explained well. The authors promise to release code upon acceptance of the paper, which should allow readers to verify results and build on top of it. Pre-trained self-supervised audio feature extractors have the potential to improve speech recognition in many domains (e.g. low resource multi-lingual speech recognition) and reducing compute is a critical step in that direction.\n\nWeaknesses:\n\nSome of the experimental results are a bit early? I would like to see results with LM rescoring after the hyper parameters have been optimized, so that results are more comparable (and hopefully consistent)? Similarly with mixed precision training. Would it be possible to get rid of the convolutional layers and build a model that is based entirely on self-attention? This should be even more efficient on GPU?\n\nUpdate: the additional tests on Chime data, and the updated decoding results address these concerns. I am updating my assessment.", "summary_of_the_paper": "The paper presents a novel approach to self-supervised speech representation learning, which promises to be simpler than existing methods such as wav2vec 2.0. The approach is inspired by the BYOL approach from CV, and is shown to be indeed largely as effective as wav2vec 2.0, while being significantly more efficient during training.", "summary_of_the_review": "Making self-supervised speech representations easier to train is a significant contribution, and this paper presents a viable approach to doing so. Some results feel a bit preliminary, but the paper is well written and the authors may have updated results but the time of the conference, and they will release code (plus presumably models?). My recommendation is thus for accept.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "EPaiaS64Di", "reviewer": "Reviewer_g6zH", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written with the following strengths.\n(1) The proposed method is self-supervised learning with the teacher-student framework that is an extension to Mean Teacher (MT) and Boostrap Your Own Latent (BYOL).\n(2) The proposed method is a novel one that aims to learn the representation denoising of perturbed data with the teacher-student framework.  In addition, the proposed method can also be combined with multi-condition training to improve the noise-robustness.\n(3) The motivation of the method sounds quite reasonable which aims to learn the representation that is invariant to perturbation so that the learnt representation is a high-level one (e.g. carrying content information) to enhance the downstream speech applications.\n(4) An in-utterance contrastive loss (proposed by Chopra et al. (2005) is adopted to avoid the model collapse problem. \n(5) Position randomization technique is further introduced to prevent the positional collapse problem.\n(6) A gradual down-sampling strategy has been adopted to train the SPIRAL model to reduce the computation cost.\n(7) Plenty of experiments have been conducted to evaluate the performance of the proposed method. \n(8) The proposed method can achieve competitive or better results than wav2vec 2.0 whereas with significantly less training cost.\n\n", "summary_of_the_paper": "The paper proposes a new speech pre-training approach named SPIRAL.  The proposed method is trained by learning representation denoising of perturbed data with the teacher-student framework in a self-supervised manner.  The motivation of the method is to learn the representation that is invariant to perturbation so that the learnt representation is a high-level one (e.g. carrying content information) that can enhance the downstream speech applications.  Compared to the state-of-the-art speech pre-training method wav2vec 2.0, the proposed method can achieve competitive or better results but with a significant reduction of the training cost.  The proposed method is also able to deal with the noise-robustness problem.", "summary_of_the_review": "Based on the above main review (especially the strengths of the paper), although some of the ideas have been borrowed from previous work (e.g. MT, BYOL, contrastive loss, etc), the paper has proposed extensions to these work by considering the sequential applications in speech processing.  And the relations with the previous work have been clearly discussed in section 2.  Hence, I think the paper could be accepted for publication on ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AFKa8ljRP-V", "reviewer": "Reviewer_fPpa", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The goal of perturbation or augmentation invariance is well motivated in speech recognition, and also ML/ representation learning more broadly.  The Librispeech 960 numbers are quite strong.  \n\n* The results using synthetic noisy data are less convincing than naturally occurring noise.  This is especially important since the synthetic noise is used during training as well.\n* In figure 1 it appears that the positional padding is applied equally to both the input and output of the teacher.  This is not as clear from the description in Section 3.1 If this is the case it would be helpful to expand on the description.\n* Section 3.3: SpecAugment typically masks both time and frequency bands.  It is not clear if this is done here, or if the masking is performed on either time or frequency.  Assuming both time and frequency are masked, is the area masked by both time and frequency replaced with zeros or gaussian noise?\n* Section 3.5: What is the rationale for using CTC ASR decoding rather than say an attention decoder, or RNN-T model?\n* Section 4.2 it would be useful to have more insight into why SPIRAL trains faster than wav2vec 2.0.  Is this due to implementation, or is there something fundamental to these algorithms that makes SPIRAL faster?\n* Section 4.2: It might be useful to compare these results to SpeechStew \n* Section 5.3.1: it is interesting that other approaches are more robust to perturbing both teacher and student inputs, while SPIRAL requires clean teacher inputs. More thorough discussion of this would be helpful, though space is tight.\n\nShorter notes: \n* Section 2: the description of Liang et al (2018) is notably brief compared to others.  The comparison of perturbation invariance in ASR training seems like a worthwhile comparison to invariance during pretraining.\n* Table 3: only row 2 includes 3 significant figures, the rest includes 2. it would be better to be consistent here.  (Same comment in tables 4. and 5)\n* Table 5: it would be better to organize the columns in by increasing or decreasing SNR.\n* Typo in Section 5.3.2 where \"When the predictor is removed, We observe\" the \"We\" shouldn't be capitalized.", "summary_of_the_paper": "This paper describes a self supervision training approach to pretrain speech encoders.  This is related to recent work on wav2vec 2.0, and SimCLR.  The contrastive loss is similar to wav2vec 2.0, while the invariance to perturbation (i.e. augmentation) is akin to SimCLR.\nPerformance is competitive with wav2vec 2 on Librispeech.", "summary_of_the_review": "This is a compelling paper. it's a well motivated and technically sound perspective on self-supervised training.  The performance on Librispeech 960 is quite strong.  The performance in noise conditions is strong, but would be more convincing if shown on more actual rather than synthetic noise.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed a self-supervised speech pre-training approach, by the name of SPIRAL, to learning perturbation-invariant representations in a teacher-student setting.  The authors introduced a variety of techniques to improve the performance and stabilize the training.  Compared to the popular unsupervised learning model wav2vec 2.0, better WERs were reported using SPIRAL with a reduced training cost.  All reviewers considered the work solid with sufficient novelty but also raised concerns regarding the generalization under unseen real-world noisy conditions and missing decoding details.  The authors responded with new Chime-3 results  and updated LM decoding results.  The new results show that, after a bug fix, SPIRAL can outperform wav2vec 2.0 when no external LM is used.  

Overall the proposed approach is technically novel.  The experiments are extensive and the results are compelling. In addition, the training time can be significantly reduced compared to wav2vec 2.0. All reviewers are supportive.  So I would recommend accept.
Decision: Accept (Poster)