SPIRAL: Self-supervised Perturbation-Invariant Representation Learning for Speech Pre-Training

1 INTRODUCTION . Industrial-scale automatic speech recognition ( ASR ) systems are usually trained with ten-thousands of hours of hand-transcribed speech data ( Galvez et al. , 2021 ) . However , labeling speech data is expensive and time-consuming , especially for languages with small speaker populations , or for specific domains ( e.g. , legal , financial , scientific ) . Recently , methods of utilizing unlabeled speech data to improve speech recognition system have achieved remarkable progress . Amongst them , self-training ( Manohar et al. , 2015 ; Kahn et al. , 2020a ; Synnaeve et al. , 2020a ; Chen et al. , 2020b ; Xu et al. , 2020 ; Park et al. , 2020b ; Xiao et al. , 2021 ) , also known as pseudo-labeling , starts by training an ASR model with labeled speech data , which is referred to as teacher model . Then the teacher model , usually combined with a language model ( LM ) , is used to produce pseudo-labels for unlabeled speech data . Finally , the labeled data and the pseudo-labeled data are combined to train a new model , which is referred to as student model . The process is repeated by taking the student model as the teacher in next iteration . Another line of work is speech pre-training ( van den Oord et al. , 2019 ; Chung & Glass , 2020 ; Wang et al. , 2020 ; Baevski et al. , 2020b ; Liu et al. , 2020 ) . Pre-training learns speech representation from unlabeled data in a self-supervised way . The pre-trained model is then fine-tuned on the labeled data . Self-training and pre-training are complementary as shown in recent work ( Xu et al. , 2021 ; Zhang et al. , 2020 ) . In this paper , we introduce a new speech pre-training method which works by learning denoising representation of perturbed data with the teacher-student framework , named as Self-supervised Perturbation-Invariant Representation Learning ( SPIRAL ) . Compared to state-of-the-art speech pretraining methods such as wav2vec 2.0 ( Baevski et al. , 2020b ) and HuBERT ( Hsu et al. , 2021 ) , our method allows end-to-end training with a single contrastive loss , and without relying on discrete unit discovery techniques such as vector quantization ( Jégou et al. , 2011 ; Baevski et al. , 2020a ; b ) or iterative clustering process ( Hsu et al. , 2021 ) . We apply multi-condition training with SPIRAL ( Seltzer et al. , 2013 ; Ko et al. , 2015 ) to improve noise-robustness for the downstream speech tasks . SPIRAL is motivated by the observation that human tolerates speech perturbations or distortions fairly well . For example , people can communicate effectively in a noisy environment , or over a distorted telephone channel . Therefore , we hypothesize that by learning representation invariant to perturbation , the model will learn high-level representation which can enhance speech applications . To learn perturbation-invariant representation in a self-supervised way , we employ a teacher-student framework similar to Tarvainen & Valpola ( 2017 ) . During pre-training , given a speech utterance , we guide the student network which consumes the perturbed utterance to learn from the teacher network which consumes the clean utterance . The student is trained to produce denoised representation of the perturbed utterance similar to teacher ’ s representation of the clean utterance . Meanwhile , the teacher , which shares the same model architecture with student , is updated as moving average of the student ’ s weights over past training steps . We apply the in-utterance contrastive loss to avoid model collapse to trivial constant representation ( Chopra et al. , 2005 ) . As speech utterance are sequential data , there is another possible trivial solution which we call positional collapse . Positional collapse occurs when the student “ cheats ” by exploiting position correlation in teacher ’ s representation to minimize the loss , while ignoring the content of the input utterance . To prevent positional collapse , we propose position randomization by adding random number of paddings on both sides of input utterance to the teacher . Large-scale speech pre-training is computationally demanding . To reduce computation cost , we adopt a gradual down-sampling strategy in SPIRAL model , which has been verified effective in speech recognition literatures with negligible performance degradation ( Peddinti et al. , 2018 ; Han et al. , 2020b ; Huang et al. , 2020 ) . We also speculate that aggressive down-sampling helps to remove redundancy in speech . To evaluate the effectiveness of SPIRAL , we conduct experiments on LibriSpeech and Libri-Light datasets . By training a small convolutional classifier on the representation of a frozen SPIRAL model , we can achieve WER of 3.5 % and 6.4 % on Librispeech test-clean and test-other respectively . SPIRAL achieves competitive or better results compared to state-of-the-art speech pre-training methods , while being much more training-efficient . We also demonstrate that multi-condition pretrained SPIRAL are more robust to noisy speech with 9.0 % - 13.3 % relative word error rate ( WER ) reduction on real noisy test data from ChiME-3 ( Barker et al. , 2015 ) , compared to the model applying multi-condition training solely in fine-tuning stage . 2 RELATED WORK . Mean Teacher ( MT ) ( Tarvainen & Valpola , 2017 ) proposes using a student network to learn from a teacher network which is the moving average version of the student in the semi-supervised learning setting . The authors apply a supervised loss for labeled data and a consistency loss between teacher and student predictions for unlabeled data . However , direct application of MT to self-supervised learning leads to representation collapse ( Grill et al. , 2020 ) . Noisy student training ( NST ) ( Xie et al. , 2020 ; Park et al. , 2020b ) is a self-training method . NST demonstrates the importance of the aggressive injection of noise into the student . Although not emphasized , no noise is injected into pseudo-labeling process of the teacher . We consider our work as an extension of self-training approach to the self-supervised learning regime . Instead of using the teacher to provide pseudo-labels , we utilize the teacher for pseudo-reference representation . Denoising autoencoders ( Vincent et al. , 2008 ) learn to recover a clean input from a corrupted version . However , speech data contain redundancy which is irrelevant to some speech applications such as speech recognition . Previous work ( Baevski et al. , 2019 ) shows that speech pre-training by recovering masked input speech features is not effective . In SPIRAL , we instead enforce latent representation of a corrupted input to resemble that of the corresponding clean input . Bootstrap Your Own Latent ( BYOL ) ( Grill et al. , 2020 ) is a self-supervised image representation learning method . The method is based on a teacher-student framework similar to MT . The authors refer to student network as online network and teacher network as target network . They observe that naive application of MT to self-supervised learning leads to trivial constant representation . They prevent the representation collapse by appending a predictor to the student network . The theory behind is under investigation ( Chen & He , 2021 ; Tian et al. , 2021 ) . Our method draws inspirations from BYOL and shares the similar architecture , but there are crucial differences . Instead of learning a single global representation for an image as in BYOL , SPIRAL learns a sequence of representation for an utterance . We aim for sequence applications such as speech recognition . In our preliminary experiments , we observe that appending a predictor to student network is not sufficient to prevent trivial constant representation for sequential representation learning . We use in-utterance contrastive loss ( Baevski et al. , 2020b ) combined with input position randomization to successfully avoid representation collapse . We still keep the predictor in SPIRAL , but only for the sake of performance improvement from our observation . Another difference is that BYOL does not perform representation denoising . BYOL applies perturbation , which they call augmentation , to both the inputs of the teacher and the student . We demonstrate that representation denoising is crucial for speech pre-training . When perturbation is applied to the teacher ’ s input , the effectiveness of speech pre-training degrades drastically . Wav2vec 2.0 ( Baevski et al. , 2020b ) is a self-supervised speech representation learning method which belongs to the masked prediction family . Masked prediction methods are effective for text pre-training ( Devlin et al. , 2019 ) , but not for speech pre-training when naively applied ( Baevski et al. , 2019 ) . The reason is that speech data contains redundancy such as speaker information , pronunciation variations , which are irrelevant to the semantic meaning of the utterance . To overcome this problem , wav2vec 2.0 perform masking in intermediate latent space and performs target discretization with a differentiable quantization scheme . However , quantization leads to a more complex model by introducing additional hyper-parameters and an additional diversity loss . SPIRAL does not utilize quantization , and still achieves competitive performance compared to wav2vec 2.0 . We hypothesize that aggressive down-sampling and learning by matching output representation may help to remove redundancy from the learned representation . We leave the investigation of whether target discretization could further improve SPIRAL for future work . Liang et al . ( 2018 ) demonstrates that under the supervised learning setting , enforcing noise-invariant representation by penalizing difference between clean and noisy data improves ASR model accuracy . 3 METHOD . 3.1 SELF-SUPERVISED PERTURBATION-INVARIANT REPRESENTATION LEARNING . ( SPIRAL ) Figure 1 shows the diagram of SPIRAL in the pre-training stage , where we use two neural networks , a student Fθ and a teacher Fθ′ . The weights of the teacher θ′ is the moving average of the weights of the student θ . At step t , the weights of the teacher θ′t are updated as θ′t ← αtθ′t−1 + ( 1− αt ) θt , ( 1 ) where αt determines the rate of weight updates . Given a speech utterance X = ( x1 , . . . , xT ) of length T , the student takes a perturbed version X̃ = s ( X ) = ( x̃1 , . . . , x̃T ) as input where s ( · ) is a perturbation function . The output of the student is a representation sequence Z = F ( X̃ ; θ ) = ( z1 , . . . , zT ) . The teacher takes the same utterance without perturbation as input and output another representation sequence Z ′ = F ( X ; θ′ ) = ( z′1 , . . . , z ′ T ) . For each representation zi ∈ Z , the student is trained to match the teacher ’ s representation z′i at the same position amongst k distracting samples . The distracting samples are randomly drawn from other positions of the same utterance in Z ′ , which is found to be more effective than samples drawn from an entire batch of utterances ( Baevski et al. , 2020b ) . The in-utterance contrastive loss is defined following Sohn ( 2016 ) ; Wu et al . ( 2018 ) as , L = − T∑ i=1 log exp ( φ ( zi , z ′ i ) /κ ) ∑ j∈Di exp ( φ ( zi , z ′ j ) /κ ) , ( 2 ) where φ ( a , b ) = aTb/‖a‖‖b‖ is cosine similarity , Di is the set of indices of distractors for the i-th position , and κ is the temperature parameter . However , applying in-utterance contrastive loss could cause a kind of representation collapse which we refer to as positional collapse . Contrastive candidates are sampled based on their positions in utterances . When a teacher ’ s representation z′i is correlated with its position i ( e.g. , correlation introduced by positional encoding in Transformer ) , the student could exploit this correlation to generate its representation zi solely based on the position index i , while ignoring content of the input . In this case , the model does not learn meaningful representation of the input content . Therefore , we prevent positional collapse by randomizing positions of teacher ’ s representation . In particular , we add random number of padding data at both ends of the input to the teacher to randomly shift the position information for each output representation z′i . The student thereby is unable to exploit the spurious position information to minimize the contrastive loss . Note that when calculating the contrastive loss , we exclude the corresponding representation of the padded data .

Summary: The paper introduces SPIRAL, a new method for self-supervised pre-training for speech. SPIRAL is based on the teacher-student framework similar to Mean Teacher (Tarvainen and Valpola, 2017) and BYOL (Grill et al., 2020) where the teacher’s weights are updated as a moving average of the student’s weights, but makes additional modifications for sequence tasks like speech:  * An in-utterance contrastive loss is used as the pre-training objective. * Position randomization of the teacher’s input is used to avoid representation collapse. * Ablation experiments are done to show that the predictor (which was essential in earlier works like BYOL and SimSiam to avoid collapse) can be replaced with a convolutional projection head without performance degradation.  Empirically, the main contributions of the paper are: 1. Achieving similar/better WER on LibriSpeech compared to wav2vec 2.0 with 35% of the training cost; and 2. Incorporating multi-condition training, which has been used in supervised training in the past, for noise-robust ASR.