SPIRAL: Self-supervised Perturbation-Invariant Representation Learning for Speech Pre-Training

1 INTRODUCTION . Industrial-scale automatic speech recognition ( ASR ) systems are usually trained with ten-thousands of hours of hand-transcribed speech data ( Galvez et al. , 2021 ) . However , labeling speech data is expensive and time-consuming , especially for languages with small speaker populations , or for specific domains ( e.g. , legal , financial , scientific ) . Recently , methods of utilizing unlabeled speech data to improve speech recognition system have achieved remarkable progress . Amongst them , self-training ( Manohar et al. , 2015 ; Kahn et al. , 2020a ; Synnaeve et al. , 2020a ; Chen et al. , 2020b ; Xu et al. , 2020 ; Park et al. , 2020b ; Xiao et al. , 2021 ) , also known as pseudo-labeling , starts by training an ASR model with labeled speech data , which is referred to as teacher model . Then the teacher model , usually combined with a language model ( LM ) , is used to produce pseudo-labels for unlabeled speech data . Finally , the labeled data and the pseudo-labeled data are combined to train a new model , which is referred to as student model . The process is repeated by taking the student model as the teacher in next iteration . Another line of work is speech pre-training ( van den Oord et al. , 2019 ; Chung & Glass , 2020 ; Wang et al. , 2020 ; Baevski et al. , 2020b ; Liu et al. , 2020 ) . Pre-training learns speech representation from unlabeled data in a self-supervised way . The pre-trained model is then fine-tuned on the labeled data . Self-training and pre-training are complementary as shown in recent work ( Xu et al. , 2021 ; Zhang et al. , 2020 ) . In this paper , we introduce a new speech pre-training method which works by learning denoising representation of perturbed data with the teacher-student framework , named as Self-supervised Perturbation-Invariant Representation Learning ( SPIRAL ) . Compared to state-of-the-art speech pretraining methods such as wav2vec 2.0 ( Baevski et al. , 2020b ) and HuBERT ( Hsu et al. , 2021 ) , our method allows end-to-end training with a single contrastive loss , and without relying on discrete unit discovery techniques such as vector quantization ( Jégou et al. , 2011 ; Baevski et al. , 2020a ; b ) or iterative clustering process ( Hsu et al. , 2021 ) . We apply multi-condition training with SPIRAL ( Seltzer et al. , 2013 ; Ko et al. , 2015 ) to improve noise-robustness for the downstream speech tasks . SPIRAL is motivated by the observation that human tolerates speech perturbations or distortions fairly well . For example , people can communicate effectively in a noisy environment , or over a distorted telephone channel . Therefore , we hypothesize that by learning representation invariant to perturbation , the model will learn high-level representation which can enhance speech applications . To learn perturbation-invariant representation in a self-supervised way , we employ a teacher-student framework similar to Tarvainen & Valpola ( 2017 ) . During pre-training , given a speech utterance , we guide the student network which consumes the perturbed utterance to learn from the teacher network which consumes the clean utterance . The student is trained to produce denoised representation of the perturbed utterance similar to teacher ’ s representation of the clean utterance . Meanwhile , the teacher , which shares the same model architecture with student , is updated as moving average of the student ’ s weights over past training steps . We apply the in-utterance contrastive loss to avoid model collapse to trivial constant representation ( Chopra et al. , 2005 ) . As speech utterance are sequential data , there is another possible trivial solution which we call positional collapse . Positional collapse occurs when the student “ cheats ” by exploiting position correlation in teacher ’ s representation to minimize the loss , while ignoring the content of the input utterance . To prevent positional collapse , we propose position randomization by adding random number of paddings on both sides of input utterance to the teacher . Large-scale speech pre-training is computationally demanding . To reduce computation cost , we adopt a gradual down-sampling strategy in SPIRAL model , which has been verified effective in speech recognition literatures with negligible performance degradation ( Peddinti et al. , 2018 ; Han et al. , 2020b ; Huang et al. , 2020 ) . We also speculate that aggressive down-sampling helps to remove redundancy in speech . To evaluate the effectiveness of SPIRAL , we conduct experiments on LibriSpeech and Libri-Light datasets . By training a small convolutional classifier on the representation of a frozen SPIRAL model , we can achieve WER of 3.5 % and 6.4 % on Librispeech test-clean and test-other respectively . SPIRAL achieves competitive or better results compared to state-of-the-art speech pre-training methods , while being much more training-efficient . We also demonstrate that multi-condition pretrained SPIRAL are more robust to noisy speech with 9.0 % - 13.3 % relative word error rate ( WER ) reduction on real noisy test data from ChiME-3 ( Barker et al. , 2015 ) , compared to the model applying multi-condition training solely in fine-tuning stage . 2 RELATED WORK . Mean Teacher ( MT ) ( Tarvainen & Valpola , 2017 ) proposes using a student network to learn from a teacher network which is the moving average version of the student in the semi-supervised learning setting . The authors apply a supervised loss for labeled data and a consistency loss between teacher and student predictions for unlabeled data . However , direct application of MT to self-supervised learning leads to representation collapse ( Grill et al. , 2020 ) . Noisy student training ( NST ) ( Xie et al. , 2020 ; Park et al. , 2020b ) is a self-training method . NST demonstrates the importance of the aggressive injection of noise into the student . Although not emphasized , no noise is injected into pseudo-labeling process of the teacher . We consider our work as an extension of self-training approach to the self-supervised learning regime . Instead of using the teacher to provide pseudo-labels , we utilize the teacher for pseudo-reference representation . Denoising autoencoders ( Vincent et al. , 2008 ) learn to recover a clean input from a corrupted version . However , speech data contain redundancy which is irrelevant to some speech applications such as speech recognition . Previous work ( Baevski et al. , 2019 ) shows that speech pre-training by recovering masked input speech features is not effective . In SPIRAL , we instead enforce latent representation of a corrupted input to resemble that of the corresponding clean input . Bootstrap Your Own Latent ( BYOL ) ( Grill et al. , 2020 ) is a self-supervised image representation learning method . The method is based on a teacher-student framework similar to MT . The authors refer to student network as online network and teacher network as target network . They observe that naive application of MT to self-supervised learning leads to trivial constant representation . They prevent the representation collapse by appending a predictor to the student network . The theory behind is under investigation ( Chen & He , 2021 ; Tian et al. , 2021 ) . Our method draws inspirations from BYOL and shares the similar architecture , but there are crucial differences . Instead of learning a single global representation for an image as in BYOL , SPIRAL learns a sequence of representation for an utterance . We aim for sequence applications such as speech recognition . In our preliminary experiments , we observe that appending a predictor to student network is not sufficient to prevent trivial constant representation for sequential representation learning . We use in-utterance contrastive loss ( Baevski et al. , 2020b ) combined with input position randomization to successfully avoid representation collapse . We still keep the predictor in SPIRAL , but only for the sake of performance improvement from our observation . Another difference is that BYOL does not perform representation denoising . BYOL applies perturbation , which they call augmentation , to both the inputs of the teacher and the student . We demonstrate that representation denoising is crucial for speech pre-training . When perturbation is applied to the teacher ’ s input , the effectiveness of speech pre-training degrades drastically . Wav2vec 2.0 ( Baevski et al. , 2020b ) is a self-supervised speech representation learning method which belongs to the masked prediction family . Masked prediction methods are effective for text pre-training ( Devlin et al. , 2019 ) , but not for speech pre-training when naively applied ( Baevski et al. , 2019 ) . The reason is that speech data contains redundancy such as speaker information , pronunciation variations , which are irrelevant to the semantic meaning of the utterance . To overcome this problem , wav2vec 2.0 perform masking in intermediate latent space and performs target discretization with a differentiable quantization scheme . However , quantization leads to a more complex model by introducing additional hyper-parameters and an additional diversity loss . SPIRAL does not utilize quantization , and still achieves competitive performance compared to wav2vec 2.0 . We hypothesize that aggressive down-sampling and learning by matching output representation may help to remove redundancy from the learned representation . We leave the investigation of whether target discretization could further improve SPIRAL for future work . Liang et al . ( 2018 ) demonstrates that under the supervised learning setting , enforcing noise-invariant representation by penalizing difference between clean and noisy data improves ASR model accuracy . 3 METHOD . 3.1 SELF-SUPERVISED PERTURBATION-INVARIANT REPRESENTATION LEARNING . ( SPIRAL ) Figure 1 shows the diagram of SPIRAL in the pre-training stage , where we use two neural networks , a student Fθ and a teacher Fθ′ . The weights of the teacher θ′ is the moving average of the weights of the student θ . At step t , the weights of the teacher θ′t are updated as θ′t ← αtθ′t−1 + ( 1− αt ) θt , ( 1 ) where αt determines the rate of weight updates . Given a speech utterance X = ( x1 , . . . , xT ) of length T , the student takes a perturbed version X̃ = s ( X ) = ( x̃1 , . . . , x̃T ) as input where s ( · ) is a perturbation function . The output of the student is a representation sequence Z = F ( X̃ ; θ ) = ( z1 , . . . , zT ) . The teacher takes the same utterance without perturbation as input and output another representation sequence Z ′ = F ( X ; θ′ ) = ( z′1 , . . . , z ′ T ) . For each representation zi ∈ Z , the student is trained to match the teacher ’ s representation z′i at the same position amongst k distracting samples . The distracting samples are randomly drawn from other positions of the same utterance in Z ′ , which is found to be more effective than samples drawn from an entire batch of utterances ( Baevski et al. , 2020b ) . The in-utterance contrastive loss is defined following Sohn ( 2016 ) ; Wu et al . ( 2018 ) as , L = − T∑ i=1 log exp ( φ ( zi , z ′ i ) /κ ) ∑ j∈Di exp ( φ ( zi , z ′ j ) /κ ) , ( 2 ) where φ ( a , b ) = aTb/‖a‖‖b‖ is cosine similarity , Di is the set of indices of distractors for the i-th position , and κ is the temperature parameter . However , applying in-utterance contrastive loss could cause a kind of representation collapse which we refer to as positional collapse . Contrastive candidates are sampled based on their positions in utterances . When a teacher ’ s representation z′i is correlated with its position i ( e.g. , correlation introduced by positional encoding in Transformer ) , the student could exploit this correlation to generate its representation zi solely based on the position index i , while ignoring content of the input . In this case , the model does not learn meaningful representation of the input content . Therefore , we prevent positional collapse by randomizing positions of teacher ’ s representation . In particular , we add random number of padding data at both ends of the input to the teacher to randomly shift the position information for each output representation z′i . The student thereby is unable to exploit the spurious position information to minimize the contrastive loss . Note that when calculating the contrastive loss , we exclude the corresponding representation of the padded data .

Summary: This paper describes a self supervision training approach to pretrain speech encoders.  This is related to recent work on wav2vec 2.0, and SimCLR.  The contrastive loss is similar to wav2vec 2.0, while the invariance to perturbation (i.e. augmentation) is akin to SimCLR. Performance is competitive with wav2vec 2 on Librispeech.