Unsupervised Learning of Efficient and Robust Speech Representations

1 INTRODUCTION . The representation of the input to machine learning models strongly determines the difficulty faced by the learning algorithm , how much data the learner will require to find a good solution , and whether it will generalize out of sample and out of the domain of the training data . Representations ( or features ) that encode relevant information about data enable models to achieve good performance on downstream tasks , while representations that are invariant to factors that are not relevant to downstream tasks can further improve generalization and sample efficiency . Traditionally , many invariances were hard-coded in feature extraction methods . For example in image representations , geometric and photometric invariance has been investigated ( Mundy et al. , 1992 ; Van De Weijer et al. , 2005 ) . For acoustic representations , it is known that the standard MFCC features are sensitive to additive noise and many modifications have been proposed to overcome those limitations ( Dev & Bansal , 2010 ; Kumar et al. , 2011 ) . In this paper , we investigate the potential of unsupervised representation learning for speech from large-scale , unlabelled and noisy data using bidirectional contrastive predictive coding ( §2–§3 ) . In previous work , van den Oord et al . ( 2018 ) showed that the unidirectional variant of this objective results in representations that are useful for frame-level phoneme classification ( even with simple linear classifiers ) , but in this paper , we go further : we vastly increase the scale of the pretraining data to be larger and much noisier and more diverse , and we also evaluate the learned representations in a speech recognition ( ASR ) system , rather than on the less interpretable per-frame phone prediction . In our experiments ( §4 ) , we find that our acoustic representations improve ASR system performance relative to standard spectrogram and log-filterbank features , and also relative to features pretrained from smaller corpora that are more matched to the training/test domain . A previous attempt at integrating pretrained acoustic representations into an ASR system from Schneider et al . ( 2019 ) found that they were able to outperform mel-frequency features in both sample efficiency and overall performance , but their work used only clean , read English speech ( from the LibriSpeech dataset ) as pretraining data , and evaluated the features when used to recognize speech in a rather similar domain ( read Wall Street Journal text ) . They found that increases in the amount of pretraining data led , in some cases , to diminished performance . However , we find the opposite : our best pretrained features are those learned from the largest and noisiest speech data , with performance far surpassing those obtained by pretraining only on the relatively clean LibriSpeech . In addition to the standard ASR evaluation where the systems are evaluated on the same domain as they are trained on , we look at two further aspects of our features : their sample efficiency and whether their use confers robustness to domain shifts in the resulting system . Here again , we find compelling results . First , for a smaller speech recognition task ( using a simpler convolutional architecture than is common ) , an ASR model trained with our representations only needs 10 % of the labelled data to achieve the same result as a the same model trained on spectrogram features using 100 % of the training data . Second , we evaluate the robustness of our representations by estimating how invariant they are to domain and language shifts . To do so , an ASR model is trained using our representations on one dataset but evaluated on the test sets of other datasets . In this experiment , we find that the representations derived from the large pretraining dataset lead the ASR model to be much more robust to domain shifts , compared to both log filterbank features as well as to pretraining just on LibriSpeech . Finally , we also evaluate the representations on four low-resource African languages ( i.e . Amharic , Fongbe , Swahili , Wolof ) and show that our representations significantly outperform both standard features and those pretrained only on clean English data . In summary , we confirm several increasingly common patterns that may be discerned in the literature on unsupervised representation learning , across a variety of modalities . First , scale matters : good representation learning requires a large amount of data . Second , contextualized representations ( that are sensitive to both past and future ) , are valuable . And finally , unsupervised representations consistently improve sample efficiency , performance , and robustness of downstream tasks . 2 CONTRASTIVE PREDICTIVE CODING : CPC . Unsupervised representation learning methods rely on differentiable objectives which quantify the degree to which representations have succeeded at capturing the relevant characteristics in data . Mutual information measures relationships between random variables ( Fano & Hawkins , 1961 ) . Mutual information maximization techniques , that learn representations that describe data by maximizing mutual information between data and representation variables , have been explored for a long time in unsupervised representation learning ( Linsker , 1988 ; Bell & Sejnowski , 1995 ) . However , since the exact computation of mutual information is not tractable for continuous variables , many estimators have been proposed for enabling unsupervised representation learning with neural networks ( Belghazi et al. , 2018 ; van den Oord et al. , 2018 ; Hjelm et al. , 2019 ) . Contrastive predictive coding ( van den Oord et al. , 2018 , CPC ) is a mutual information maximization method that has been successfully applied to many modalities such as images and speech ( Hénaff et al. , 2019 ; Bachman et al. , 2019 ; Schneider et al. , 2019 ) . The objective is designed to extract features that allow the model to make long-term predictions about future observations . This is done by maximizing the mutual information of these features with those extracted from future timesteps . The intuition is that the representations capture different levels of structure dependent on how far ahead the model predicts . For example , if the model only predicts a few steps ahead , the resulting representations can capture local structures . On the other hand , if the model predicts further in the future , the representations will need to infer “ slow features ” ( Wiskott & Sejnowski , 2002 ) ; more global structures such as phonemes , words and utterances in speech . The overall unsupervised learning process is visualized in Figure . 1 . Given a raw audio signal of length L ( x = x1 , x2 , . . . , xL , xi ∈ R where xi represents the acoustic amplitude at time i ) , a function genc encodes the audio signals into vector representations ( z = z1 , z2 . . . , zM , z ∈ Rdz ) . Next , an auto-regressive function gar , such as a recurrent neural network , summarizes the past representations and produces context vectors ( c = c1 , c2 . . . , cM , c ∈ Rdc ) . The representations are learned to maximize mutual information between context vectors ( c ) and future latent representations ( z ) as follows : ∑ t , k I ( ct , zt+k ) = ∑ t , k ∑ ct , zt+k p ( ct , zt+k | k ) log p ( zt+k | ct , k ) p ( zt+k ) . Since the mutual information is not tractable for high dimensional data , it is common to use a lower-bound on the mutual information such as InfoNCE ( van den Oord et al. , 2018 ) which is a loss function based on noise contrastive estimation ( Gutmann & Hyvärinen , 2010 ) . Given a set Z = { z1 , . . . zN } which contains one positive sample from p ( zt+k|ct ) and N − 1 negative samples from a “ noise ” distribution p ( z ) , the approximated lower-bound is written as I ( ct , zt+k ) ≥ EZ [ log fk ( ct , zt+k ) 1 N ∑ z̃∈Z fk ( ct , z̃ ) ] = LNCEtk , ( 1 ) where fk ( ct , zt+k ) is a scoring function . We used the standard log-bilinear model as follows : fk ( ct , zt+k ) = exp ( c T t Wkzt+k ) . The loss function we maximize is a sum of the InfoNCE loss for each step , LNCE = ∑ t ∑ k LNCEtk and the negatives are uniformly sampled from representations in the same audio signal ( z ) and/or mini-batch . 3 METHODS In this section , we describe our models and objectives for unsupervised representation learning and downstream speech recognition . First , an acoustic feature extractor is trained with a bidirectional variant of contrastive predictive coding ( CPC ) on an unlabeled audio dataset . Next , the parameters of this model are frozen and its output representations are used as input to train various speech recognition models , potentially on a different or smaller labeled dataset ( Figure 1 ) . 3.1 UNSUPERVISED LEARNING WITH BI-DIRECTIONAL CONTRASTIVE PREDICTIVE CODING . Following the success of bidirectional models in representation learning ( Peters et al. , 2018 ; Devlin et al. , 2019 ) , we extend the original CPC method explained above with bidirectional context networks . The encoder function genc is shared for both directions but there are two auto-regressive models ( gfwdar and g bwd ar ) which read encoded observations ( z ) from the forward and backward contexts , respectively . The forward and backward context representations cfwdt , c bwd t are learned with separate InfoNCE losses . When they are used for downstream tasks , a concatenation of two representations ct = [ c fwd t ; c bwd t ] is used . A similar technique has been used in image representation learning where representations are learned along different spatial dimensions ( Hénaff et al. , 2019 ) . All audio signals have a sampling rate of 16kHz and we normalize the mean and variance of the input signals over each utterance in order to mitigate volume differences between samples . For architectures , we use encoder and auto-regressive models similar to Schneider et al . ( 2019 ) . The encoder function genc , is a stack of causal convolutions with kernel sizes ( 10 , 8 , 4 , 4 , 4 , 1 , 1 ) and stride sizes ( 5 , 4 , 2 , 2 , 2 , 1 , 1 ) , corresponding to a receptive field of 10 ms of audio . For auto-regressive functions , we use a 13 layer causal convolution architecture with kernel sizes ( 1 , 2 , . . . , 12 , 13 ) and stride size 1 , for both forward and backward functions . Layer-normalization across the temporal and feature dimensions is applied to every layer . Also , each layer has dense skip connections with layers below as in DenseNet ( Huang et al. , 2017 ) . The objective function we optimize is the sum of the forward and backward InfoNCE losses ( eq.1 ) . 3.2 SEMI-SUPERVISED SPEECH RECOGNITION . Once the acoustic representations are trained , the resulting context vectors ( c ) are used as inputs to character-level speech recognition models which predict transcriptions of audio-signals character by character . The model first predicts frame-level character probabilities with a series of convolution layers while the CTC forward algorithm ( Graves et al. , 2006 ) calculates conditional probabilities of a transcription given an audio signal . The model parameters are trained to maximize likelihood of the data . The training is terminated when the word error rate on development set stops improving or the model is trained more than certain epochs . The models are evaluated on standard word error rate on test data . During training , the parameters in speech recognition models are trained with supervision but the parameters of the pretrained models are fixed . For decoding , we use greedy CTC decoding without a language model ( LM ) in order to focus on discerning the effects of the acoustic representations , although we also include results with a 4-gram LM to facilitate comparisons with published results . Common practice in unsupervised representation learning is to evaluate learned representations using a linear classifier . However , we found a simple linear layer followed by a CTC decoder does not have enough capacity to recognize speech . Thus , for our first set of experiments we use a smaller version of DeepSpeech2 ( Amodei et al. , 2016 ) to predict the frame-level character probabilities . The model has two 2d-convolutions with kernel sizes ( 11 , 41 ) and ( 11 , 21 ) and stride sizes ( 2 , 2 ) and ( 1 , 2 ) and one recurrent neural network ( GRU ) on top of the output from the convolution layers . A linear transformation and a softmax function are applied to predict frame-level character probabilities . We refer to DeepSpeech2 small for the model specifics ( Amodei et al. , 2016 ) . In order to further investigate how the representations interact with larger speech recognition models , we use the time-delay neural networks ( TDNN ) that are commonly used in speech recognition ( Collobert et al. , 2016 ; Kuchaiev et al. , 2018 ) . These consist of 17 layers of 1d-convolutions followed by 2 fully connected layers . Refer to OpenSeq2Seq for a detailed description.1 These large models have enough capacity to learn to recognize speech from log-filterbank features with purely supervised learning on sufficient data , so they represent a challenging test case for learned representations .

Summary: This work uses contrastive predictive coding (CPC) to learn unsupervised speech representations on large amounts of unlabelled speech data and then uses the resulting features in downstream speech recognition systems. Unlabelled data is obtained from several sources (spanning different languages). Supervised systems are then built on top of these features and sample-efficiency and cross-domain robustness is investigated using English data sets. Finally, the approach is applied to four African languages.