Unsupervised Speech Recognition

1 Introduction . Speech recognition performance on the much studied English Librispeech benchmark [ Panayotov et al. , 2015 ] has seen rapid improvement over the last few years due to advances in model architectures [ Dong et al. , 2018 , Synnaeve et al. , 2020 , Gulati et al. , 2020 ] , semi-supervised learning [ Xu et al. , 2020b , Park et al. , 2020 ] and self-supervised learning [ van den Oord et al. , 2018 , Chung and Glass , 2018 , Chung et al. , 2019b , Baevski et al. , 2020c ] . However , all of these techniques require transcribed speech data which is not available for the vast majority of the nearly 7,000 languages of the world [ Lewis et al. , 2016 ] . As a result , speech recognition technology is only available for about 125 different languages [ Google , 2021 ] . On the other hand , humans learn a lot about speech simply by listening to others around them and without explicit supervision [ Werker and Tees , 1984 , Hirsh-Pasek et al. , 1987 , Polka and Werker , 1994 , Jusczyk et al. , 1999 , Johnson and Jusczyk , 2001 ] . Unsupervised learning has been very successful in machine translation resulting in systems that obtain remarkable accuracy given no labeled training data at all [ Conneau et al. , 2018 , Lample et al. , 2018 , Artetxe et al. , 2018 ] . Inspired by this , there has been some work on unsupervised speech recognition based on learning to align unlabeled text and audio [ Yeh et al. , 2019 ] or adversarial learning [ Liu et al. , 2018 , Chen et al. , 2019 ] . These approaches showed promising initial results but their error rates are still high , with evaluation being limited to the small-scale and clean TIMIT benchmark . In this work , we introduce a framework for unsupervised learning of speech recognition models . Wav2vec-U , or wav2vec Unsupervised , leverages self-supervised representations from wav2vec 2.0 [ Baevski et al. , 2020c ] to embed the speech audio and to segment the audio into units with a simple k-means clustering method ( see Figure 1 for an illustration of our approach ) . We find that the ∗Work done while at Facebook AI . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . quality of the audio representations is key to the success of unsupervised speech recognition . Similar to Liu et al . [ 2018 ] and Chen et al . [ 2019 ] , we learn a mapping between segments and phonemes using adversarial training but different to their work , we also enable the algorithm to label segments as silences . We also introduce an unsupervised cross-validation metric to enable model development without labeled development data . Our unsupervised speech recognition model , the generator , is very lightweight : it consists of a single temporal convolution comprising only about 90k parameters to which we input frozen wav2vec 2.0 representations . Experimental results demonstrate the viability of the framework for a variety of settings and languages . wav2vec-U improves the phone error rate ( PER ) on the small-scale TIMIT benchmark from 26.1 to 11.3 compared to the next best known unsupervised approach . To get a better sense of the performance compared to the best supervised methods , we measure performance on the larger Librispeech benchmark where our method achieves word error rate ( WER ) 5.9 on test-other . We also evaluate on six other European languages of the multilingual Librispeech benchmark [ Pratap et al. , 2020 ] and on three non-European low-resource languages . 2 Speech and Text Representations . Next , we describe how we build suitable speech and text representations for unsupervised learning . Good representations are essential to learning a mapping from speech to text without supervision . 2.1 Self-supervised Learning of Speech Audio Representations . In the first step , we learn representations of the speech audio signal using self-supervised learning . There has been a lot of recent work in this direction which has shown strong performance in extremely low-labeled data setups across a range of languages [ Conneau et al. , 2020 ] and tasks [ Fan et al. , 2021 , Pepino et al. , 2021 , Wang et al. , 2021 ] . Wav2vec 2.0 consists of a convolutional feature encoder f : X 7→ Z that maps a raw audio sequence X to latent speech representations z1 , . . . , zT , which a Transformer g : Z 7→ C then turns into context representations c1 , . . . , cT [ Baevski et al. , 2020b , a ] . Each zt represents about 25ms of audio strided by 20ms and the Transformer architecture follows BERT [ Vaswani et al. , 2017 , Devlin et al. , 2019 ] . During training , latent representations are discretized to q1 , . . . , qT with a quantization module Z 7→ Q to represent the targets in the objective . Quantization uses a Gumbel softmax to choose entries from two codebooks [ Jegou et al. , 2011 , Jang et al. , 2016 , Baevski et al. , 2020b ] . In our experiments , we use the publicly available English model pre-trained on 53k hours of LibriLight [ Kahn et al. , 2020b ] as well as XLSR-53 which was pre-trained on nearly 60k hours of speech audio in 53 languages [ Conneau et al. , 2020 ] . 2.2 Pre-processing and Embedding the Audio Data . Removing Silences . Most datasets we use for our experiments have audio data with silences . However , these parts of the audio do not correspond to any transcription and we therefore remove silences as much as possible . We apply rVAD , an unsupervised voice activity detection ( VAD ) model which determines the segments in the audio data corresponding to silences , and we remove these sections [ Tan et al. , 2020 ] . We ablate this choice in Appendix C. Speech Audio Representations . After silence removal , we embed the unlabeled speech audio with wav2vec 2.0 to obtain speech representations . Specifically , we use the representations of the context Transformer network c1 , . . . , cT ( § 2.1 ) . The context network contains 24 Transformer blocks and we denote the output of block l at time-step t as clt . Our goal is to learn a model which can map from audio representations clt to phonemes using no supervision . However , the representations of the uppermost block of wav2vec 2.0 may not be well suited for this task . These features are trained to directly predict masked latent representations spanning 25ms of speech audio which is much shorter than the typical duration of a phoneme . To get a better sense of this , we train supervised phoneme recognizers with a CTC loss [ Graves et al. , 2006 ] on top of the frozen representations of each of the 24 blocks of the English wav2vec 2.0 LARGE model pre-trained on Libri-Light . We then evaluate phone error rate ( PER ) with respect to the phonemized transcriptions of Librispeech dev-other . The classifier takes as input clt and contains a single softmax-normalized linear layer mapping to the phoneme inventory . Figure 2 shows that most of the first ten blocks as well as the final blocks provide very poor performance , while blocks 15-19 provide error rates below 9 % PER . Block 15 achieves the best error rate of 7.5 % PER . A similar insight has been used in the concurrent work of Hsu et al . [ 2021b ] . Appendix A shows that this choice generalizes to other languages . For brevity we drop the superscript l and refer to block 15 representations simply as c1 , . . . , cT . 2.3 Segmenting the Audio Signal . Once the speech signal is embedded , we identify segments corresponding to meaningful units that can be mapped to phonemes . Segmentation has been shown to be crucial in prior work [ Chung et al. , 2018 ] since the right boundaries in the input representations make it more aligned to phonetic sequences . There has been a lot of prior work in unsupervised speech segmentation [ Kamper et al. , 2017a , b , Rasanen et al. , 2015 , Kreuk et al. , 2020 ] but here we simply use a method based on clustering the wav2vec 2.0 speech representations c1 , . . . , cT . In a first step , we collect all the speech representations for the unlabeled speech data and perform k-means clustering to identify K = 128 clusters . We use the FAISS library to do fast clustering on GPUs [ Johnson et al. , 2019 ] . Next , each ct is labeled with the corresponding cluster ID it ∈ { 1 , . . . , K } and we introduce speech segment boundaries whenever the cluster ID changes . Once the speech audio representations are segmented , we compute a 512-dimensional PCA over all speech representations output by wav2vec 2.0 for the training set . Next , we mean-pool the PCA representations for a particular segment to obtain an average representation of the segment . The PCA retains only the most important features and we found this to be effective . Segment boundaries are noisy due to the lack of supervision and we therefore found it useful to also mean-pool pairs of adjacent segment representations to increase robustness . This results in sequences of speech segment representation S = s1 , . . . , sT , S ∼ S for a given utterance . Appendix B shows an illustration of the segmentation strategy on an actual example as well as a quantitative evaluation of the strategy compared to human segmented data . 2.4 Pre-processing the Text Data . Similar to how we segment the unlabeled speech audio data into suitable units for unsupervised learning , we do the same for the unlabeled text data . We apply two pre-processing steps to the text data : phonemization and silence token insertion . Phonemes characterize the different sounds which distinguish words from each other , e.g. , for the word cat there are three phonemes corresponding to the three distinct sounds in the pronunciation of the word : /K/ , /AE/ , /T/ . We phonemize the text data because we found it easier to learn a mapping between speech audio and the different sounds of a word rather than between audio and words or letters . Phonemization converts a sequence of words Y into a sequence of phonemes P = [ p1 , · · · , pM ] , where pm ∈ O and O is the phoneme inventory . We use off-the-shelf tools for this step which we detail in Appendix § E.2 . The unlabeled speech audio data is pre-processed by applying unsupervised silence removal . However , this process is not always accurate and many silences in the speech audio remain . To deal with this , we enable the unsupervised model to label some segments with a phonemic silence token ( SIL ; § 3.1 ) . However , the phonemized unlabeled text data does not contain any silence tokens and this may pose difficulties for adversarial learning ( § 3 ) . We remedy this by inserting silence markers at the beginning and end of the phonemized unlabeled text data ; we also randomly insert SIL between words , or groups of phonemes corresponding to words at a rate of 25 % . Appendix C evaluates these choices . 3 Unsupervised Learning . We use adversarial training to train an unsupervised speech recognition model using the representations of the unlabeled speech audio data and the unlabeled phonemized text data [ Liu et al. , 2018 , Chen et al. , 2019 ] . In the following , we detail the model architecture , the training objective as well as the unsupervised cross-validation metric we developed .

Summary: This paper builds upon previous work on unsupervised speech recognition. This work introduces several improvements which are crucial to achieve impressive performance on a variety of datasets. These improvements are: using a new strategy for cluster segmentation for better alignment with the phonemes, matching the number of silences for the GAN generator. Additionally, this work uses a lightweight GAN. A distinguishing features of this work comparing to prior ones is that it proposes and uses an unsupervised metric for the development set for the model selection.   The experiments include several popular datasets such as LibriSpeech, TIMIT, as well as low resource language datasets for Tatar, Kyrgyz, Swahili. The proposed method shows performance comparable to some supervised methods and superior to previous unsupervised ones.