Fully Convolutional Speech Recognition

Current state-of-the-art speech recognition systems build on recurrent neural networks for acoustic and/or language modeling, and rely on feature extraction pipelines to extract mel-filterbanks or cepstral coefficients. In this paper we present an alternative approach based solely on convolutional neural networks, leveraging recent advances in acoustic models from the raw waveform and language modeling. This fully convolutional approach is trained end-to-end to predict characters from the raw waveform, removing the feature extraction step altogether. An external convolutional language model is used to decode words. On Wall Street Journal, our model matches the current state-of-the-art. On Librispeech, we report state-of-the-art performance among end-to-end models, including Deep Speech 2 trained with 12 times more acoustic data and significantly more linguistic data.

section_name: Introduction
paragraphs: Recent work on convolutional neural network architectures showed that they are competitive with recurrent architectures even on tasks where modeling long-range dependencies is critical, such as language modeling BIBREF0 , machine translation BIBREF1 , BIBREF2 and speech synthesis BIBREF3 . In end-to-end speech recognition however, recurrent architectures are still prevalent for acoustic and/or language modeling BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8 .
There is a history of using convolutional networks in speech recognition, but only as part of an otherwise more traditional pipeline. They have been first introduced as TDNNs to predict phoneme classes BIBREF9 , and later to generate HMM posteriorgrams BIBREF10 . They have more recently been used in end-to-end frameworks, but only in combination with recurrent layers BIBREF6 , or n-gram language models BIBREF11 , or for phone recognition BIBREF12 , BIBREF13 . Nonetheless, convolutional architectures are prevalent when learning from the raw waveform BIBREF14 , BIBREF15 , BIBREF16 , BIBREF13 , BIBREF17 , because they naturally model the computation of standard features such as mel-filterbanks. Given the evidence that they are also suitable on long-range dependency tasks, we expect convolutional neural networks to be competitive at all levels of the speech recognition pipeline.
In this paper, we present a fully convolutional approach to end-to-end speech recognition. Building on recent advances in convolutional learnable front-ends for speech BIBREF13 , BIBREF17 , convolutional acoustic models BIBREF11 , and convolutional language models BIBREF0 , the paper has four main contributions:
section_name: Model
paragraphs: Our approach, described in this section, is illustrated in Fig. FIGREF5 .
section_name: Convolutional Front end
paragraphs: Several proposals to learn the front-end of speech recognition systems have been made BIBREF15 , BIBREF16 , BIBREF13 , BIBREF17 . Following the comparison in BIBREF17 , we consider their best architecture, called "scattering based" (hereafter refered to as learnable front-end). The learnable front-end contains first a convolution of width 2 that emulates the pre-emphasis step used in mel-filterbanks. It is followed by a complex convolution of width 25ms and INLINEFORM0 filters. After taking the squared absolute value, a low-pass filter of width 25ms and stride 10ms performs decimation. The front-end finally applies a log-compression and a per-channel mean-variance normalization (equivalent to an instance normalization layer BIBREF18 ). Following BIBREF17 , the "pre-emphasis" convolution is initialized to INLINEFORM1 , and then trained with the rest of the network. The low-pass filter is kept constant to a squared Hanning window, and the complex convolutional layer is initialized randomly. In addition to the INLINEFORM2 filters used by BIBREF17 , we experiment with INLINEFORM3 filters. Notice that since the stride is the same as for mel-filterbanks, acoustic models on top of the learnable front-ends can also be applied to mel-filterbanks (simply modifying the number of input channels if INLINEFORM4 ).
section_name: Convolutional Acoustic Model
paragraphs: The acoustic model is a convolutional neural network with gated linear units BIBREF0 , which is fed with the output of the learnable front-end. Following BIBREF11 , the networks uses a growing number of channels, and dropout BIBREF19 for regularization. These acoustic models are trained to predict letters directly with the Auto Segmentation Criterion (ASG) BIBREF20 . The only differences between the WSJ and Librispeech models are their depth, the number of feature maps per layer, the receptive field and the amount of dropout.
section_name: Convolutional Language Model
paragraphs: The convolutional language model (LM) is the GCNN-14B from BIBREF0 , which achieved competitive results on several language modeling benchmarks. The network contains 14 convolutional residual blocks BIBREF21 with a growing number of channels, and uses gated linear units as activation function.
The language model is used to score candidate transcriptions in addition to the acoustic model in the beam search decoder described in the next section. Compared to n-gram LMs, convolutional LMs allow for much larger context sizes. Our detailed experiments study the effect of context size on the final speech recognition performance.
section_name: Beam-search decoder
paragraphs: We use the beam-search decoder presented in BIBREF11 to generate word sequences given the output from our acoustic model. The decoder finds the word transcription INLINEFORM0 to maximize: INLINEFORM1 
where INLINEFORM0 is the value for the INLINEFORM1 th frame in the path leading to INLINEFORM2 and INLINEFORM3 is the (unnormalized) acoustic model score of the transcription INLINEFORM4 . The hyperparameters INLINEFORM5 respectively control the weight of the language model, the word insertion reward, and the silence insertion penalty. The other parameters are the beam size and the beam score, a threshold under which candidates are discarded even if the beam is not full. These are chosen according to a trade-off between (near-)optimality of the search and computational cost.
section_name: Experiments
paragraphs: We evaluate our approach on the large vocabulary task of the Wall Street Journal (WSJ) dataset BIBREF25 , which contains 80 hours of clean read speech, and Librispeech BIBREF26 , which contains 1000 hours with separate train/dev/test splits for clean and noisy speech. Each dataset comes with official textual data to train language models, which contain 37 million tokens for WSJ, 800 million tokens for Librispeech. Our language models are trained separately for each dataset on the official text data only. These datasets were chosen to study the impact of the different components of our system at different scales of training data and in different recording conditions.
The models are evaluated in Word Error Rate (WER). Our experiments use the open source codes of wav2letter for the acoustic model, and fairseq for the language model. More details on the experimental setup are given below.
Baseline Our baseline for each dataset follows BIBREF11 . It uses the same convolutional acoustic model as our approach but a mel-filterbanks front-end and a 4-gram language model.
Training/test splits On WSJ, models are trained on si284. nov93dev is used for validation and nov92 for test. On Librispeech, we train on the concatenation of train-clean and train-other. The validation set is dev-clean when testing on test-clean, and dev-other when testing on test-other.
Acoustic model architecture The architecture for the convolutional acoustic model is the "high dropout" model from BIBREF11 for Librispeech, which has 19 layers in addition to the front-end (mel-filterbanks for the baseline, or the learnable front-end for our approach). On WSJ, we use the lighter version used in BIBREF17 , which has 17 layers. Dropout is applied at each layer after the front-end, following BIBREF20 . The learnable front-end uses 40 or 80 filters. Language model architecture As described in Section SECREF8 , we use the GCNN-14B model of BIBREF0 with dropout at each convolutional and linear layer on both WSJ and Librispeech. We keep all the words (162K) in WSJ training corpus. For Librispeech, we only use the most frequent 200K tokens (out of 900K).
Hyperparameter tuning The acoustic models are trained following BIBREF11 , BIBREF17 , using SGD with a decreasing learning rate, weight normalization and gradient clipping at 0.2 and a momentum of 0.9. The language models are trained with Nesterov accelerated gradient BIBREF27 . Following BIBREF0 , we also use weight normalization and gradient clipping.
The parameters of the beam search (see Section SECREF9 ) INLINEFORM0 , INLINEFORM1 and INLINEFORM2 are tuned on the validation set with a beam size of 2500 and a beam score of 26 for computational efficiency. Once INLINEFORM3 are chosen, the test WER is computed with a beam size of 3000 and a beam score of 50.
section_name: Word Error Rate results
paragraphs: Table TABREF11 shows Word Error Rates (WER) on WSJ for the current state-of-the-art and our models. The current best model trained on this dataset is an HMM-based system which uses a combination of convolutional, recurrent and fully connected layers, as well as speaker adaptation, and reaches INLINEFORM0 WER on nov92. DeepSpeech 2 shows a WER of INLINEFORM1 but uses 150 times more training data for the acoustic model and huge text datasets for LM training. Finally, the state-of-the-art among end-to-end systems trained only on WSJ, and hence the most comparable to our system, uses lattice-free MMI on augmented data (with speed perturbation) and gets INLINEFORM2 WER. Our baseline system, trained on mel-filterbanks, and decoded with a n-gram language model has a INLINEFORM3 WER. Replacing the n-gram LM by a convolutional one reduces the WER to INLINEFORM4 , and puts our model on par with the current best end-to-end system. Replacing the speech features by a learnable frontend finally reduces the WER to INLINEFORM5 and then to INLINEFORM6 when doubling the number of learnable filters, improving over DeepSpeech 2 and matching the performance of the best HMM-DNN system.
Table TABREF10 reports WER on the Librispeech dataset. The CAPIO BIBREF22 ensemble model combines the lattices from 8 individual HMM-DNN systems (using both convolutional and LSTM layers), and is the current state-of-the-art on Librispeech. CAPIO (single) is the best individual system, selected either on dev-clean or dev-other. The sequence-to-sequence baseline is an encoder-decoder with attention and a BPE-level BIBREF28 LM, and currently the best end-to-end system on this dataset. We can observe that our fully convolutional model improves over CAPIO (Single) on the clean part, and is the current best end-to-end system on test-other with an improvement of INLINEFORM0 absolute. Our system also outperforms DeepSpeech 2 on both test sets by a significant margin. An interesting observation is the impact of each convolutional block. While replacing the 4-gram LM by a convolutional LM improves similarly on the clean and noisier parts, learning the speech frontend gives similar performance on the clean part but significantly improves the performance on noisier, harder utterances, a finding that is consistent with previous literature BIBREF15 .
section_name: Analysis of the convolutional language model
paragraphs: Since this paper uses convolutional language models for speech recognition systems for the first time, we present additional studies of the language model in isolation. These experiments use our best language model on Librispeech, and evaluations in WER are carried out using the baseline system trained on mel-filterbanks. The decoder parameters are tuned using the grid search described in Section SECREF3 , a beam size is fixed to 2500 and a beam score to 30.
Correlation between perplexity and WER Figure FIGREF18 shows the correlation between perplexity and WER as the training progresses. As perplexity decreases, the WER on both dev-clean and dev-other also decreases following the same trend. It illustrates that perplexity on the linguistic data is a good surrogate of the final performance of the speech recognition pipeline. Architectural choices or hyper-parameter tuning can thus be carried out mostly using perplexity alone.
Influence of context size By limiting the context passed into the LM from the decoder, Table TABREF19 reports WER obtained for context sizes ranging from 3 (comparable to the n-gram baseline) to 50 for our best language model. The WER decreases monotonically until a context size of about 20, and then almost stays still. We observe that the convolutional LM already improves on the n-gram model even with the same context size. Increasing the context gives a significant boost in performance, with the major gains obtained between a context of 3 to 9 ( INLINEFORM0 absolute WER).
section_name: Conclusion
paragraphs: We introduced the first fully convolutional pipeline for speech recognition, that can directly process the raw waveform and shows state-of-the art performance on Wall Street Journal and on Librispeech among end-to-end systems. This first attempt at exploiting convolutional language models in speech recognition shows significant improvement over a 4-gram language model on both datasets. Replacing mel-filterbanks by a learnable front-end gives additional gains in performance, that appear to be more prevalent on noisy data. This suggests learning the front-end is a promising avenue for speech recognition with challenging recording conditions.

Questions and Answers:
question: what is the state of the art on WSJ?
answers: answer: unanswerable: no
free_form_answer: CNN-DNN-BLSTM-HMM
evidence: Table TABREF11 shows Word Error Rates (WER) on WSJ for the current state-of-the-art and our models. The current best model trained on this dataset is an HMM-based system which uses a combination of convolutional, recurrent and fully connected layers, as well as speaker adaptation, and reaches INLINEFORM0 WER on nov92. DeepSpeech 2 shows a WER of INLINEFORM1 but uses 150 times more training data for the acoustic model and huge text datasets for LM training. Finally, the state-of-the-art among end-to-end systems trained only on WSJ, and hence the most comparable to our system, uses lattice-free MMI on augmented data (with speed perturbation) and gets INLINEFORM2 WER. Our baseline system, trained on mel-filterbanks, and decoded with a n-gram language model has a INLINEFORM3 WER. Replacing the n-gram LM by a convolutional one reduces the WER to INLINEFORM4 , and puts our model on par with the current best end-to-end system. Replacing the speech features by a learnable frontend finally reduces the WER to INLINEFORM5 and then to INLINEFORM6 when doubling the number of learnable filters, improving over DeepSpeech 2 and matching the performance of the best HMM-DNN system.
FLOAT SELECTED: Table 1: WER (%) on the open vocabulary task of WSJ.
answer: unanswerable: no
extractive_spans: HMM-based system
evidence: Table TABREF11 shows Word Error Rates (WER) on WSJ for the current state-of-the-art and our models. The current best model trained on this dataset is an HMM-based system which uses a combination of convolutional, recurrent and fully connected layers, as well as speaker adaptation, and reaches INLINEFORM0 WER on nov92. DeepSpeech 2 shows a WER of INLINEFORM1 but uses 150 times more training data for the acoustic model and huge text datasets for LM training. Finally, the state-of-the-art among end-to-end systems trained only on WSJ, and hence the most comparable to our system, uses lattice-free MMI on augmented data (with speed perturbation) and gets INLINEFORM2 WER. Our baseline system, trained on mel-filterbanks, and decoded with a n-gram language model has a INLINEFORM3 WER. Replacing the n-gram LM by a convolutional one reduces the WER to INLINEFORM4 , and puts our model on par with the current best end-to-end system. Replacing the speech features by a learnable frontend finally reduces the WER to INLINEFORM5 and then to INLINEFORM6 when doubling the number of learnable filters, improving over DeepSpeech 2 and matching the performance of the best HMM-DNN system.

Figure and Table Captions:
caption: Figure 1: Overview of the fully convolutional architecture.
caption: Table 1: WER (%) on the open vocabulary task of WSJ.
caption: Table 2: WER (%) on Librispeech.
caption: Figure 2: Evolution of WER (%) on Librispeech with the perplexity of the language model.
caption: Figure 3: Center frequency of the front-end filters, for the melfilterbank baseline and the learnable front-ends.
caption: Figure 4: Power heatmap of the 40 mel-filters (left) and of the frequency response of the 40 convolutional filters learned from the raw waveform on Librispeech (right).
caption: Table 3: Evolution of WER (%) on Librispeech with the context size of the language model.