Title: WAV2LETTER: AN END-TO-END CONVNET-BASED SPEECH RECOGNITION SYSTEM

Abstract: This paper presents a simple end-to-end model for speech recognition, combining a convolutional network based acoustic model and a graph decoding. It is trained to output letters, with transcribed speech, without the need for force alignment of phonemes. We introduce an automatic segmentation criterion for training from sequence annotation without alignment that is on par with CTC (Graves et al., 2006) while being simpler. We show competitive results in word error rate on the Librispeech corpus (Panayotov et al., 2015) with MFCC features, and promising results from raw waveform.

1. INTRODUCTION:

We present an end-to-end system to speech recognition, going from the speech signal (e.g. Mel-Frequency Cepstral Coefficients (MFCC), power spectrum, or raw waveform) to the transcription. The acoustic model is trained using letters (graphemes) directly, which take out the need for an intermediate (human or automatic) phonetic transcription. Indeed, the classical pipeline to build state of the art systems for speech recognition consists in first training an HMM/GMM model to force align the units on which the final acoustic model operates (most often context-dependent phone states). This approach takes its roots in HMM/GMM training (Woodland & Young, 1993) . The improvements brought by deep neural networks (DNNs) (Mohamed et al., 2012; Hinton et al., 2012) and convolutional neural networks (CNNs) (Sercu et al., 2015; Soltau et al., 2014) for acoustic modeling only extend this training pipeline.

The current state of the art on Librispeech (the dataset that we used for our evaluations) uses this approach too (Panayotov et al., 2015; Peddinti et al., 2015b) , with an additional step of speaker adaptation (Saon et al., 2013; Peddinti et al., 2015a) . Recently, Senior et al. (2014) proposed GMMfree training, but the approach still requires to generate a force alignment. An approach that cut ties with the HMM/GMM pipeline (and with force alignment) was to train with a recurrent neural network (RNN) (Graves et al., 2013) for phoneme transcription. There are now competitive end-to-end approaches of acoustic models toppled with RNNs layers as in (Hannun et al., 2014; Miao et al., 2015; Saon et al., 2015; Amodei et al., 2015) , trained with a sequence criterion (Graves et al., 2006) . However these models are computationally expensive, and thus take a long time to train.

Compared to classical approaches that need phonetic annotation (often derived from a phonetic dictionary, rules, and generative training), we propose to train the model end-to-end, using graphemes directly. Compared to sequence criterion based approaches that train directly from speech signal to graphemes (Miao et al., 2015) , we propose a simple(r) architecture (23 millions of parameters for our best model, vs. 100 millions of parameters in (Amodei et al., 2015) ) based on convolutional networks for the acoustic model, toppled with a graph transformer network (Bottou et al., 1997) , trained with a simpler sequence criterion. Our word-error-rate on clean speech is slightly better than (Hannun et al., 2014) , and slightly worse than (Amodei et al., 2015) , in particular factoring that they train on 12,000 hours while we only train on the 960h available in LibriSpeech's train set. Finally, some of our models are also trained on the raw waveform, as in (Palaz et al., 2013; 2015; Sainath et al., 2015) . The rest of the paper is structured as follows: the next section presents the convolutional networks used for acoustic modeling, along with the automatic segmentation criterion. The following section shows experimental results comparing different features, the criterion, and our current best word error rates on LibriSpeech.

2. ARCHITECTURE:

Our speech recognition system is a standard convolutional neural network (LeCun & Bengio, 1995) fed with various different features, trained through an alternative to the Connectionist Temporal Classification (CTC) (Graves et al., 2006) , and coupled with a simple beam search decoder. In the following sub-sections, we detail each of these components. We consider three types of input features for our model: MFCCs, power-spectrum, and raw wave. MFCCs are carefully designed speech-specific features, often found in classical HMM/GMM speech systems (Woodland & Young, 1993) because of their dimensionality compression (13 coefficients are often enough to span speech frequencies). Power-spectrum features are found in most recent deep learning acoustic modeling features (Amodei et al., 2015) . Raw wave has been somewhat explored in few recent work (Palaz et al., 2013; 2015) . ConvNets have the advantage to be flexible enough to be used with either of these input feature types.

2.1. FEATURES:

Our acoustic models output letter scores (one score per letter, given a dictionary L).

2.2. CONVNET ACOUSTIC MODEL:

The acoustic models we considered in this paper are all based on standard 1D convolutional neural networks (ConvNets). ConvNets interleave convolution operations with pointwise non-linearity operations. Often ConvNets also embark pooling layers: these type of layers allow the network to "see" a larger context, without increasing the number of parameters, by locally aggregating the previous convolution operation output. Instead, our networks leverage striding convolutions. Given (x t ) t=1...Tx an input sequence with T x frames of d x dimensional vectors, a convolution with kernel width kw, stride dw and d y frame size output computes the following:

EQUATION

where b ∈ R dy and w ∈ R dy×dx×kw are the parameters of the convolution (to be learned).

Pointwise non-linear layers are added after convolutional layers.

In our experience, we surprisingly found that using hyperbolic tangents, their piecewise linear counterpart HardTanh (as in (Palaz et al., 2015) ) or ReLU units lead to similar results.

There are some slight variations between the architectures, depending on the input features. MFCC-based networks need less striding, as standard MFCC filters are applied with large strides on the input raw sequence. With power spectrum-based and raw wave-based networks, we observed that the overall stride of the network was more important than where the convolution with strides were placed. We found thus preferrable to set the strided convolutions near the first input layers of the network, as it leads to the fastest architectures: with power spectrum features or raw wave, the input sequences are very long and the first convolutions are thus the most expensive ones.

The last layer of our convolutional network outputs one score per letter in the letter dictionary (d y = |L|). Our architecture for raw wave is shown in Figure 1 and is inspired by (Palaz et al., 2015) . The architectures for both power spectrum and MFCC features do not include the first layer. The full network can be seen as a non-linear convolution, with a kernel width of size 31280 and stride equal to 320; given the sample rate of our data is 16KHz, label scores are produced using a window of 1955 ms, with steps of 20ms.

2.3. INFERRING SEGMENTATION WITH AUTOSEGCRITERION:

Most large labeled speech databases provide only a text transcription for each audio file. In a classification framework (and given our acoustic model produces letter predictions), one would need the segmentation of each letter in the transcription to train properly the model. Unfortunately, manually labeling the segmentation of each letter would be tedious. Several solutions have been explored in the speech community to alleviate this issue: HMM/GMM models use an iterative EM procedure: (i) during the Estimation step, the best segmentation is inferred, according to the current model, by maximizing the joint probability of the letter (or any sub-word unit) transcription and input sequence. (ii) During the Maximization step the model is optimized by minimizing a frame-level criterion, based on the (now fixed) inferred segmentation. This approach is also often used to boostrap the training of neural network-based acoustic models.

Other alternatives have been explored in the context of hybrid HMM/NN systems, such as the MMI criterion (Bahl et al., 1986 ) which maximizes the mutual information between the acoustic sequence and word sequences or the Minimum Bayse Risk (MBR) criterion (Gibson & Hain, 2006) .

More recently, standalone neural network architectures have been trained using criterions which jointly infer the segmentation of the transcription while increase the overall score of the right transcription (Graves et al., 2006; Palaz et al., 2014) . The most popular one is certainly the Connectionist Temporal Classification (CTC) criterion, which is at the core of Baidu's Deep Speech architecture (Amodei et al., 2015) . CTC assumes that the network output probability scores, normalized at the frame level. It considers all possible sequence of letters (or any sub-word units), which can lead to a to a given transcription. CTC also allow a special "blank" state to be optionally inserted between each letters. The rational behind the blank state is two-folds: (i) modeling "garbage" frames which might occur between each letter and (ii) identifying the separation between two identical consecutive letters in a transcription. Figure 2a shows an example of the sequences accepted by CTC for a given transcription. In practice, this graph is unfolded as shown in Figure 2b , over the available frames output by the acoustic model. We denote G ctc (θ, T ) an unfolded graph over T frames for a given transcription θ, and π = π 1 , . . . , π T ∈ G ctc (θ, T ) a path in this graph representing a (valid) sequence of letters for this transcription. At each time step t, each node of the graph is assigned with the corresponding log-probability letter (that we denote f t (•)) output by the acoustic model. CTC aims at maximizing the "overall" score of paths in G ctc (θ, T ); for that purpose, it minimizes the Forward score:

EQUATION

where the "logadd" operation, also often called "log-sum-exp" is defined as logadd(a, b) = exp(log(a) + log(b)). This overall score can be efficiently computed with the Forward algorithm. To put things in perspective, if one would replace the logadd(•) by a max(•) in (2) (which can be then efficiently computed by the Viterbi algorithm, the counterpart of the Forward algorithm), one would then maximize the score of the best path, according to the model belief. The logadd(•) can be seen as a smooth version of the max(•): paths with similar scores will be attributed the same weight in the overall score (and hence receive the same gradient), and paths with much larger score will have much more overall weight than paths with low scores. In practice, using the logadd(•) works much better than the max(•). It is also worth noting that maximizing (2) does not diverge, as the acoustic model is assumed to output normalized scores (log-probabilities)

f i (•). ∅ C ∅ A ∅ T ∅ (a) ∅ ∅ ∅ C C C C ∅ ∅ ∅ A A A A ∅ ∅ ∅ T T T T ∅ ∅ ∅ (b)

Figure 2 : The CTC criterion graph. (a) Graph which represents all the acceptable sequences of letters (with the blank state denoted "∅"), for the transcription "cat". (b) Shows the same graph unfolded over 5 frames. There are no transitions scores. At each time step, nodes are assigned a conditional probability output by the neural network acoustic model.

In this paper, we explore an alternative to CTC, with three differences: (i) there are no blank labels, (ii) un-normalized scores on the nodes (and possibly un-normalized transition scores on the edges) (iii) global normalization instead of per-frame normalization:

• The advantage of (i) is that it produces a much simpler graph (see Figure 3a and Figure 3b ). We found that in practice there was no advantage of having a blank class to model the possible "garbage" frames between letters. Modeling letter repetitions (which is also an important quality of the blank label in CTC) can be easily replaced by repetition character labels (we used two extra labels for two and three repetitions). For example "caterpillar" could be written as "caterpil2ar", where "2" is a label to represent the repetition of the previous letter. Not having blank labels also simplifies the decoder. • With (ii) one can easily plug an external language model, which would insert transition scores on the edges of the graph. This could be particularly useful in future work, if one wanted to model representations more high-level than letters. In that respect, avoiding normalized transitions is important to alleviate the problem of "label bias" Bottou (1991) ; Lafferty et al. (2001) . In this work, we limited ourselves to transition scalars, which are learned together with the acoustic model. • The normalization evoked in (iii) is necessary when using un-normalized scores on nodes or edges; it insures incorrect transcriptions will have a low confidence.

In the following, we name our criterion "Auto Segmentation Criterion" (ASG). Considering the same notations than for CTC in (2), and an unfolded graph G asg (θ, T ) over T frames for a given transcription θ (as in Figure 3b ), as well as a fully connected graph G f ull (θ, T ) over T frames (representing all possible sequence of letters, as in Figure 3c ), ASG aims at minimizing:

ASG(θ, T ) = -logadd π∈Gasg(θ,T ) T t=1 (f πt (x) + g πt-1,πt (x)) + logadd π∈G f ull (θ,T ) T t=1 (f πt (x) + g πt-1,πt (x)) ,

(3) where g i,j (•) is a transition score model to jump from label i to label j. The left-hand part of 3 promotes sequences of letters leading to the right transcription, and the right-hand part demotes all sequences of letters. As for CTC, these two parts can be efficiently computed with the Forward algorithm. Derivatives with respect to f i (•) and g i,j (•) can be obtained (maths are a bit tedious) by applying the chain rule through the Forward recursion.

2.4. BEAM-SEARCH DECODER:

We wrote our own one-pass decoder, which performs a simple beam-search with beam threholding, histogram pruning and language model smearing Steinbiss et al. (1994) . We kept the decoder as simple as possible (under 1000 lines of C code). We did not implement any sort of model adaptation before decoding, nor any word graph rescoring. Our decoder relies on KenLM Heafield et al. (2013) for the language modeling part. It also accepts un-normalized acoustic scores (transitions and emissions from the acoustic model) as input. The decoder attempts to maximize the following:

EQUATION

EQUATION

where P lm (θ) is the probability of the language model given a transcription θ, α and β are two hyper-parameters which control the weight of the language model and the word insertion penalty respectively.

3. EXPERIMENTS:



3.1. SETUP:

We implemented everything using Torch71 . The ASG criterion as well as the decoder were implemented in C (and then interfaced into Torch).

We consider as benchmark LibriSpeech, a large speech database freely available for download (Panayotov et al., 2015) . LibriSpeech comes with its own train, validation and test sets. Except when specified, we used all the available data (about 1000h of audio files) for training and validating our models. We use the original 16 KHz sampling rate. The vocabulary L contains 30 graphemes: the standard English alphabet plus the apostrophe, silence, and two special "repetition" graphemes which encode the duplication (once or twice) of the previous letter (see Section 2.3).

The architecture hyper-parameters, as well the decoder ones were tuned using the validation set. In the following, we either report letter-error-rates (LERs) or word-error-rates (WERs). WERs have been obtained by using our own decoder (see Section 2.4), with the standard 4-gram language model provided with LibriSpeech2 . MFCC features are computed with 13 coefficients, a 25 ms sliding window and 10 ms stride. We included first and second order derivatives. Power spectrum features are computed with a 25 ms window, 10 ms stride, and have 257 components. All features are normalized (mean 0, std 1) per input sequence.

3.2. RESULTS:

Table 1 reports a comparison between CTC and ASG, in terms of LER and speed. Our ASG criterion is implemented in C (CPU only), leveraging SSE instructions when possible. Our batching is done with an OpenMP parallel for. We picked the CTC criterion implementation provided by Baidu3 . Both criteria lead to the same LER. For comparing the speed, we report performance for sequence sizes as reported initially by Baidu, but also for longer sequence sizes, which corresponds to our average use case. ASG appears faster on long sequences, even though it is running on CPU only. Baidu's GPU CTC implementation seems more aimed at larger vocabularies (e.g. 5000 Chinese characters).

We also investigated the impact of the training size on the dataset, as well as the effect of a simple data augmentation procedure, where shifts were introduced in the input frames, as well as stretching.

For that purpose, we tuned the size of our architectures (given a particular size of the dataset), to avoid over-fitting. Figure 4a shows the augmentation helps for small training set size. However, with enough training data, the effect of data augmentation vanishes, and both type of features appear to perform similarly. Figure 4b reports the WER with respect to the available training data size. We observe that we compare very well against Deep Speech 1 & 2 which were trained with much more data Hannun et al. (2014) ; Amodei et al. (2015) .

Finally, we report in Table 2 the best results of our system so far, trained on 1000h of speech, for each type of features. The overall stride of architectures is 320 (see Figure 1 ), which produces a label every 20 ms. We found that one could squeeze out about 1% in performance by refining the precision of the output. This is efficiently achieved by shifting the input sequence, and feeding it to the network several times. Results in Table 2 were obtained by a single extra shift of 10 ms. Both power spectrum and raw features are performing slightly worse than MFCCs. One could expect, however, that with enough data (see Figure 4 ) the gap would vanish. 

4. CONCLUSION:

We have introduced a simple end-to-end automatic speech recognition system, which combines a standard 1D convolutional neural network, a sequence criterion which can infer the segmentation, and a simple beam-search decoder. The decoding results are competitive on the LibriSpeech corpus with MFCC features (7.2% WER), and promising with power spectrum and raw speech (9.4% WER and 10.1% WER respectively). We showed that our AutoSegCriterion can be faster than CTC (Graves et al., 2006) , and as accurate (table 1). Our approach breaks free from HMM/GMM pre-training and force-alignment, as well as not being as computationally intensive as RNN-based approaches (Amodei et al., 2015) (on average, one LibriSpeech sentence is processed in less than 60ms by our ConvNet, and the decoder runs at 8.6x on a single thread).

:

http://www.torch.ch.

http://www.openslr.org/11.

https://github.com/baidu-research/warp-ctc.

Figures and Tables:

Figure 1: Our neural network architecture for raw wave. First two layers are convolutions with strides. Last two layers are convolutions with kw = 1, which are equivalent to fully connected layers. Power spectrum and MFCC based networks do not have the first layer.

Figure 3: The ASG criterion graph. (a) Graph which represents all the acceptable sequences of letters for the transcription "cat". (b) Shows the same graph unfolded over 5 frames. (c) Shows the corresponding fully connected graph, which describe all possible sequences of letter; this graph is used for normalization purposes. Un-normalized transitions scores are possible on the edges. At each time step, nodes are assigned a conditional un-normalized score, output by the neural network acoustic model.

Figure 4: Valid LER (a) and WER (b) v.s. training set size (10h, 100h, 200h, 1000h). This compares MFCC-based and power spectrum-based (POW) architectures. AUG experiments include data augmentation. In (b) we provide Baidu Deep Speech 1 and 2 numbers on LibriSpeech, as a comparison Hannun et al. (2014); Amodei et al. (2015).

CTC vs ASG. CTC is Baidu's implementation. ASG is implemented on CPU (C with OpenMP). Timings (in ms) for small sequences (input frames: 150, letter vocabulary size: 28, transcription size: 40) and long sequences (input frames: 700, letter vocabulary size: 28, transcription size: 200) are reported in (a) and (b) respectively. (c) reports performance in LER. Timings include both forward and backward passes. CPU implementations use 8 threads.

Table content: <table><tr><td/><td>(a)</td><td/><td/><td/><td>(b)</td><td/></tr><tr><td>batch</td><td>CTC</td><td/><td>ASG</td><td>batch</td><td colspan="2">CTC</td><td>ASG</td></tr><tr><td>size</td><td colspan="3">CPU GPU CPU</td><td>size</td><td colspan="3">CPU GPU CPU</td></tr><tr><td>1</td><td>1.9</td><td>5.9</td><td>2.5</td><td>1</td><td>40.9</td><td colspan="2">97.9 16.0</td></tr><tr><td>4</td><td>2.0</td><td>6.0</td><td>2.8</td><td>4</td><td>41.6</td><td colspan="2">99.6 17.7</td></tr><tr><td>8</td><td>2.0</td><td>6.1</td><td>2.8</td><td>8</td><td colspan="3">41.7 100.3 19.2</td></tr><tr><td/><td/><td/><td>(c)</td><td/><td/><td/></tr><tr><td/><td/><td/><td colspan="2">ASG CTC</td><td/><td/></tr><tr><td/><td/><td/><td colspan="2">dev-clean 10.4 10.7</td><td/><td/></tr><tr><td/><td/><td/><td colspan="2">test-clean 10.1 10.5</td><td/><td/></tr></table>

LER/WER of the best sets of hyper-parameters for each feature types.

Table content: <table><tr><td/><td colspan="2">MFCC</td><td>PS</td><td/><td colspan="2">Raw</td></tr><tr><td/><td colspan="6">LER WER LER WER LER WER</td></tr><tr><td>dev-clean</td><td>6.9</td><td/><td>9.3</td><td/><td>10.3</td><td/></tr><tr><td>test-clean</td><td>6.9</td><td>7.2</td><td>9.1</td><td>9.4</td><td>10.6</td><td>10.1</td></tr></table>

Peer review (iclr2017):
This submission proposes a letter-level decoder with a variation of the CTC approach they call ASG, where the blank symbol is dropped and replaced by letter repetition symbols, and where explicit normalization is dropped. Both the description of a letter-level model (though not novel), as well as the CTC-variant are interesting. The approach is evaluated on the LibriSpeech task. The authors claim that their approach is competitive. They compare their modelling variant ASG to CTC, but a comparison of the letter-level approach to available word-level results are missing. Compared to the results obtained in Panayotov et al. 2015, the performance obtained here seems only comparable to word-level GMM/HMM models, but worse than word-level hybrid DNN/HMM models, though Panayotov et al. also appled speaker adaptation, which was not done, as far as I can see. I suggest to add a comparison to Panyotov*s results (in addition to mentioning Baidu*s results on Librispeech, which are not comparable due to much larger amounts of training data), to allow readers to get a quantitative idea. As pointed out by the authors in the text, Baidu*s GPU implementation for CTC is more aimed at larger vocabularies, therefore the comparison to GPU in Tables 1a-c do not seem to be helpful for this work, without further discussing the implementations. You are using quite a huge analysis window (nearly 2s). Even though other authors also use windows up to 0.5s to 1s (e.g. MRASTA features), some comments on how you arrive at such a large window, and what advantages you observe for it, would be interesting. The submission is well written, though more details on the experiences with using non-normalized (transition) scores and beam pruning would be desirable. Table 1 would be better readable if the units of the numbers shown in a/b/c would be shown within the tables, and not only in the caption. Prior (partial) publications of this work (your NIPS end-to-end workshop paper) should clearly be mentioned/referenced. What do you mean by transition *scalars*? I do not repeat further comments here, which were already given in the pre-review period. Minor comments: - Sec. 2.3, end of 2nd sentence: train properly the model -> train the model properly End of same paragraph: boostrap -> bootstrap (such errors should be avoided by performing an automatic spell check) - Sec. 2.3: Bayse -> Bayes - definition of logadd is wrong (see comment) - (applies also for your NIPS end-to-end workshop paper). - line before Eq. (3): all possible sequence of letters -> all possible sequences of letters (plural) - Sec. 2.4, first line: threholding -> thresholding (spell check..) - Figure 4: mention the corpus used here - dev?