Single headed attention based sequence-to-sequence model for state-of-the-art results on Switchboard-300	It is generally believed that direct sequence-to-sequence (seq2seq) speech recognition models are competitive with hybrid models only when a large amount of data, at least a thousand hours, is available for training. In this paper, we show that state-of-the-art recognition performance can be achieved on the Switchboard-300 database using a single headed attention, LSTM based model. Using a cross-utterance language model, our single-pass speaker independent system reaches 6.4% and 12.5% word error rate (WER) on the Switchboard and CallHome subsets of Hub5'00, without a pronunciation lexicon. While careful regularization and data augmentation are crucial in achieving this level of performance, experiments on Switchboard-2000 show that nothing is more useful than more data.

Comparison with the literature	For comparison with results in the literature we refer to the Switchboard-300 results in , , ,  and the Switchboard-2000 results in , , , , , . Our 300-hour model not only outperforms the previous best attention based encoder-decoder model  by a large margin, it also surpasses the best hybrid systems with multiple LMs . Our result on Switchboard-2000 is also better than any single system results reported to date, and reaches the performance of the best system combinations.

Experimental results ::: Experiments on Switchboard-2000	As a contrast to our best results on Switchboard-300, we also train a seq2seq model on the 2000-hour Switchboard+Fisher data. This model consists of 10 encoder layers, and is trained for only 50 epochs. Our overall results on the Hub5'00 and other evaluation sets are summarized in Table TABREF14. The results in Fig. FIGREF12 and Table TABREF14 show that adding more training data greatly improves the system, by around 30% relative in some cases. For comparison with others, the 2000-hour system reaches 8.7% and 7.4% WER on rt02 and rt04. We observe that the regularization techniques, which are extremely important on the 300h setup, are still beneficial but have a significantly smaller effect.

We show that consistent application of various regularization techniques brings a simple, single-head LSTM attention based encoder-decoder model to state-of-the-art performance on Switchboard-300, a task where data sparsity is more severe than LibriSpeech. We also note that remarkable performance has been achieved with single-head LSTM models in a recent study on language modeling .

Experimental setup	This study focuses on Switchboard-300, a standard 300-hour English conversational speech recognition task. Our acoustic and text data preparation follows the Kaldi  s5c recipe. Our attention based seq2seq model is similar to ,  and follows the structure of . We extract 80-dimensional log-Mel filterbank features over 25ms frames every 10ms from the input speech signal. The input audio is speed and/or tempo perturbed with 56 probability. Following , sequence noise mixed from up to 4 utterances is injected with 40% probability and 0.3 weight. The filterbank output is mean-and-variance normalized at the speaker level, and first ($\Delta $) and second ($\Delta \Delta $) derivatives are also calculated. The final features presented to the network are also processed through a SpecAugment block that uses the SM policy  with $p=0.3$ and no time warping. The encoder network comprises 8 bidirectional LSTM layers with 1536 nodes per direction per layer , . As shown in Fig.

When greedy search (beam 1) is used, the external language model increases WER, an effect that might be mitigated with re-optimized hyperparameters. Nevertheless, if a beam of at least 2 hypotheses is used, the positive effect of the language model is clear. We also observe that without the language model the search saturates much earlier, around beam 8, fluctuating within only a few absolute errors afterwards. On the contrary, decoding with the language model, we measure consistent but small gains with larger beams. The minimum number of word errors was measured with a relatively large beam of 240. The figure also shows that the effect of a cross-utterance language model grows with larger beams. Lastly, if the model is trained on 2000 hours of speech data (see next section), the extremely fast greedy decoding gives remarkably good performance. Although the importance of beam search decreases with an increased amount of training data, we still measure 10% relative degradation compared to a system with a cross-utterance LM and wide (240) beam search.

Conclusions	We presented an attention based encoder-decoder setup which achieves state-of-the-art performance on Switchboard-300. A rather simple model built from LSTM layers and a decoder with a single-headed attention mechanism outperforms the standard hybrid approach. This is particularly remarkable given that in our model neither a pronunciation lexicon nor a speech model with explicit hidden state representations is needed. We also demonstrated that excellent results are possible with smaller models and with practically search-free, greedy decoding. The best results were achieved with a speaker independent model in a single decoding pass, using a minimalistic search algorithm, and without any attention mechanism in the language model. Thus, we believe that further improvements are still possible if we apply a more complicated sequence-level training criterion and speaker adaptation.

Experimental results ::: Optimizing the language model	The following experiments summarize our optimization of the LM. Compared to our previous LM , we measure better perplexity and WER if no bottleneck is used before the softmax layer (rows 1 and 3 in Table TABREF9). Increasing the model capacity to 122M parameters results in a significant gain in PPL only after the dropout rates are tuned (rows 3, 5 and 6). Similar to , , significant PPL gain is observed if the LM was trained across utterances. However, this PPL improvement does not translate into reduced WER with a bigger model when cross utterance modeling is used (rows 4 and 7). Thus, in all other experiments we use the smaller, 57M-parameter model.

Zoneout, which is also inspired by dropout and was especially developed for recurrent models , stochastically forces some hidden units to maintain their previous values. In LSTMs, the method is applied on the cell state or on the recurrent feedback of the output. Label smoothing interpolates the hard label targets with a uniform distribution over targets, and improves generalization in many classification tasks . Batch normalization (BN) accelerates training by standardizing the distribution of each layer's input . In order to reduce the normalization mismatch between training and testing, we modify the original approach by freezing the batch normalization layers in the middle of the training when the magnitude of parameter updates is small. After freezing, the running statistics are not updated, batch statistics are ignored, and BN layers approximately operate as global normalization. Scheduled sampling stochastically uses the token produced by a sequence model instead of the true previous token during training to mitigate the effects of exposure bias . Residual networks address the problem of vanishing and exploding gradients by including skip connections  in the model that force the neural network to learn a residual mapping function using a stack of layers. Optimization of this residual mapping is easier, allowing the use of much deeper structures.

Experimental results ::: Effect of beam size and number of parameters	A 280M-parameter model may be larger than is practical in many applications. Thus, we also conduct experiments to see if this model size is necessary for reasonable ASR performance. Models are trained without changing the training configuration, except that the size or number of LSTM layers is reduced. As Table TABREF11 shows, although our smallest attention based model achieves reasonable results on this task, a significant loss is indeed observed with decreasing model size, especially on CallHome. Nevertheless, an external language model reduces the performance gap. A small, 57M-parameter model together with a similar size language model is only 5% relative worse than our largest model. We note that this model already outperforms the best published attention based seq2seq model , with roughly 66% fewer parameters. Additional experiments are carried out to characterize the search and modeling errors in decoding. The results of tuning the beam size and keeping the other search hyperparameters unchanged are shown in Fig. FIGREF12. “Small” denotes the 57M model, while “large” denotes the 280M model.

The weight decay parameter is 4e-6, the label smoothing parameter is 0.35, and teacher forcing is fixed to 0.8 throughout training. In the first 3 epochs the learning rate is warmed up and batch size is gradually increased from 8 to 32 . In the first 35 epochs, the neural network is trained on sequences sorted in ascending order of length of the input. Afterwards, batches are randomized within length buckets, ensuring that a batch always contains sequences with similar length. Weight noise from a normal distribution with mean 0.0 and variance 0.015 is switched on after 70 epochs. After 110 epochs, the updates of sufficient statistics in the batch-normalization layers are turned off, converting them into fixed affine transformations. The learning rate is annealed by 0.9 per epoch after 180 epochs of training, and simultaneously label smoothing is also switched off. The external language model (LM) is built on the BPE segmentation of 24M words from the Switchboard and Fisher corpora. It is trained for 40 epochs using label smoothing of 0.15 in the first 20 epochs.

Experimental results ::: Effect of data preparation	We first investigate the importance of different data processing steps. The s5c Kaldi recipe includes a duplicate filtering step, in which the maximum number of occurrences of utterances with the same content is limited. We measure the impact of duplicate filtering and also the effect of filtering out word fragments and noise tokens from the training transcripts. Since the LM is trained on identically filtered transcripts from Fisher+Switchboard data, word fragment and noise token filters were applied consistently. The results are summarized in Table TABREF5. Deactivating the duplicate filter is never harmful when an external LM is used, and the gains on CallHome can be substantial. Considering performance on the complete Hub5'00 data, the best systems either explicitly handle both word fragments and noise tokens or filter them all out. When an external LM is used, the best results are obtained when word fragment and noise token filters are activated and the duplicate filter is deactivated. This setting is also appealing in cases where the external LM may be trained on text data that will not contain word fragments or noise; thus, the remaining experiments are carried out with this system setting.

The output of both LSTMs is reduced to 256 dimensions by a linear bottleneck . Fixed sequence-level weight dropout of 15% is applied in the decoder LSTMs, a dropout of 5% is applied to the embeddings, and a dropout of 15% is applied to the decoder LSTM outputs. The second LSTM in the decoder also uses zoneout, where the cell state update is deactivated with 15% probability and the recurrent feedback from the output maintains its previous value with 5% probability. Overall, the model has 280M parameters, of which only 5.4M are in the decoder. Aiming at the best word error rate, this design choice is based on our observation that an external language model has significantly larger effect if the decoder is not over-parametrized . The model is trained for 250 epochs on 32 P100 GPUs in less than 4 days using a PyTorch  implementation of distributed synchronous SGD with up to 32 sequences per GPU per batch. Training uses a learning rate of 0.03 and Nesterov momentum  of 0.9.

Curriculum learning simplifies deep neural network training by presenting training examples in a meaningful order, usually by increasing order of difficulty . In seq2seq models, the input acoustic sequences are frequently sorted in order of increasing length . Speed and tempo perturbation changes the rate of speech, typically by $\pm $10%, with or without altering the pitch and timbre of the speech signal , . The goal of these methods is to increase the amount of training data for the model. Sequence noise injection adds structured sequence level noise generated from speech utterances to training examples to improve the generalization of seq2seq models . As previously shown, input noise during neural network training encourages convergence to a local optimum with lower curvature, which indicates better generalization . Weight noise adds noise directly to the network parameters to improve generalization . This form of noise can be interpreted as a simplified form of Bayesian inference that optimizes a minimum description length loss . SpecAugment masks blocks of frequency channels and blocks of time steps  and also warps the spectrogram along the time axis to perform data augmentation. It is closely related to .

Methods to improve seq2seq models	In contrast to traditional hybrid models, where even recurrent networks are trained on randomized, aligned chunks of labels and features , , whole sequence models are more prone to memorizing the training samples. In order to improve generalization, many of the methods we investigate introduce additional noise, either directly or indirectly, to stochastic gradient descent (SGD) training to avoid narrow, local optima. The other techniques we study address the highly non-convex nature of training neural networks, ease the optimization process, and speed up convergence. Weight decay adds the $l_2$ norm of the trainable parameters to the loss function, which encourages the weights to stay small unless necessary, and is one of the oldest techniques to improve neural network generalization. As shown in , weight decay can improve generalization by suppressing some of the effects of static noise on the targets. Dropout randomly deactivates neurons with a predefined probability in every training step  to reduce co-adaptation of neurons. DropConnect, which is similar in spirit to dropout, randomly deactivates connections between neurons by temporarily zeroing out weights .

The baseline LM has 57M parameters and consists of 2 unidirectional LSTM layers with 2048 nodes  trained with drop-connect and dropout probabilities of 15%. The embedding layer has 512 nodes, and the output of the last LSTM is projected to 128 dimensions. When the LM is trained and evaluated across utterances, consecutive segments of a single-channel recording are grouped together up to 40 seconds. Perplexities (PPL) are measured at the word level on the concatenation of ground truth transcripts, while the WER is obtained by retaining the LM state of the single-best hypothesis of the preceding utterance. Decoding uses simple beam search with a beam width of 60 hypotheses and no lexical prefix tree constraint . The search performs shallow fusion of the encoder-decoder score, the external language model score, a length normalization term, and a coverage term , , . For more details, please refer to . Hub5'00 is used as a development set to optimize decoding hyperparameters, while Hub5'01 and RT03 are used as final test sets.

FIGREF1, each LSTM block in the encoder includes a residual connection with a linear transformation that bypasses the LSTM, a 1024-dimensional linear reduction layer on the LSTM output, and batch-normalization (BN) of the block output. A pyramidal structure  in the first two LSTM layers reduces the frame rate by a factor of 4. The final dimension of the encoder output is 256, enforced by a linear bottleneck. We apply 30% dropout to the LSTM outputs and 30% drop-connect to the hidden-to-hidden matrices , . As suggested by , the weight dropout is fixed for a batch of sequences. The attention based decoder model is illustrated in Fig. FIGREF1. The decoder models the sequence of 600 BPE units estimated on characters , where the BPE units are embedded in 256 dimensions. We use additive, location aware attention, without key/value transformations, and the attention is smoothed by 256, 5-dimensional kernels . The decoder block consists of 2 unidirectional LSTM layers: one is a dedicated language-model-like component with 512 nodes that operates only on the embedded predicted symbol sequence, and the other is a 768 unit layer processing acoustic and symbol information.

Experimental results	Our current setup is the result of incremental development. Keeping in mind that several other equally powerful setups probably exist, the focus of the following experiments is to investigate ours around the current optimum.

Experimental results ::: Ablation study	In a second set of experiments, we characterize the importance of each of the regularization methods described in Sec. SECREF2 for our model performance by switching off one training method at a time without re-optimizing the remaining settings. In these experiments, decoding is performed without an external language model. Curriculum learning is evaluated by either switching to randomized batches after 35 epochs or leaving the sorting on throughout training. We also test the importance of $\Delta $ and $\Delta \Delta $ features . Sorting the results by decreasing number of absolute errors on Hub5'00, Table TABREF7 indicates that each regularization method contributes to the improved WER. SpecAugment is by far the most important method, while using $\Delta $ and $\Delta \Delta $ features or switching off the curriculum learning in the later stage of training have marginal but positive effects. Other direct input level perturbation steps (speed/tempo perturbation and sequence noise injection) are also key techniques that can be found in the upper half of the table. If we compare the worst and baseline models, we find that the relative performance difference between them is nearly unchanged by including the external LM in decoding.

Without the LM, the gap is 18% relative, while with the LM the gap is 17% relative. This clearly underlines the importance of the regularization techniques.

Introduction	Powerful neural networks have enabled the use of “end-to-end” speech recognition models that directly map a sequence of acoustic features to a sequence of words without conditional independence assumptions. Typical examples are attention based encoder-decoder  and recurrent neural network transducer models . Due to training on full sequences, an utterance corresponds to a single observation from the view point of these models; thus, data sparsity is a general challenge for such approaches, and it is believed that these models are effective only when sufficient training data is available. Indeed, many end-to-end speech recognition papers focus on LibriSpeech, which has 960 hours of training audio. Nevertheless, the best performing systems follow the traditional hybrid approach , outperforming attention based encoder-decoder models , , , , and when less training data is used, the gap between “end-to-end” and hybrid models is more prominent , . Several methods have been proposed to tackle data sparsity and overfitting problems; a detailed list can be found in Sec. SECREF2. Recently, increasingly complex attention mechanisms have been proposed to improve seq2seq model performance, including stacking self and regular attention layers and using multiple attention heads in the encoder and decoder , .

Question: How much bigger is Switchboard-2000 than Switchboard-300 database?
Answer: Switchboard-2000 contains 1700 more hours of speech data.