Introduction
End-to-end (E2E) acoustic models, particularly with the attention-based encoder-decoder framework BIBREF0, have achieved a competitive recognition accuracy in a wide range of speech datasets BIBREF1. This model directly learns the mapping from the input acoustic signals to the output transcriptions without decomposing the problems into several different modules such as lexicon modeling, acoustic modeling and language modeling as in the conventional hybrid architecture. While this kind of E2E approach significantly simplifies the speech recognition pipeline, the weakness is that it is difficult to tune the strength of each component. One particular problem from our observations is that the attention based E2E model tends to make grammatical errors, which indicates that the language modeling power of the model is weak, possibly due to the small amount of training data, or the mismatch between the training and evaluation data. However, due to the jointly model approach in the attention model, it is unclear how to improve the strength of the language modeling power, i.e., attributing more weights to the previous output tokens in the decoder, or to improve the strength of the acoustic modeling power, i.e., attributing more weights to the context vector from the encoder.
While an external language model may be used to mitigate the weakness of the language modeling power of an attention-based E2E model, by either re-scoring the hypothesis or through shallow or deep fusion BIBREF2, the improvements are usually limited, and it incurs additional computational cost. Inspired by SpecAgument BIBREF3 and BERT BIBREF4, we propose a semantic mask approach to improve the strength of the language modeling power in the attention-based E2E model, which, at the same time, improves the generalization capacity of the model as well. Like SpecAugment, this approach masks out partial of the acoustic features during model training. However, instead of using a random mask as in SpecAugment, our approach masks out the whole patch of the features corresponding to an output token during training, e.g., a word or a word-piece. The motivation is to encourage the model to fill in the missing token (or correct the semantic error) based on the contextual information with less acoustic evidence, and consequently, the model may have a stronger language modeling power and is more robust to acoustic distortions.
In principle, our approach is applicable to the attention-based E2E framework with any type of neural network encoder. To constrain our research scope, we focus on the transformer architecture BIBREF5, which is originally proposed for neural machine translation. Recently, it has been shown that the transformer model can achieve competitive or even higher recognition accuracy compared with the recurrent neural network (RNN) based E2E model for speech recognition BIBREF6. Compared with RNNs, the transformer model can capture the long-term correlations with a computational complexity of $O(1)$, instead of using many steps of back-propagation through time (BPTT) as in RNNs. We evaluate our transformer model with semantic masking on Librispeech and TedLium datasets. We show that semantic masking can achieve significant word error rate reduction (WER) on top of SpecAugment, and we report the lowest WERs on the test sets of the Librispeech corpus with an E2E model.
Related Work
As aforementioned, our approach is closely related to SpecAugment BIBREF3, which applies a random mask to the acoustic features to regularize an E2E model. However, our masking approach is more structured in the sense that we mask the acoustic signals corresponding to a particular output token. Besides the benefit in terms of model regularization, our approach also encourages the model to reconstruct the missing token based on the contextual information, which improves the power of the implicit language model in the decoder. The masking approach operates as the output token level is also similar to the approach used in BERT BIBREF4, but with the key difference that our approaches works in the acoustic space.
In terms of the model structure, the transformer-based E2E model has been investigated for both attention-based framework as well as RNN-T based models BIBREF7. Our model structure generally follows BIBREF8, with a minor difference that we used a deeper CNN before the self-attention blocks. We used a joint CTC/Attention loss to train our model following BIBREF6.
Semantic Masking ::: Masking Strategy
Our masking approach requires the alignment information in order to perform the token-wise masking as shown in Figure FIGREF2. There are multiple speech recognition toolkits available to generate such kind of alignments. In this work, we used the Montreal Forced Alignertrained with the training data to perform forced-alignment between the acoustic signals and the transcriptions to obtain the word-level timing information. During model training, we randomly select a percentage of the tokens and mask the corresponding speech segments in each iteration. Following BIBREF4, in our work, we randomly sample 15% of the tokens and set the masked piece to the mean value of the whole utterance.
It should be noted that the semantic masking strategy is easy to combine with the previous SpecAugment masking strategy. Therefore, we adopt a time warp, frequency masking and time masking strategy in our masking strategy.
Semantic Masking ::: Why Semantic Mask Works?
Spectrum augmentation BIBREF3 is similar to our method, since both propose to mask spectrum for E2E model training. However, the intuitions behind these two methods are different. SpecAugment randomly masks spectrum in order to add noise to the source input, making the E2E ASR problem harder and prevents the over-fitting problem in a large E2E model.
In contrast, our model aims to force the decoder to learn a better language model. Suppose that if a few words' speech features are masked, the E2E model has to predict the token based on other signals, such as tokens that have generated or other unmasked speech features. In this way, we might alleviate the over-fitting issue that generating words only considering its corresponding speech features while ignoring other useful features. We believe our model is more effective when the input is noisy, because a model may generate correct tokens without considering previous generated tokens in a noise-free setting but it has to consider other signals when inputs are noisy, which is confirmed in our experiment.
Model
Following BIBREF8, we add convolution layers before Transformer blocks and discard the widely used positional encoding component. According to our preliminary experiments, the convolution layers slightly improve the performance of the E2E model. In the following, we will describe the CNN layers and Transformer block respectively.
Model ::: CNN Layer
We represent input signals as a sequence of log-Mel filter bank features, denoted as $\mathbf {X}=(x_0 \ldots , x_n)$, where $x_i$ is a 83-dim vector. Since the length of spectrum is much longer than text, we use VGG-like convolution block BIBREF9 with layer normalization and max-pooling function. The specific architecture is shown in Figure FIGREF6 . We hope the convolution block is able to learn local relationships within a small context and relative positional information. According to our experiments, the specific architecture outperforms Convolutional 2D subsampling method BIBREF6. We also use 1D-CNN in the decoder to extract local features replacing the position embedding .
Model ::: Transformer Block
Our Transformer architecture is implemented as BIBREF6, depicting in Figure FIGREF15. The transformer module consumes the outputs of CNN and extracts features with a self-attention mechanism. Suppose that $Q$, $K$ and $V$ are inputs of a transformer block, its outputs are calculated by the following equation
where $d_k$ is the dimension of the feature vector. To enable dealing with multiple attentions, multi-head attention is proposed, which is formulated as
where $d_{head}$ is the number of attention heads. Moreover, residual connection BIBREF10, feed-forward layer and layer normalization BIBREF11 are indispensable parts in Transformer, and their combinations are shown in Figure FIGREF15.
Model ::: ASR Training and Decoding
Following previous work BIBREF6, we employ a multi-task learning strategy to train the E2E model. Formally speaking, both the E2E model decoder and the CTC module predict the frame-wise distribution of $Y$ given corresponding source $X$, denoted as $P_{s2s}(\mathbf {Y}|\mathbf {X})$ and $P_{ctc}(\mathbf {Y}|\mathbf {X})$. We weighted averaged two negative log likelihoods to train our model
where $\alpha $ is set to 0.7 in our experiment.
We combine scores of E2E model $P_{s2s}$, CTC score $P_{ctc}$ and a RNN based language model $P_{rnn}$ in the decoding process, which is formulated as
where $\beta _1$ and $\beta _2$ are tuned on the development set. Following BIBREF12, we rescore our beam outputs based on another transformer based language model $P_{trans\_lm}(\mathbf {Y})$ and the sentence length penalty $\text{Wordcount}(\mathbf {Y})$.
where $P_{trans\_lm}$ denotes the sentence generative probability given by a Transformer language model.
EXPERIMENT
In this section, we describe our experiments on LibriSpeech BIBREF1 and TedLium2 BIBREF13. We compare our results with state-of-the-art hybrid and E2E systems. We implemented our approach based on ESPnet BIBREF6, and the specific settings on two datasets are the same with BIBREF6, except the decoding setting. We use the beam size 20, $\beta _1 = 0.5$, and $\beta _2=0.7$ in our experiment.
EXPERIMENT ::: Librispeech 960h
We represent input signals as a sequence of 80-dim log-Mel filter bank with 3-dim pitch features BIBREF17. SentencePiece is employed as the tokenizer, and the vocabulary size is 5000. The hyper-parameters in Transformer and SpecAugment follow BIBREF6 for a fair comparison. We use Adam algorithm to update the model, and the warmup step is 25000. The learning rate decreases proportionally to the inverse square root of the step number after the 25000-th step. We train our model 100 epochs on 4 P40 GPUs, which approximately costs 5 days to coverage. We also apply speed perturbation by changing the audio speed to 0.9, 1.0 and 1.1. Following BIBREF6, we average the last 5 checkpoints as the final model. Unlike BIBREF14 and BIBREF15, we use the same checkpoint for test-clean and test-other dataset.
The RNN language model uses the released LSTM language model provided by ESPnet. The Transformer language model for rescoring is trained on LibriSpeech language model corpus with the GPT-2 base setting (308M parameters). We use the code of NVIDIA Megatron-LM to train the Transformer language model.
We evaluate our model in different settings. The baseline Transformer represents the model with position embedding. The comparison between baseline Transformer and our architecture (Model with SpecAugment) indicates the improvements attributed to the architecture. Model with semantic mask is we use the semantic mask strategy on top of SpecAugment, which outperforms Model with SpecAugment with a large margin in a no external language model fusion setting, demonstrating that our masking strategy helps the E2E model to learn a better language model. The gap becomes smaller when equipped with a language model fusion component, which further confirms our motivation in Section SECREF1. Speed Perturbation does not help model performance on the clean dataset, but it is effective on the test-other dataset. Rescore is beneficial to both test-clean and test-other datasets.
As far as we know, our model is the best E2E ASR system on the Librispeech testset, which achieves a comparable result with wav2letter Transformer on test-clean dataset and a better result on test-other dataset, even though our model (75M parameters) is much smaller than the wav2letter Transformer (210M parameters). The reason might be that our semantic masking is more suitable on a noisy setting, because the input features are not reliable and the model has to predict the next token relying on previous ones and the whole context of the input. Our model is built upon the code base of ESPnet, and achieves relative $10\%$ gains due to the better architecture and masking strategy. Comparing with hybrid methods, our model obtains a similar performance on the test-clean set, but is still worse than the best hybrid model on the test-other dataset.
We also analyze the performance of different masking strategies, showing in Table TABREF20, where all models are shallow fused with the RNN language model. The SpecAugment provides 30$\%$ relative gains on test-clean and other datasets. According to the comparison between the second line and the third line, we find that the word masking is more effective on test-other dataset. The last row indicates word mask is complementary to random mask on the time axis.
EXPERIMENT ::: TedLium2
To verify the generalization of the semantic mask, we further conduct experiments on TedLium2 BIBREF18 dataset, which is extracted from TED talks. The corpus consists of 207 hours of speech data accompanying 90k transcripts. For a fair comparison, we use the same data-preprocessing method, Transformer architecture and hyperparameter settings as in BIBREF6. Our acoustic features are 80-dim log-Mel filter bank and 3-dim pitch features, which is normalized by the mean and the standard deviation for training set. The utterances with more than 3000 frames or more than 400 characters are discarded. The vocabulary size is set to 1000.
The experiment results are listed in Table TABREF21, showing a similar trend as the results in Librispeech dataset. Semantic mask is complementary to specagumentation, which enables better S2S language modeling training in an E2E model, resulting in a relative 4.5$\%$ gain. The experiment proves the effectiveness of semantic mask on a different and smaller dataset.
Conclusion
This paper presents a semantic mask method for E2E speech recognition, which is able to train a model to better consider the whole audio context for the disambiguation. Moreover, we elaborate a new architecture for E2E model, achieving state-of-the-art performance on the Librispeech test set in the scope of E2E models.

Question: How do they define their tokens (words, word-piece)?
Answer: Unanswerable

a word or a word-piece

Question: By how much do they outperform existing state-of-the-art model on end-to-end Speech recognition?s 
Answer: relative 4.5$\%$ gain, built upon the code base of ESPnet, and achieves relative $10\%$ gains due to the better architecture and masking strategy

10%