Attention-based Wav2Text with Feature Transfer Learning

Conventional automatic speech recognition (ASR) typically performs multi-level pattern recognition tasks that map the acoustic speech waveform into a hierarchy of speech units. But, it is widely known that information loss in the earlier stage can propagate through the later stages. After the resurgence of deep learning, interest has emerged in the possibility of developing a purely end-to-end ASR system from the raw waveform to the transcription without any predefined alignments and hand-engineered models. However, the successful attempts in end-to-end architecture still used spectral-based features, while the successful attempts in using raw waveform were still based on the hybrid deep neural network - Hidden Markov model (DNN-HMM) framework. In this paper, we construct the first end-to-end attention-based encoder-decoder model to process directly from raw speech waveform to the text transcription. We called the model as"Attention-based Wav2Text". To assist the training process of the end-to-end model, we propose to utilize a feature transfer learning. Experimental results also reveal that the proposed Attention-based Wav2Text model directly with raw waveform could achieve a better result in comparison with the attentional encoder-decoder model trained on standard front-end filterbank features.

section_name: Introduction
paragraphs: Conventional large-vocabulary continuous speech recognition (LVCSR) systems typically perform multi-level pattern recognition tasks that map the acoustic speech waveform into a hierarchy of speech units such as sub-words (phonemes), words, and strings of words (sentences). Such systems basically consist of several sub-components (feature extractor, acoustic model, pronunciation lexicon, language model) that are trained and tuned separately BIBREF0 . First, the speech signal is processed into a set of observation features based on a carefully hand-crafted feature extractor, such as Mel frequency cepstral coefficients (MFCC) or Mel-scale spectrogram. Then the acoustic model classifies the observation features into sub-unit or phoneme classes. Finally, the search algorithm finds the most probable word sequence based on the evidence of the acoustic model, the lexicon, and the language model. But, it is widely known that information loss in the earlier stage can propagate through the later stages.
Deep learning algorithms have produced many state-of-the-art performances in various tasks that have revitalized the use of neural networks for ASR. One of the important factors behind the popularity of deep learning is the possibility of simplifying many complicated hand-engineered models by letting DNNs find their way to map from input to output spaces. Interest has emerged recently in the possibility of learning DNN-based acoustic models directly from the raw speech waveform without any predefined alignments and hand-engineered models. In this way, the feature extractor and acoustic model can be integrated into a single architecture. Palaz et al. BIBREF1 , BIBREF2 proposed a convolutional neural network (CNN) to directly train an acoustic model from the raw speech waveform. Sainath et al. BIBREF3 used time-convolutional layers over raw speech and trained them jointly with the long short-term memory deep neural network (CLDNN) acoustic model. The results showed that raw waveform CLDNNs matched the performance of log-mel CLDNNs on a voice search task. Ghahremani et al. BIBREF4 recently proposed a CNN time-delay neural network (CNN-TDNN) with network-in-network (NIN) architecture, and also showed that their model outperformed MFCC-based TDNN on the Wall Street Journal (WSJ) BIBREF5 task. But despite significant progress that has been made, the successful models were mostly demonstrated only within the hybrid DNN-HMM speech recognition frameworks.
On the other hand, some existing works constructed end-to-end neural network models for ASR and replaced the acoustic model, the lexicon model, and the language model with a single integrated model, thus simplifying the pipeline. Graves et al. BIBREF6 , BIBREF7 successfully built an end-to-end ASR based on the connectionist temporal classification (CTC) framework. Amodei et al. BIBREF8 also constructed an end-to-end CTC-based ASR that directly produced character strings instead of phoneme sequences. But the CTC-based architecture still predicts the target outputs for every frame without any implicit knowledge about the language model. Another approach uses a sequence-to-sequence attention-based encoder-decoder that explicitly uses the history of previous outputs. Chorowski et al. BIBREF9 and Chan et al. BIBREF10 has successfully demonstrated encoder-decoder based ASR frameworks. Unfortunately, most of these works still used the standard spectral features (i.e., Mel-scale spectrogram, MFCC) as the input. The only attempt on end-to-end speech recognition for a raw waveform was recently proposed by BIBREF11 . Their system used a deep CNN and was trained with the automatic segmentation criterion (ASG) as an alternative to CTC. However, similar with CTC, the model did not explicitly use the history of the previous outputs assuming they were conditionally independent of each other. Furthermore, its performance was only reported using a very large data set (about 1000h of audio files).
To the best of our knowledge, few studies have explored a single end-to-end ASR architecture trained on raw speech waveforms to directly output text transcription, and none of those models were built based on an encoder-decoder architecture. In this paper, we take a step forward to construct an end-to-end ASR using an attentional-based encoder-decoder model for processing raw speech waveform, naming it as “Attention-based Wav2Text". We investigate the performance of our proposed models on standard ASR datasets. In practice, optimizing an encoder-decoder framework is more difficult than a standard neural network architecture BIBREF10 . Therefore, we propose a feature transfer learning method to assist the training process for our end-to-end attention-based ASR model.
section_name: Attention-based Encoder Decoder for Raw Speech Recognition
paragraphs: The encoder-decoder model is a neural network that directly models conditional probability INLINEFORM0 where INLINEFORM1 is the source sequence with length INLINEFORM2 and INLINEFORM3 is the target sequence with length INLINEFORM4 . It consists of encoder, decoder and attention modules. The encoder task processes an input sequence INLINEFORM5 and outputs representative information INLINEFORM6 for the decoder. The attention module is an extension scheme that assists the decoder to find relevant information on the encoder side based on the current decoder hidden states BIBREF12 , BIBREF13 . Usually, the attention module produces context information INLINEFORM7 at time INLINEFORM8 based on the encoder and decoder hidden states: DISPLAYFORM0 
 There are several variations for score function : DISPLAYFORM0 
 where INLINEFORM0 , INLINEFORM1 is the number of hidden units for the encoder and INLINEFORM2 is the number of hidden units for the decoder. Finally, the decoder task, which predicts the target sequence probability at time INLINEFORM3 based on previous output and context information INLINEFORM4 , can be formulated as: DISPLAYFORM0 
The most common input INLINEFORM0 for speech recognition tasks is a sequence of feature vectors such as log Mel-spectral spectrogram and/or MFCC. Therefore, INLINEFORM1 where D is the number of the features and S is the total length of the utterance in frames. The output INLINEFORM2 can be either phoneme or grapheme (character) sequence.
In this work, we use the raw waveform as the input representation instead of spectral-based features and a grapheme (character) sequence as the output representation. In contrast to most encoder-decoder architectures, which are purely based on recurrent neural network (RNNs) framework, we construct an encoder with several convolutional layers BIBREF14 followed by NIN layers BIBREF15 as the lower part in the encoder and integrate them with deep bidirectional long short-term memory (Bi-LSTM) BIBREF16 at the higher part. We use convolutional layers because they are suitable for extracting local information from raw speech. We use a striding mechanism to reduce the dimension from the input frames BIBREF17 , while the NIN layer represents more complex structures on the top of the convolutional layers. On the decoder side, we use a standard deep unidirectional LSTM with global attention BIBREF13 that is calculated by a multi-layer perceptron (MLP) as described in Eq. EQREF2 . For more details, we illustrate our architecture in Figure FIGREF4 .
section_name: Feature Transfer Learning
paragraphs: Deep learning is well known for its ability to learn directly from low-level feature representation such as raw speech BIBREF1 , BIBREF3 . However, in most cases such models are already conditioned on a fixed input size and a single target output (i.e., predicting one phoneme class for each input frame). In the attention-based encoder-decoder model, the training process is not as easy as in a standard neural network model BIBREF10 because the attention-based model needs to jointly optimize three different modules simultaneously: (1) an encoder module for producing representative information from a source sequence; (2) an attention module for calculating the correct alignment; and (3) a decoder module for generating correct transcriptions. If one of these modules has difficulty fulfilling its own tasks, then the model will fail to produce good results.
To ease the burden on training the whole encoder-decoder architecture directly to predict the text transcription given the raw speech waveform, we utilize a transfer learning method on the encoder part. Specifically, we only train the encoder's lower layers consisting of the convolutional and NIN layers to predict the spectral features given the corresponding raw waveform. In this work, we utilize two widely used spectral features: MFCC and log Mel-scale spectrogram as the transfer learning target. Figure FIGREF5 shows our feature transfer learning architecture. First, given segmented raw speech waveform INLINEFORM0 , we extract corresponding INLINEFORM1 -dimensional spectral features INLINEFORM2 . Then we process raw speech INLINEFORM3 with several convolutions, followed by NIN layers in the encoder part. In the last NIN-layer, we set a fixed number of channels as INLINEFORM4 channels and apply mean-pooling across time. Finally, we get predictions for corresponding spectral features INLINEFORM5 and optimize all of the parameters by minimizing the mean squared error between predicted spectral features INLINEFORM6 and target spectral features INLINEFORM7 : DISPLAYFORM0 
In this paper, we also explore multi target feature transfer using a similar structure as in Figure FIGREF5 but with two parallel NIN layers, followed by mean-polling at the end. One of the output layers is used to predicts log Mel-scale spectrogram and another predicts MFCC features. We modify the single target loss function from Eq. EQREF6 into the following: DISPLAYFORM0 
 where INLINEFORM0 are the predicted Mel-scale spectrogram and the MFCC values, and INLINEFORM1 are the real Mel-scale spectrogram and MFCC features for frame INLINEFORM2 . After optimizing all the convolutional and NIN layer parameters, we transfer the trained layers and parameters and integrate them with the Bi-LSTM encoder. Finally, we jointly optimize the whole structure together.
section_name: Speech Data
paragraphs: In this study, we investigate the performance of our proposed models on WSJ BIBREF5 . We used the same definitions of the training, development and test set as the Kaldi s5 recipe BIBREF18 . The raw speech waveforms were segmented into multiple frames with a 25ms window size and a 10ms step size. We normalized the raw speech waveform into the range -1 to 1. For spectral based features such as MFCC and log Mel-spectrogram, we normalized the features for each dimension into zero mean and unit variance. For WSJ, we separated into two experiments by using WSJ-SI84 only and WSJ-SI284 data. We used dev_93 for our validation set and eval_92 for our test set. We used the character sequence as our decoder target and followed the preprocessing step proposed by BIBREF19 . The text from all the utterances was mapped into a 32-character set: 26 (a-z) alphabet, apostrophe, period, dash, space, noise, and “eos".
section_name: Model Architectures
paragraphs: Our attention-based Wav2Text architecture uses four convolutional layers, followed by two NIN layers at the lower part of the encoder module. For all the convolutional layers, we used a leaky rectifier unit (LReLU) BIBREF20 activation function with leakiness INLINEFORM0 . Inside the first NIN layers, we stacked three consecutive filters with LReLU activation function. For the second NIN layers, we stacked two consecutive filters with tanh and identity activation function. For the feature transfer learning training phase, we used Momentum SGD with a learning rate of 0.01 and momentum of 0.9. Table TABREF11 summarizes the details of the layer settings for the convolutional and NIN layers.
On the top layers of the encoder after the transferred convolutional and NIN layers, we put three bidirectional LSTMs (Bi-LSTM) with 256 hidden units (total 512 units for both directions). To reduce the computational time, we used hierarchical subsampling BIBREF21 , BIBREF22 , BIBREF10 . We applied subsampling on all the Bi-LSTM layers and reduced the length by a factor of 8.
On the decoder side, the previous input phonemes / characters were converted into real vectors by a 128-dimensional embedding matrix. We used one unidirectional LSTM with 512 hidden units and followed by a softmax layer to output the character probability. For the end-to-end training phase, we froze the parameter values from the transferred layers from epoch 0 to epoch 10, and after epoch 10 we jointly optimized all the parameters together until the end of training (a total 40 epochs). We used an Adam BIBREF23 optimizer with a learning rate of 0.0005.
In the decoding phase, we used a beam search strategy with beam size INLINEFORM0 and we adjusted the score by dividing with the transcription length to prevent the decoder from favoring shorter transcriptions. We did not use any language model or lexicon dictionary for decoding. All of our models were implemented on the PyTorch framework .
For comparison, we also evaluated the standard attention-based encoder decoder with Mel-scale spectrogram input as the baseline. Here, we used similar settings as the proposed model, except we replaced the convolutional and NIN layers with a feedforward layer (512 hidden units).
section_name: Result
paragraphs: An example of our transfer learning results is shown in Figure FIGREF8 , and Table TABREF14 shows the speech recognition performance in CER for both the WSJ-SI84 and WSJ-SI284 datasets. We compared our method with several published models like CTC, Attention Encoder-Decoder and Joint CTC-Attention model that utilize CTC for training the encoder part. Besides, we also train our own baseline Attention Encoder-Decoder with Mel-scale spectrogram. The difference between our Attention Encoder-Decoder (“Att Enc-Dec (ours)", “Att Enc-Dec Wav2Text") with Attention Encoder-Decoder from BIBREF24 (“Att Enc-Dec Content", “Att Enc-Dec Location") is we used the current hidden states to generate the attention vector instead of the previous hidden states. Another addition is we utilized “input feedback" method BIBREF13 by concatenating the previous context vector into the current input along with the character embedding vector. By using those modifications, we are able to improve the baseline performance.
Our proposed Wav2Text models without any transfer learning failed to converge. In contrast, with transfer learning, they significantly surpassed the performance of the CTC and encoder-decoder from Mel-scale spectrogram features. This suggests that by using transfer learning for initializing the lower part of the encoder parameters, our model also performed better then their original features.
section_name: Related Work
paragraphs: Transfer learning is the ability of a learning algorithm to convey knowledge across different tasks. The initial idea is to reuse previously obtained knowledge to enhance the learning for new things. The standard procedure are : first, train the model on a base dataset and task, then the learned features and/or parameters are reused for learning a second target dataset and task. Bengio et al. BIBREF25 provided deep reviews about multi-task and transfer learning on deep learning models. Jason et al. BIBREF26 showed that a model with transferred parameter consistently outperformed a randomly initialized one.
In speech recognition research, transfer learning has been studied for many years, including successful cases of speaker adaptation and cross-lingual acoustic modeling BIBREF27 . One popular scheme for utilizing DNNs for transfer learning within ASR frameworks is a tandem approach BIBREF28 . This idea first trains a DNN with a narrow hidden bottleneck layer to perform phoneme classification at the frame level and then reuses the activations from the narrow hidden bottleneck layer as discriminative features in conventional GMM-HMM or hybrid DNN-HMM models BIBREF29 . Another study introduced a convolutional bottleneck network as an alternative tandem bottleneck feature architecture BIBREF30 . However, although such a feature transfer learning framework provides many advantages in ASR, the usage in an end-to-end attention-based ASR framework has not been explored.
This study performs feature transfer learning on the encoder part of the end-to-end attention-based ASR architecture. We train the convolutional encoder to predict the spectral features given the corresponding raw speech waveform. After that, we transfer the trained layers and parameters, integrate them with the LSTM encoder-decoder, and eventually optimize the whole structure to predict the correct output text transcription given the raw speech waveform.
section_name: Conclusion
paragraphs: This paper described the first attempt to build an end-to-end attention-based encoder-decoder speech recognition that directly predicts the text transcription given raw speech input. We also proposed feature transfer learning to assist the encoder-decoder model training process and presented a novel architecture that combined convolutional, NIN and Bi-LSTM layers into a single encoder part for raw speech recognition. Our results suggest that transfer learning is a very helpful method for constructing an end-to-end system from such low-level features as raw speech signals. With transferred parameters, our proposed attention-based Wav2Text models converged and matched the performance with the attention-based encoder-decoder model trained on standard spectral-based features. The best performance was achieved by Wav2Text models with transfer learning from multi target scheme.
section_name: Acknowledgements
paragraphs: Part of this work was supported by JSPS KAKENHI Grant Numbers JP17H06101 and JP 17K00237. 

Questions and Answers:
question: Which architecture do they use for the encoder and decoder?
answers: answer: unanswerable: no
extractive_spans: we construct an encoder with several convolutional layers BIBREF14 followed by NIN layers BIBREF15 as the lower part in the encoder and integrate them with deep bidirectional long short-term memory (Bi-LSTM) BIBREF16 at the higher part
On the decoder side, we use a standard deep unidirectional LSTM with global attention BIBREF13 that is calculated by a multi-layer perceptron (MLP)
evidence: In this work, we use the raw waveform as the input representation instead of spectral-based features and a grapheme (character) sequence as the output representation. In contrast to most encoder-decoder architectures, which are purely based on recurrent neural network (RNNs) framework, we construct an encoder with several convolutional layers BIBREF14 followed by NIN layers BIBREF15 as the lower part in the encoder and integrate them with deep bidirectional long short-term memory (Bi-LSTM) BIBREF16 at the higher part. We use convolutional layers because they are suitable for extracting local information from raw speech. We use a striding mechanism to reduce the dimension from the input frames BIBREF17 , while the NIN layer represents more complex structures on the top of the convolutional layers. On the decoder side, we use a standard deep unidirectional LSTM with global attention BIBREF13 that is calculated by a multi-layer perceptron (MLP) as described in Eq. EQREF2 . For more details, we illustrate our architecture in Figure FIGREF4 .
answer: unanswerable: no
free_form_answer: In encoder they use convolutional, NIN and bidirectional LSTM layers and in decoder they use unidirectional LSTM 
evidence: On the top layers of the encoder after the transferred convolutional and NIN layers, we put three bidirectional LSTMs (Bi-LSTM) with 256 hidden units (total 512 units for both directions). To reduce the computational time, we used hierarchical subsampling BIBREF21 , BIBREF22 , BIBREF10 . We applied subsampling on all the Bi-LSTM layers and reduced the length by a factor of 8.
On the decoder side, the previous input phonemes / characters were converted into real vectors by a 128-dimensional embedding matrix. We used one unidirectional LSTM with 512 hidden units and followed by a softmax layer to output the character probability. For the end-to-end training phase, we froze the parameter values from the transferred layers from epoch 0 to epoch 10, and after epoch 10 we jointly optimized all the parameters together until the end of training (a total 40 epochs). We used an Adam BIBREF23 optimizer with a learning rate of 0.0005.
question: How does their decoder generate text?
answers: answer: unanswerable: no
extractive_spans: decoder task, which predicts the target sequence probability at time INLINEFORM3 based on previous output and context information
evidence: where INLINEFORM0 , INLINEFORM1 is the number of hidden units for the encoder and INLINEFORM2 is the number of hidden units for the decoder. Finally, the decoder task, which predicts the target sequence probability at time INLINEFORM3 based on previous output and context information INLINEFORM4 , can be formulated as: DISPLAYFORM0
answer: unanswerable: no
free_form_answer: Decoder predicts the sequence of phoneme or grapheme at each time based on the previous output and context information with a beam search strategy
evidence: where INLINEFORM0 , INLINEFORM1 is the number of hidden units for the encoder and INLINEFORM2 is the number of hidden units for the decoder. Finally, the decoder task, which predicts the target sequence probability at time INLINEFORM3 based on previous output and context information INLINEFORM4 , can be formulated as: DISPLAYFORM0
The most common input INLINEFORM0 for speech recognition tasks is a sequence of feature vectors such as log Mel-spectral spectrogram and/or MFCC. Therefore, INLINEFORM1 where D is the number of the features and S is the total length of the utterance in frames. The output INLINEFORM2 can be either phoneme or grapheme (character) sequence.
In the decoding phase, we used a beam search strategy with beam size INLINEFORM0 and we adjusted the score by dividing with the transcription length to prevent the decoder from favoring shorter transcriptions. We did not use any language model or lexicon dictionary for decoding. All of our models were implemented on the PyTorch framework .
question: Which dataset do they use?
answers: answer: unanswerable: no
extractive_spans: WSJ
evidence: In this study, we investigate the performance of our proposed models on WSJ BIBREF5 . We used the same definitions of the training, development and test set as the Kaldi s5 recipe BIBREF18 . The raw speech waveforms were segmented into multiple frames with a 25ms window size and a 10ms step size. We normalized the raw speech waveform into the range -1 to 1. For spectral based features such as MFCC and log Mel-spectrogram, we normalized the features for each dimension into zero mean and unit variance. For WSJ, we separated into two experiments by using WSJ-SI84 only and WSJ-SI284 data. We used dev_93 for our validation set and eval_92 for our test set. We used the character sequence as our decoder target and followed the preprocessing step proposed by BIBREF19 . The text from all the utterances was mapped into a 32-character set: 26 (a-z) alphabet, apostrophe, period, dash, space, noise, and “eos".
answer: unanswerable: no
extractive_spans: WSJ-SI84
WSJ-SI284
evidence: An example of our transfer learning results is shown in Figure FIGREF8 , and Table TABREF14 shows the speech recognition performance in CER for both the WSJ-SI84 and WSJ-SI284 datasets. We compared our method with several published models like CTC, Attention Encoder-Decoder and Joint CTC-Attention model that utilize CTC for training the encoder part. Besides, we also train our own baseline Attention Encoder-Decoder with Mel-scale spectrogram. The difference between our Attention Encoder-Decoder (“Att Enc-Dec (ours)", “Att Enc-Dec Wav2Text") with Attention Encoder-Decoder from BIBREF24 (“Att Enc-Dec Content", “Att Enc-Dec Location") is we used the current hidden states to generate the attention vector instead of the previous hidden states. Another addition is we utilized “input feedback" method BIBREF13 by concatenating the previous context vector into the current input along with the character embedding vector. By using those modifications, we are able to improve the baseline performance.

Figure and Table Captions:
caption: Fig. 1. Attention-based Wav2Text architecture.
caption: Fig. 2. Feature transfer learning: train lower layers of the encoder (convolutional and NIN layers) to predict spectral features given corresponding raw waveform; then transfer the trained layers and parameters (marked by orange square) into attention-based encoder decoder model (see Figure 1).
caption: Fig. 3. Example of our transfer learning model output: top is the original Mel-spectrogram, and bottom is the predicted Melspectrogram.
caption: Table 1. Layer setting details for convolutional and NIN layers. Sorted from the input layer to the output layer.
caption: Table 2. Character error rate (CER) result from baseline and proposed models on WSJ0 and WSJ1 dataset. All of these results are produced without using language model or lexicon dictionary. Word error rate (WER) for Att Wav2Text + transfer multi-target is 17.04%, compared to Joint CTC+Att (MTL)[25] 18.2% and standard Enc-Dec Att [23] 18.6%.