Abstract: Conventional automatic speech recognition (ASR) typically performs multi-level pattern recognition tasks that map the acoustic speech waveform into a hierarchy of speech units. But, it is widely known that information loss in the earlier stage can propagate through the later stages. After the resurgence of deep learning, interest has emerged in the possibility of developing a purely end-to-end ASR system from the raw waveform to the transcription without any predefined alignments and hand-engineered models. However, the successful attempts in end-to-end architecture still used spectral-based features, while the successful attempts in using raw waveform were still based on the hybrid deep neural network - Hidden Markov model (DNN-HMM) framework. In this paper, we construct the first end-to-end attention-based encoder-decoder model to process directly from raw speech waveform to the text transcription. We called the model as"Attention-based Wav2Text". To assist the training process of the end-to-end model, we propose to utilize a feature transfer learning. Experimental results also reveal that the proposed Attention-based Wav2Text model directly with raw waveform could achieve a better result in comparison with the attentional encoder-decoder model trained on standard front-end filterbank features.

Introduction: Conventional large-vocabulary continuous speech recognition (LVCSR) systems typically perform multi-level pattern recognition tasks that map the acoustic speech waveform into a hierarchy of speech units such as sub-words (phonemes), words, and strings of words (sentences). Such systems basically consist of several sub-components (feature extractor, acoustic model, pronunciation lexicon, language model) that are trained and tuned separately BIBREF0 . First, the speech signal is processed into a set of observation features based on a carefully hand-crafted feature extractor, such as Mel frequency cepstral coefficients (MFCC) or Mel-scale spectrogram. Then the acoustic model classifies the observation features into sub-unit or phoneme classes. Finally, the search algorithm finds the most probable word sequence based on the evidence of the acoustic model, the lexicon, and the language model. But, it is widely known that information loss in the earlier stage can propagate through the later stages. Deep learning algorithms have produced many state-of-the-art performances in various tasks that have revitalized the use of neural networks for ASR. One of the important factors behind the popularity of deep learning is the possibility of simplifying many complicated hand-engineered models by letting DNNs find their way to map from input to output spaces. Interest has emerged recently in the possibility of learning DNN-based acoustic models directly from the raw speech waveform without any predefined alignments and hand-engineered models. In this way, the feature extractor and acoustic model can be integrated into a single architecture. Palaz et al. BIBREF1 , BIBREF2 proposed a convolutional neural network (CNN) to directly train an acoustic model from the raw speech waveform. Sainath et al. BIBREF3 used time-convolutional layers over raw speech and trained them jointly with the long short-term memory deep neural network (CLDNN) acoustic model. The results showed that raw waveform CLDNNs matched the performance of log-mel CLDNNs on a voice search task. Ghahremani et al. BIBREF4 recently proposed a CNN time-delay neural network (CNN-TDNN) with network-in-network (NIN) architecture, and also showed that their model outperformed MFCC-based TDNN on the Wall Street Journal (WSJ) BIBREF5 task. But despite significant progress that has been made, the successful models were mostly demonstrated only within the hybrid DNN-HMM speech recognition frameworks. On the other hand, some existing works constructed end-to-end neural network models for ASR and replaced the acoustic model, the lexicon model, and the language model with a single integrated model, thus simplifying the pipeline. Graves et al. BIBREF6 , BIBREF7 successfully built an end-to-end ASR based on the connectionist temporal classification (CTC) framework. Amodei et al. BIBREF8 also constructed an end-to-end CTC-based ASR that directly produced character strings instead of phoneme sequences. But the CTC-based architecture still predicts the target outputs for every frame without any implicit knowledge about the language model. Another approach uses a sequence-to-sequence attention-based encoder-decoder that explicitly uses the history of previous outputs. Chorowski et al. BIBREF9 and Chan et al. BIBREF10 has successfully demonstrated encoder-decoder based ASR frameworks. Unfortunately, most of these works still used the standard spectral features (i.e., Mel-scale spectrogram, MFCC) as the input. The only attempt on end-to-end speech recognition for a raw waveform was recently proposed by BIBREF11 . Their system used a deep CNN and was trained with the automatic segmentation criterion (ASG) as an alternative to CTC. However, similar with CTC, the model did not explicitly use the history of the previous outputs assuming they were conditionally independent of each other. Furthermore, its performance was only reported using a very large data set (about 1000h of audio files). To the best of our knowledge, few studies have explored a single end-to-end ASR architecture trained on raw speech waveforms to directly output text transcription, and none of those models were built based on an encoder-decoder architecture. In this paper, we take a step forward to construct an end-to-end ASR using an attentional-based encoder-decoder model for processing raw speech waveform, naming it as “Attention-based Wav2Text". We investigate the performance of our proposed models on standard ASR datasets. In practice, optimizing an encoder-decoder framework is more difficult than a standard neural network architecture BIBREF10 . Therefore, we propose a feature transfer learning method to assist the training process for our end-to-end attention-based ASR model.

Question: Which architecture do they use for the encoder and decoder?
Answer: In encoder they use convolutional, NIN and bidirectional LSTM layers and in decoder they use unidirectional LSTM