Dual-mode ASR: Unify and Improve Streaming ASR with Full-context Modeling

1 INTRODUCTION . “ Ok Google . Hey Siri . Hi Alexa. ” have featured a massive boom of smart speakers in recent years , unveiling a trend towards ubiquitous and ambient Artificial Intelligence ( AI ) for better daily lives . As the communication bridge between human and machine , low-latency streaming ASR ( a.k.a. , online ASR ) is of central importance , whose goal is to emit each hypothesized word as quickly and accurately as possible on the fly as they are spoken . On the other hand , there are some scenarios where full-context ASR ( a.k.a. , offline ASR ) is sufficient , for example , offline video captioning on video-sharing platforms . While low-latency streaming ASR is generally preferred in most of the speech recognition scenarios , it often has worse prediction accuracy as measured in Word Error Rate ( WER ) , due to the lack of future context compared with full-context ASR . Improving both WER and emission latency has been shown to be highly challenging ( He et al. , 2019 ; Li et al. , 2020a ; Sainath et al. , 2020 ) in streaming ASR systems . Since the acoustic , pronunciation , and language model ( AM , PM , and LM ) of a conventional ASR system have been evolved into a single end-to-end ( E2E ) all-neural network , modern streaming and full-context ASR models share most of the neural architectures and training recipes in common , such as , Mel-spectrogram inputs , data augmentations , neural network meta-architectures , training objectives , model regularization techniques and decoding methods . The most significant difference is that streaming ASR encoders are auto-regressive models , with the prediction of the current timestep conditioned on previous ones ( no future context is permitted ) . Specifically , let x and y be the input and output sequence , t as frame index , T as total length of frames . Streaming ASR encoders model the output yt as a function of input x1 : t while full-context ASR encoders model the output yt as a function of input x1 : T . Streaming ASR encoders can be built with uni-directional LSTMs , causal convolution and left-context attention layers in streaming ASR encoders ( Chiu & Raffel , 2018 ; Fan et al. , 2018 ; Han et al. , 2020 ; Gulati et al. , 2020 ; Huang et al. , 2020 ; Moritz et al. , 2020 ; Miao †equal contribution et al. , 2020 ; Tsunoo et al. , 2020 ; Zhang et al. , 2020 ; Yeh et al. , 2019 ) . Recurrent Neural Network Transducers ( RNN-T ) ( Graves , 2012 ) are commonly used as the decoder in both streaming and fullcontext models , which predicts the token of the current input frame based on all previous tokens using uni-directional recurrent layers . Figure 1 illustrates a simplified example of the similarity and difference between streaming and full-context ASR models with E2E neural networks . Albeit the similarities , streaming and full-context ASR models are usually developed , trained , and deployed separately . In this work , we propose Dual-mode ASR , a framework to unify streaming and full-context speech recognition networks with shared weights . Dual-mode ASR comes with many immediate benefits , including reduced model download and storage on devices and simplified development and deployment workflows . To accomplish this goal , we first introduce Dual-mode Encoders , which can run in both streaming mode and full-context mode . Dual-mode encoders are designed to reuse the same set of model weights for both modes with zero or near-zero parameters overhead . We propose the design principles of a dual-mode encoder and show examples on how to design dual-mode convolution , dual-mode pooling , and dual-mode attention layers . We also investigate into different training algorithms for Dual-mode ASR , specifically , randomly sampled training and joint training . We show that joint training significantly outperforms randomly sampled training in terms of model quality and training stability . Moreover , motivated by Inplace Knowledge Distillation ( Yu & Huang , 2019b ) in which a large model is used to supervise a small model , we propose to distill knowledge from the full-context mode ( teacher ) into the streaming mode ( student ) on the fly during the training within the same Dual-mode ASR model , by encouraging consistency of the predicted token probabilities . We demonstrate that the emission latency and prediction accuracy of streaming ASR significantly benefit from weight sharing and joint training of its full-context mode , especially with inplace knowledge distillation during the training . We present extensive experiments with two state-of-theart ASR networks , convolution-based ContextNet ( Han et al. , 2020 ) and conv-transformer hybrid Conformer ( Gulati et al. , 2020 ) , on two datasets , a widely used public dataset LibriSpeech ( Panay- otov et al. , 2015 ) ( 970 hours of English reading speech ) and a large-scale dataset MultiDomain ( Narayanan et al. , 2018 ) ( 413,000 hours speech of a mixture across multiple domains including Voice Search , Farfield Speech , YouTube and Meetings ) . For each proposed technique , we also present ablation study and analysis to demonstrate and understand the effectiveness . With Dual-mode ASR , we achieve new state-of-the-art streaming ASR results on both LibriSpeech and MultiDomain in terms of accuracy and latency . 2 RELATED WORK . Streaming ASR Networks . There has been a growing interest in building streaming ASR systems based on E2E Recurrent Neural Network Transducers ( RNN-T ) ( Graves , 2012 ) . Compared with sequence-to-sequence models ( Chorowski et al. , 2014 ; 2015 ; Chorowski & Jaitly , 2016 ; Bahdanau et al. , 2016 ; Chan et al. , 2016 ) , RNN-T models are naturally streamable and have shown great potentials for low-latency streaming ASR ( Chang et al. , 2019 ; He et al. , 2019 ; Tsunoo et al. , 2019 ; Sainath et al. , 2019 ; Shen et al. , 2019 ; Li et al. , 2020a ; b ; Sainath et al. , 2020 ; Huang et al. , 2020 ; Moritz et al. , 2020 ; Narayanan et al. , 2020 ) . In this work , we mainly focus on RNN-T based models . He et al . specifically studied how to optimize the RNN-T streaming ASR model for mobile devices , and proposed a bag of techniques including using layer normalization and large batch size to stabilize training ; using word-piece targets ( Wu et al. , 2016 ) ; using a time-reduction layer to speed up training and inference ; quantizing network parameters to reduce memory footprint and speed up computation ; applying shallow-fusion to bias towards user-specific context . To support streaming modeling in E2E ASR models , various efforts have also been made by modifying attention-based models such as monotonic attention ( Raffel et al. , 2017 ; Chiu & Raffel , 2017 ; Fan et al. , 2018 ; Arivazhagan et al. , 2019 ) , GMM attention ( Graves , 2013 ; Chiu et al. , 2019 ) , triggered attention ( TA ) ( Moritz et al. , 2019 ) , Scout Network ( Wang et al. , 2020 ) , and approaches that segment encoder output into non-overlapping chunks ( Jaitly et al. , 2016 ; Tsunoo et al. , 2020 ) . Tsunoo et al . also applied knowledge distillation from the non-streaming model to the streaming model , but their streaming and non-streaming models do not share weights and are trained separately . To improve the latency of RNN-T streaming models , Li et al . investigated additional early and late penalties on Endpointer prediction ( Chang et al. , 2019 ) to reduce the emission latency , and employed the minimum word error rate ( MWER ) training ( Prabhavalkar et al. , 2018 ) to remedy accuracy degradation . Sainath et al . further proposed to improve quality by using two-pass models ( Sainath et al. , 2019 ) , i.e. , a second-pass LAS-based rescore model on top of the hypotheses from first-pass RNN-T streaming output . More recently , Li et al . proposed parallel rescoring by replacing LSTMs with Transformers ( Vaswani et al. , 2017 ) in rescoring models . Chang et al . further proposed Prefetching to reduce system latency by submitting partial recognition results for subsequent processing such as obtaining assistant server responses or second-pass rescoring before the recognition result is finalized . Unlike these approaches , our work explores the unification of streaming and fullcontext ASR networks , thus can be generally applied as an add-on technique without requiring extra runtime support during inference . Weight Sharing for Multi-tasking . Sharing model weights of a deep neural network for multiple tasks has been widely explored in the literature to reduce overall model sizes . In the broadest sense , tasks can refer to different objectives or same objective but different settings , ranging from natural language processing and speech recognition to computer vision and reinforcement learning . In speech recognition , Kannan et al . employed a single ASR network for multilingual ASR , and showed accuracy improvements over monolingual ASR systems . Wu et al . proposed dynamic sparsity neural networks ( DSNN ) for speech recognition on mobile devices with resource constraints . A single trained DSNN ( Wu et al. , 2020 ) can transform into multiple networks of different sparsities for adaptive inference in real-time . Chang et al . trained a single RNN-T model with LSTMs ( Hochreiter & Schmidhuber , 1997 ) for Joint Endpointing ( i.e. , predicting both recognition tokens and the end of an utterance transcription ) in streaming ASR systems . Moreover , Watanabe et al . proposed a hybrid CTC and attention architecture for ASR based on multi-objective learning to eliminate the use of linguistic resources . Another related research work in Computer Vision is Slimmable Neural Networks ( Yu et al. , 2018 ; Yu & Huang , 2019a ; b ; Yu et al. , 2020 ) . Yu et al . proposed an approach to train a single neural network running at different widths , permitting instant and adaptive accuracy efficiency trade-offs at runtime . We also adapt the training rules introduced in slimmable networks , that is , using independent normalization layers for different sub-networks ( tasks ) as conditional parameters and using the prediction of teacher network to supervise student network as inplace distillation during the training . Unlike slimmable networks in which a large model is used to supervise a small model , we propose to distill the knowledge from full-context mode ( teacher ) into streaming mode ( student ) on the fly within the same Dual-mode ASR model . Knowledge Distillation . Hinton et al . explored a simple method to “ transfer ” knowledge from a teacher neural network to a student neural network by enforcing their predictions to be close measured by KL-divergence , ` 1 or ` 2 distance . It is shown that such distillation method is effective to compress neural networks ( Yu & Huang , 2019b ) , accelerate training ( Chen et al. , 2015 ) , improve robustness ( Carlini & Wagner , 2017 ; Papernot et al. , 2016 ) , estimate model uncertainty ( Blundell et al. , 2015 ) and transfer learned domain to other domains ( Tzeng et al. , 2015 ) . 3 DUAL-MODE ASR . Most neural sequence transduction networks for ASR have an encoder-decoder structure ( Graves , 2012 ; Sainath et al. , 2020 ; He et al. , 2019 ; Li et al. , 2020a ) , as shown in Figure 1 . Without loss of generality , here we discuss how to design Dual-mode ASR networks under the most commonly used RNN-T model ( Graves , 2012 ) . In RNN-T models , we first extract mel-spectrogram feature from input speech waveform . The Mel-spectrogram feature is then fed into a neural-net encoder , which usually consists of feed-forward layers , RNN/LSTM layers , convolution layers , attention layers , pooling ( time-reduction ) layers , and residual or dense connections . In neural-net encoders , streaming ASR model requires all components to be auto-regressive , whereas full-context ASR model has no such requirement . The ASR decoder then predicts the token of current frame based on the output from the encoder and previous predicted tokens ( inference ) or target tokens ( training with teacher forcing ( Williams & Zipser , 1989 ) ) . The decoder is commonly an auto-regressive model in both streaming and full-context ASR models , thus is fully shared in Dual-mode ASR . The prediction from decoder is finally used either in decoding algorithm during inference ( e.g. , beam search ) or learning algorithm during training ( e.g. , RNN-T loss ) . As discussed above and shown in Figure 1 , it becomes clear that the major difference between streaming and full-context ASR models is in the neural-net encoder . In the following , we will first discuss the design principles of dual-mode encoder to support both streaming and full-context ASR . We provide examples including dual-mode convolution , dual-mode average pooling , and dual-mode attention layers , which are widely used in the state-of-the-art ASR networks ContextNet ( Han et al. , 2020 ) and Conformer ( Gulati et al. , 2020 ) . We will then discuss the training algorithm of Dual-mode ASR networks including joint training and inplace knowledge distillation .

Summary: The paper proposes a pragmatic approach to unifying end-to-end speech-recognition models for whole-utterance and streaming models. Streaming models are defined as not using any future audio, while whole-utterance (or "full context") models can look at the entire audio recording. In short, the paper shows that multi-task training is effective, where each minibatch uses each utterance twice: Once as using the whole-utterance model structure, and once using a modified model structure where all model weights referring to future information are forced to be zero.