Dual-mode ASR: Unify and Improve Streaming ASR with Full-context Modeling

1 INTRODUCTION . “ Ok Google . Hey Siri . Hi Alexa. ” have featured a massive boom of smart speakers in recent years , unveiling a trend towards ubiquitous and ambient Artificial Intelligence ( AI ) for better daily lives . As the communication bridge between human and machine , low-latency streaming ASR ( a.k.a. , online ASR ) is of central importance , whose goal is to emit each hypothesized word as quickly and accurately as possible on the fly as they are spoken . On the other hand , there are some scenarios where full-context ASR ( a.k.a. , offline ASR ) is sufficient , for example , offline video captioning on video-sharing platforms . While low-latency streaming ASR is generally preferred in most of the speech recognition scenarios , it often has worse prediction accuracy as measured in Word Error Rate ( WER ) , due to the lack of future context compared with full-context ASR . Improving both WER and emission latency has been shown to be highly challenging ( He et al. , 2019 ; Li et al. , 2020a ; Sainath et al. , 2020 ) in streaming ASR systems . Since the acoustic , pronunciation , and language model ( AM , PM , and LM ) of a conventional ASR system have been evolved into a single end-to-end ( E2E ) all-neural network , modern streaming and full-context ASR models share most of the neural architectures and training recipes in common , such as , Mel-spectrogram inputs , data augmentations , neural network meta-architectures , training objectives , model regularization techniques and decoding methods . The most significant difference is that streaming ASR encoders are auto-regressive models , with the prediction of the current timestep conditioned on previous ones ( no future context is permitted ) . Specifically , let x and y be the input and output sequence , t as frame index , T as total length of frames . Streaming ASR encoders model the output yt as a function of input x1 : t while full-context ASR encoders model the output yt as a function of input x1 : T . Streaming ASR encoders can be built with uni-directional LSTMs , causal convolution and left-context attention layers in streaming ASR encoders ( Chiu & Raffel , 2018 ; Fan et al. , 2018 ; Han et al. , 2020 ; Gulati et al. , 2020 ; Huang et al. , 2020 ; Moritz et al. , 2020 ; Miao †equal contribution et al. , 2020 ; Tsunoo et al. , 2020 ; Zhang et al. , 2020 ; Yeh et al. , 2019 ) . Recurrent Neural Network Transducers ( RNN-T ) ( Graves , 2012 ) are commonly used as the decoder in both streaming and fullcontext models , which predicts the token of the current input frame based on all previous tokens using uni-directional recurrent layers . Figure 1 illustrates a simplified example of the similarity and difference between streaming and full-context ASR models with E2E neural networks . Albeit the similarities , streaming and full-context ASR models are usually developed , trained , and deployed separately . In this work , we propose Dual-mode ASR , a framework to unify streaming and full-context speech recognition networks with shared weights . Dual-mode ASR comes with many immediate benefits , including reduced model download and storage on devices and simplified development and deployment workflows . To accomplish this goal , we first introduce Dual-mode Encoders , which can run in both streaming mode and full-context mode . Dual-mode encoders are designed to reuse the same set of model weights for both modes with zero or near-zero parameters overhead . We propose the design principles of a dual-mode encoder and show examples on how to design dual-mode convolution , dual-mode pooling , and dual-mode attention layers . We also investigate into different training algorithms for Dual-mode ASR , specifically , randomly sampled training and joint training . We show that joint training significantly outperforms randomly sampled training in terms of model quality and training stability . Moreover , motivated by Inplace Knowledge Distillation ( Yu & Huang , 2019b ) in which a large model is used to supervise a small model , we propose to distill knowledge from the full-context mode ( teacher ) into the streaming mode ( student ) on the fly during the training within the same Dual-mode ASR model , by encouraging consistency of the predicted token probabilities . We demonstrate that the emission latency and prediction accuracy of streaming ASR significantly benefit from weight sharing and joint training of its full-context mode , especially with inplace knowledge distillation during the training . We present extensive experiments with two state-of-theart ASR networks , convolution-based ContextNet ( Han et al. , 2020 ) and conv-transformer hybrid Conformer ( Gulati et al. , 2020 ) , on two datasets , a widely used public dataset LibriSpeech ( Panay- otov et al. , 2015 ) ( 970 hours of English reading speech ) and a large-scale dataset MultiDomain ( Narayanan et al. , 2018 ) ( 413,000 hours speech of a mixture across multiple domains including Voice Search , Farfield Speech , YouTube and Meetings ) . For each proposed technique , we also present ablation study and analysis to demonstrate and understand the effectiveness . With Dual-mode ASR , we achieve new state-of-the-art streaming ASR results on both LibriSpeech and MultiDomain in terms of accuracy and latency . 2 RELATED WORK . Streaming ASR Networks . There has been a growing interest in building streaming ASR systems based on E2E Recurrent Neural Network Transducers ( RNN-T ) ( Graves , 2012 ) . Compared with sequence-to-sequence models ( Chorowski et al. , 2014 ; 2015 ; Chorowski & Jaitly , 2016 ; Bahdanau et al. , 2016 ; Chan et al. , 2016 ) , RNN-T models are naturally streamable and have shown great potentials for low-latency streaming ASR ( Chang et al. , 2019 ; He et al. , 2019 ; Tsunoo et al. , 2019 ; Sainath et al. , 2019 ; Shen et al. , 2019 ; Li et al. , 2020a ; b ; Sainath et al. , 2020 ; Huang et al. , 2020 ; Moritz et al. , 2020 ; Narayanan et al. , 2020 ) . In this work , we mainly focus on RNN-T based models . He et al . specifically studied how to optimize the RNN-T streaming ASR model for mobile devices , and proposed a bag of techniques including using layer normalization and large batch size to stabilize training ; using word-piece targets ( Wu et al. , 2016 ) ; using a time-reduction layer to speed up training and inference ; quantizing network parameters to reduce memory footprint and speed up computation ; applying shallow-fusion to bias towards user-specific context . To support streaming modeling in E2E ASR models , various efforts have also been made by modifying attention-based models such as monotonic attention ( Raffel et al. , 2017 ; Chiu & Raffel , 2017 ; Fan et al. , 2018 ; Arivazhagan et al. , 2019 ) , GMM attention ( Graves , 2013 ; Chiu et al. , 2019 ) , triggered attention ( TA ) ( Moritz et al. , 2019 ) , Scout Network ( Wang et al. , 2020 ) , and approaches that segment encoder output into non-overlapping chunks ( Jaitly et al. , 2016 ; Tsunoo et al. , 2020 ) . Tsunoo et al . also applied knowledge distillation from the non-streaming model to the streaming model , but their streaming and non-streaming models do not share weights and are trained separately . To improve the latency of RNN-T streaming models , Li et al . investigated additional early and late penalties on Endpointer prediction ( Chang et al. , 2019 ) to reduce the emission latency , and employed the minimum word error rate ( MWER ) training ( Prabhavalkar et al. , 2018 ) to remedy accuracy degradation . Sainath et al . further proposed to improve quality by using two-pass models ( Sainath et al. , 2019 ) , i.e. , a second-pass LAS-based rescore model on top of the hypotheses from first-pass RNN-T streaming output . More recently , Li et al . proposed parallel rescoring by replacing LSTMs with Transformers ( Vaswani et al. , 2017 ) in rescoring models . Chang et al . further proposed Prefetching to reduce system latency by submitting partial recognition results for subsequent processing such as obtaining assistant server responses or second-pass rescoring before the recognition result is finalized . Unlike these approaches , our work explores the unification of streaming and fullcontext ASR networks , thus can be generally applied as an add-on technique without requiring extra runtime support during inference . Weight Sharing for Multi-tasking . Sharing model weights of a deep neural network for multiple tasks has been widely explored in the literature to reduce overall model sizes . In the broadest sense , tasks can refer to different objectives or same objective but different settings , ranging from natural language processing and speech recognition to computer vision and reinforcement learning . In speech recognition , Kannan et al . employed a single ASR network for multilingual ASR , and showed accuracy improvements over monolingual ASR systems . Wu et al . proposed dynamic sparsity neural networks ( DSNN ) for speech recognition on mobile devices with resource constraints . A single trained DSNN ( Wu et al. , 2020 ) can transform into multiple networks of different sparsities for adaptive inference in real-time . Chang et al . trained a single RNN-T model with LSTMs ( Hochreiter & Schmidhuber , 1997 ) for Joint Endpointing ( i.e. , predicting both recognition tokens and the end of an utterance transcription ) in streaming ASR systems . Moreover , Watanabe et al . proposed a hybrid CTC and attention architecture for ASR based on multi-objective learning to eliminate the use of linguistic resources . Another related research work in Computer Vision is Slimmable Neural Networks ( Yu et al. , 2018 ; Yu & Huang , 2019a ; b ; Yu et al. , 2020 ) . Yu et al . proposed an approach to train a single neural network running at different widths , permitting instant and adaptive accuracy efficiency trade-offs at runtime . We also adapt the training rules introduced in slimmable networks , that is , using independent normalization layers for different sub-networks ( tasks ) as conditional parameters and using the prediction of teacher network to supervise student network as inplace distillation during the training . Unlike slimmable networks in which a large model is used to supervise a small model , we propose to distill the knowledge from full-context mode ( teacher ) into streaming mode ( student ) on the fly within the same Dual-mode ASR model . Knowledge Distillation . Hinton et al . explored a simple method to “ transfer ” knowledge from a teacher neural network to a student neural network by enforcing their predictions to be close measured by KL-divergence , ` 1 or ` 2 distance . It is shown that such distillation method is effective to compress neural networks ( Yu & Huang , 2019b ) , accelerate training ( Chen et al. , 2015 ) , improve robustness ( Carlini & Wagner , 2017 ; Papernot et al. , 2016 ) , estimate model uncertainty ( Blundell et al. , 2015 ) and transfer learned domain to other domains ( Tzeng et al. , 2015 ) . 3 DUAL-MODE ASR . Most neural sequence transduction networks for ASR have an encoder-decoder structure ( Graves , 2012 ; Sainath et al. , 2020 ; He et al. , 2019 ; Li et al. , 2020a ) , as shown in Figure 1 . Without loss of generality , here we discuss how to design Dual-mode ASR networks under the most commonly used RNN-T model ( Graves , 2012 ) . In RNN-T models , we first extract mel-spectrogram feature from input speech waveform . The Mel-spectrogram feature is then fed into a neural-net encoder , which usually consists of feed-forward layers , RNN/LSTM layers , convolution layers , attention layers , pooling ( time-reduction ) layers , and residual or dense connections . In neural-net encoders , streaming ASR model requires all components to be auto-regressive , whereas full-context ASR model has no such requirement . The ASR decoder then predicts the token of current frame based on the output from the encoder and previous predicted tokens ( inference ) or target tokens ( training with teacher forcing ( Williams & Zipser , 1989 ) ) . The decoder is commonly an auto-regressive model in both streaming and full-context ASR models , thus is fully shared in Dual-mode ASR . The prediction from decoder is finally used either in decoding algorithm during inference ( e.g. , beam search ) or learning algorithm during training ( e.g. , RNN-T loss ) . As discussed above and shown in Figure 1 , it becomes clear that the major difference between streaming and full-context ASR models is in the neural-net encoder . In the following , we will first discuss the design principles of dual-mode encoder to support both streaming and full-context ASR . We provide examples including dual-mode convolution , dual-mode average pooling , and dual-mode attention layers , which are widely used in the state-of-the-art ASR networks ContextNet ( Han et al. , 2020 ) and Conformer ( Gulati et al. , 2020 ) . We will then discuss the training algorithm of Dual-mode ASR networks including joint training and inplace knowledge distillation .

Summary: This paper proposes a unified single neural network architecture to realize both streaming and full-context ASR systems. The idea is simple but very efficient. It uses the same model for both streaming and full-context ASR systems, but when we use the streaming mode, some of the network operations that use the future context are ignored (e.g., ignore the future context kernel in CNN, self-attention, or pooling is performed only with the current and history frames). The two modes interact with each other by teacher-student training, where the teacher is the full-context mode while the student is the streaming mode. The paper is well-written overall, but it requires some clarification (see my comments below).