Dual-mode ASR: Unify and Improve Streaming ASR with Full-context Modeling
Authors: Jiahui Yu
Wei Han
Anmol Gulati
Chung-Cheng Chiu
Bo Li
Tara N Sainath
Yonghui Wu
Ruoming Pang
Venue: ICLR 2021 Poster (2021)
Keywords: Speech Recognition
Streaming ASR
Low-latency ASR
Dual-mode ASR

Abstract: Streaming automatic speech recognition (ASR) aims to emit each hypothesized word as quickly and accurately as possible, while full-context ASR waits for the completion of a full speech utterance before emitting completed hypotheses. In this work, we propose a unified framework, Dual-mode ASR, to train a single end-to-end ASR model with shared weights for both streaming and full-context speech recognition. We show that the latency and accuracy of streaming ASR significantly benefit from weight sharing and joint training of full-context ASR, especially with inplace knowledge distillation during the training. The Dual-mode ASR framework can be applied to recent state-of-the-art convolution-based and transformer-based ASR networks. We present extensive experiments with two state-of-the-art ASR networks, ContextNet and Conformer, on two datasets, a widely used public dataset LibriSpeech and a large-scale dataset MultiDomain. Experiments and ablation studies demonstrate that Dual-mode ASR not only simplifies the workflow of training and deploying streaming and full-context ASR models, but also significantly improves both emission latency and recognition accuracy of streaming ASR. With Dual-mode ASR, we achieve new state-of-the-art streaming ASR results on both LibriSpeech and MultiDomain in terms of accuracy and latency.

Published as a conference paper at ICLR 2021
DUAL-MODE ASR: UNIFY AND IMPROVE STREAMING ASR WITH FULL-CONTEXT MODELING
Jiahui Yu²
Wei Han
Anmol Gulati
Chung-Cheng Chiu Ruoming Pang'
Bo Li
Tara N. S
Yonghui Wu'
Google Brain
Google LLC
(jiahuiyu, =pang)
ABSTRACT
Streaming automatic speech recognition (ASR) aims to emit each hypothesized word as quickly and accurately as possible, while full-coniext ASR waits for the completion of a full speech utterance before emitting completed hypotheses. this work, propose unified framework. Dual-mode ASR, to train a single end-to-end ASR model with shared weights for both streaming and full-context speech recognition. We show that the latency and accuracy of streaming ASR significantly benefit from weight sharing and joint training of full-context ASR, especially with implace knowledge distillation during the training. The Dual-mode ASR framework can be applied to recent state-of-the-art convolution-based and transformer-bssed ASR networks. We present extensive experiments with two state-of-ther tfe------herrtt art ASR networks, ContextNet and Conformer, on two datasets, a widely used public dataset LibriSpeech and large-scale scale dataset MultiDomain. Experiments and ablation studies demonstrate that Dual-mode ASR not only sim- plifies the workflow of training and deploying streaming and full-context ASR models, but also significantly improves both emission latency and recognition ac- curacy of streaming ASR. With Dual-mode ASR, we achieve new state-of-the-art streaming ASR results on both LibriSpeech and MultiDomain in terms of accu- racy and latency.
INTRODUCTION
"Ok Google. Hey Siri. Hi Alexa." have featured massive boom of smart speakers in recent years, unveiling a trend towards ubiquitous and ambient Artificial Intelligence (Al) for better daily lives. As the communication bridge between human and machine, low-latency streaming ASR (a.k.a,, online ASR) of central importance, whose goal is to emit each hypothesized word as quickly and accurately as possible on the fly as they are spoken. On the other hand, there are some scenarios where full-context ASR (a.k.a,, offline ASR) sufficient, for example, offline video captioning on video-sharing platforms. While low-latency streaming ASR is generally preferred in most of the speech recognition scenarios, it often has worse prediction accuracy as measured in Word Error Rate (WER), due to the lack of future context compared with full-context ASR. Improving both WER and emission latency has been shown to be highly challenging (He et al. 2019 et al. 2020 in streaming ASR systems.
al. 2020a Sainath
Since the acoustic, pronunciation. and language model (AM, PM, and LM) of a conventional ASR system have been evolved into single end-to-end (E2E) all-neural network, modern streaming and full-context ASR models share most of the neural architectures and training recipes in common, such as, Mel-spectrogram inputs, data augmentations, neural network meta- metaaarchitectures, training objec- tives, model regularization techniques and decoding methods. The most significant difference is that streaming ASR encoders are cuto-regressive models, with the prediction of the current timestep conditioned on previous ones (no future context is permitted). Specifically, let x and be the input and output sequence, t as frame index, T total length of frames. Streaming ASR encoders model the output V/: function of input I while full-context ASR encoders model the output y/, function input JI:T. Streaming ASR encoders can be built with uni-directional LSTMs, causal convolution and left-context attention layers streaming ASR encoders (Chiu & Raffel 2018 Fan
ct al. 2018 Han et al.) 2020 Gulati et al., 2020: Huang et al. 2020 Moritz et al.) 2020:
Miao
cqual contribution

---

Published as a conference paper at ICLR 2021
Prediction Lous
hh" e. [
hh' e
Decoder
Encader Cupput
SetF-artent
Fees-forwa1d
Neura-nnet
Convolution
Feod forward
Encoder Input
Hol Spectrocram
Speech Woveforr
Streaming ASR with Auto- regresegee
Full-context ASR with Full-context Encoder
Figure 1: A simplified illustration of the similarity and difference between Streaming ASR and Full- context ASR networks. Modern end-to-end streaming and full-context ASR models share most of the neural architectures and training recipes in common, with the most significant difference in the ASR encoder (highlighted). Streaming ASR encoders are auto-regressive models, with each pre- diction of the current timestep conditioned previous ones (no future context). We show examples of feed-forward layer. convolution layer and self-attention layer in the encoder streaming and full-context ASR respectively. With Dual-mode ASR, unify them without parameters overhead.
et al. 2020; Tsuooo et al. 2020 Zhang et al. 2020 Yeh et al. 2019). Recurrent Neural Network Transducers (RNN-T) (Graves 2012) are commonly used the decoder both streaming and full- context models, which predicts the token of the current input frame based on all previous tokens using uni-directional recurrent layers. Figure | iillustrates simplified example of the similarity and difference between streaming and full-context ASR models with E2E neural networks.
Albeit the similarities, streaming and full-context ASR models are usually developed, trained, and deployed separately. In this work, we propose Dual-mode ASR, framework to unify streaming and full-context speech recognition networks with shared weights. Dual-mode ASR comes with many immediate benefits, including reduced model download and storage devices and simplified development and deployment workflows. To accomplish this goal, we first introduce Dual-mode Encoders, which run in both streaming mode and full-context mode. Dual-mode encoders designed to reuse the same set of model weights for both modes with zero or near-zero parameters overhead. We propose the design principles of a dual-mode encoder and show examples on how to design dual-mode convolution, dual-mode pooling, and dual-mode attention layers. We also investigate into different training algorithms for Dual-mode ASR, specifically, randomly sampled training and joint training. We show that joint training significantly outperforms randomly sampled training terms of model quality and training stability. Moreover, motivated by Inplace Knowledge Distillation YYu & Huang (209b) in which a large model is used to supervise a small model, we propose to distill knowledge from the full-contexi mode (teacher) into the streaming mode (student) on the fly during the training within the same Dual-mode ASR model, by encouraging consistency of the predicted token probabilities.
We demonstrate that the emission latency and prediction accuracy of streaming ASR significantly benefit from weight sharing and joint training of its full-context mode, especially with implace knowledge distillation during the training. We present extensive experiments with two sfate-of-the. art ASR networks, convolution-base ContextNet (Han et al.) 2020) and conv-trame transfranformer hybrid Conformer (uulaii al. 2020), two datasets, widely used public dataset LibriSpeech (Panay-
2

---

Published as a conference paper at ICLR 2021
otov et al., 2013) (970 hours of English reading speech) and large-scale dataset MultiDo- main (Narayanan et al. 2018) (413,000 hours speech of a mixture across multiple domains in- cluding Voice Search, Farfild Speech, YouTube and Meetings). For each proposed technique, also present ablation study and analysis to demonstrate and understand the effectiveness. With Dual-mode ASR, we achieve new state-of-the-art streaming ASR results both LibriSpeech and MultiDomain in terms of accuracy and latency.
RELATED WORK
Streaming ASR Networks. There has been a growing interest in building streaming ASR systems
based E2E Recurrent Neural Network Transducers (RNN-T) (Graves 2012). Compared with
sequence-tee weceennccettt-uuueeee to-sequence models (Chorowski et al., 2014, 2015 Chorowski & Jaitly 2016 Bahdanau
jet al., 2016 Chan et al. 2016), RNN-T models
naturally streamable and have shown great po-
tentials for low-latency streaming ASR (Chang et al. 2019; He et al.] 2019 Tsunoo et al.
2019
Sainath al. 2019: Shen et al., 2019 Li al. 2020a Sainath et al. 2020 Huang et al.
2020
Moritz et al. 2020 Narayanan et al. (020). In this work, we mainly focus on RNN-T'base models.
He et al. specifically studied how to optimize the RNN-T streaming ASR model for mobile devices, and proposed a bag of techmiques including using layer normalization and large batch size to sta- bilize training: using word-piece targets (Wu et al.) 2016); using a time-reduction layer to speed up training and inference; quantizing network parameters to reduce memory footprint and speed up computation; applying shallow-fusion to bias towards user-specific context. To support streaming modeling in E2E ASR models, various efforts have also been made by modifying attention-based models such as monotonic attention (Raffel et al.] 2017: Chiu & Raffel 2017; Fan et al., 2018 Arivazhagan et al. 2019), GMM attention (Graves 2013 Chiu et al. 2019), triggered attention (TA) (Moritz et al. 2019), Scout Network (Wang et al. (2200). and approaches that segment en- coder output into oon-overlapping chunks (Jaitly al. [
Tsunoo et al., 2020). [suunoo et al. also applied knowledge distillation from the non-streaming model to the streaming model, but their streaming and non-streaming models do not share weights and are trained separately.
To improve the latency of RNN-T streaming models, Li et al. investigated additional early and late penalties Endpointer prediction (Chang et al., 2019) reduce the emission latency, and employed the minimum word error rate (MWER) training (Prabhavalkar et al. 2018) to remedy accuracy degradation. (Sainath et al.] further proposed to improve quality by using two-pass mod- els (Sainath et al.) 2019), i.e., a second-pass LAS-based rescore model on top of the hypotheses from first-pass RNN-T streaming output. More recently, et al. proposed parallel rescoring by replacing LSTMs with Transformers ( (Vaswani et al.) (017) in rescoring models. [Chang et al. further proposed Prefetching to reduce system latency by submitting partial recognition results for subsequent pro- cessing such as obtaining assistant server responses or second-pass rescoring before the recognition result is finalized. Unlike these approaches, our work explores the unification of streaming and full- context ASR networks, thus can be generally applied as an add-on technique without requiring extra runtime support during inference.
Weight Sharing for Multi-tasking, Sharing model weights of a deep neural network for multi- ple tasks has been widely explored in the literature to reduce overall model sizes. In the broadest sense, tasks can refer different objectives or same objective but different settings, ranging from natural language processing and speech recognition to computer vision and reinforcement learning. In speech recognition, KKnnan et al. employed single ASR network for multilingual ASR, and showed accuracy improvements over monolingual ASR systems. Wu et al. proposed dynamic spar- sity neural networks (DSNN) for speech recognition on mobile devices with resource constraints. A single trained DSNN (Wu et al. 2020) can transform into multiple networks of different sparsities for adaptive inference in real-time. Chang et al. trained single RNN-T model with LSTMs (Hochreiter) & Schmidhuber 1997) for Joint Endpointing (i.e., predicting both recognition tokens and the end of an utterance transcription) in streaming ASR systems. Moreover, Watanabe et al. proposed hybrid CTC and attention architecture for ASR based on multi-objective learning to eliminate the use of linguistic resources.
Another related research work in Computer Vision is Slimmable Neural Networks (Yu et al. 2018 Yu & Huang, 2019a b. Yu et al., (2020). Yu et al.) proposed an approach to train a single neural network running different widths, permitting instant and adaptive accuracy efficiency trade-offs at
3

---

Published as a conference paper at ICLR 2021
runtime. We also adapt the training rules introduced in slimmable networks, that is, using indepen- dent normalization layers for different sub-networks (tasks) as conditional parameters and using the prediction of teacher network to supervise student network inplace distillation during the training. Unlike slimmable networks in which a large model is used to supervise small model, we propose to distill the knowledge from full-context mode (teacher) into streaming mode (student) the fly within the same Dual-mode ASR model.
Knowledge Distillation. Hinton et al.) explored a simple method "transfer" knowledge from a teacher neural network to a student neural network by enforcing their predictions to be close measured by KL-divergence. L &y distance. It is shown that such distillation method is effective to compress neural networks (Yu & Huang (2019b). accelerate training (Chen et al. (2015). improve robustness (Carlini & Wagnerj 2017]] Papernot et al., 2016), estimate model uncertainty (Blundell et al. 2015) and transfer learned domain to other domains (1 (Izeng et 2015).
DUAL-MODE ASR
Most neural sequence transduction networks for ASR have an encoder-decoder structure (Graves) 2012 Sainath et al. 2020 He et al. 2019 Li et al. 2020a), as shown in Figure [ Without loss of generality, here we discuss how design Dual-mode ASR networks under the most commonly used RNN-T model (Graves] 2012). In RNN-T models, we first extract mel-spectrogram feature from input speech waveform. The -el-spectrogram feature is then fed into a neural-net encoder, which usually consists of feed-forward layers, RNN/LSTM layers, convolution layers, attention layers, pooling (timerreduction) layers, and residual or dense connections. In neural-net encoders, streaming ASR model requires all components to be auto-regressive, whereas full-context ASR model has no such requirement. The ASR decoder then predicts the token of current frame based on the output from the encoder and previous predicted tokens (inference) target tokens (training with teacher forcing (Williams & Zipser 1989)). The decoder is commonly an auto-regressive model in both streaming and full-context ASR models, thus is fully shared in Dual-mode ASR. The prediction from decoder is finally used either in decoding algorithm during inference (e.g., beam search) or learning algorithm during training (e.g., RNN-T loss).
As discussed above and shown in Figure I it becomes clear that the major difference between streaming and full-context ASR models is in the neural-net encoder. In the following, we will first discuss the design principles of dual-mode encoder to support both streaming and full-context ASR. We provide examples including dual-mode convolution, dual-mode average pooling, and dual-mode attention layers, which are widely used in the state-of-the the-art-r ASR networks ContextNet (Han et al., 2020) and Conformer (Gulati al. 2020p. We will then discuss the training algorithm of Dual-mode ASR networks including joint training and inplace knowledge distillation.
3.1 DUAL-MODE ENCODER
Unifying streaming and full-context ASR models requires two design principles of Dual-mode En- coder:
1. Each layer in a dual-mode encoder should be either dual-mode streaming (a.ka., causal). Since streaming encoder has to be auto-regressive which prohibits any future context, any full-conted (a.k.a,, non-causal) layer violates this constraint.
2. The design of dual-mode layer should not introduce significant amount of additional parameters, compared with its streaming model. We aim at supporting full-context ASR on top of the streaming model with near-zero parameters overhead.
We show examples below by applying the above two design principles to ContextNet (Han et al.) 2020) and Conformer (Gulati al. 2020), in which the encoders are composed of pointwise oper- ators (feed-forward net, residual connections, activation layers, striding, dropout, etc.), convolution, average pooling, self-attention and normalization layers.
Pointwise operators are naturally dual-mode layers. Neural network layers that connect input and output neurons within each timestep (no across-connections among different timesteps) are often referred as pointwise operators (Chollet 2017), including feed-forward layers (a.k.a., fully- connected layers or 1 x convolution layers), activation layers (e.g., ReL.U, Swish (Ramachandran
4

---

Published as a conference paper at ICLR 2021
Connected both streaming full-context mode.
Connected only in full-context mode.
ReduteMean
Dual-mode Temporal-wise Convolution
Dual-mode Temporal-wise Average Pooling (left-context pool vs. full-context pool)
(kernel size: 3
Figure 2: Dual-mode convolution and average pooling layer for Dual-mode ASR.
et al., 2017)) residual and dense connections (He et al. 2016 Huang et al.) 2017), striding layers, dropout layers (Srivastava et al. 2014) and element-wise multiplications. As there is no information propagation through time, pointwise operators are naturally dual-mode layers and can be directly used in Dual-mode ASR encoders.
Dual-mode Convolution. Convolution layers, however, convolve feature across its neighbor timesteps within a fixed window (e.g.. kernel size is 3, 5, or larger), and has been widely used in sequence modeling (Gehring et al. 2017 Han et al. 2020 Gulati et al. 2020).
conv-based
streaming ASR models, causal convolution layers (Oord et al. 2016) are used where the convolu- tion window is biased to the left (self-included). As shown Figure 2 on the left, to support both streaming and full-context modes with shared weights, we first construct normal symmetric con- volution of kernel size & which will be applied in full-context mode. Then we mimic the causal convolution of kernel size (k 1)/2 by constructing Boolean mask and multiplying with the full- context convolution kernel before applying the actual convolution of streaming mode in Dual-mode ASR encoders.
The design of dual-mode convolution introduces (A: 1)/2 additional parameters to support full- context convolution (k) compared with streaming convolution 1)/2). However, note that in convolution-base models, these temporal-wise convolution layers only take a tiny amount of total model size and most of the weights are operators. For example, in ContextNet (Han et al. 2020), temporal convolution has less than 1% of total model size, thus parameters overhead negligible.
convolution layers which are fully shared pointwise
Dual-mode Average Pooling, sguuezz-annd-exiittiie (Hu et al. 2018) (SE) modules are used in ContextNet enhance the global context encoding. Each SE module a sequential stack of av- erage pooling (through time) layer, feed-forward layer, activation layer, another ford-forward layer and elementwise multiplication. To support both modes, dual-mode average pooling layer used as shown in Figure on the right. Dual-mode average pooling layer is parameter-free thus does not introduce additional model parameters. also trains in parallel in streaming mode, easily imple- mented with "cumsum" function in both TensorFlow and PyTorch.
Dual-mode Self-attention. Self-attention (a.k.a. intra-attention) is an attention mecha- nism weighting different positions of a single sequence in order to compute representation of the same sequence. It is heavily used in Con- former (Gulati et al. 2020) ASR networks. The attention layer itself is parameter-free (projec- tion layers before attention are fully shared), and is composed of matrix multiplication of the key and the query, followed by softmax over keys, before another matrix multiplication with the value. As shown in Figure 3 in dual-mode attention layer, the softmax is performed on the left context only in streaming mode (rectangle with solid line), compared with the full-context mode (rectangle with dash line). We find this simple form of dual-mode self-attention works well in practice.
Mastedd
((x))
Suroniing Adtcoro
Query
Figure Dual-mode self-attention layer.
5

---

Published as a conference paper at ICLR 2021
Algorithm Pseudocode of training Dual-mode ASR networks.
Requizes: data loader) context manager with support of mode switching by network mode ()
@oall__ode_nnttree with support of running both modes under context managor;
for X, y in data dotd_ # Lead e minibatch of speech Laput x and text label y, Switch context to fuulcoonext' mode.
with waw___odd_netwrr..ooo
Fullzontexe')
Compute I-onntxt prediction given speech input moa__mooe_network.. network. seevarddencodeerdeedddee (X, y) Compute RNN-T less of full-context mode.
text label y.
Iu] fullcontext pred - dual_moce
lcontext_loss
ennt_lass(fu)) (full concext y)
with doa__oode_netwock.rr
stresming' Switch context to "streaming' mode. Compute stzesming prediction given speech input x text lsbal y. st (creaming_pred dual__ma poodee_eework... wor.or...oowwrd................d sevardd_eccderrrdeeddee (X, y) Compute RNV-T loss of streaming mode. y)
streaming_lose zoat_loas(streaming
Add inplace knovledge dist! loas (full-context prediction as teacher). distill_loss inplace_dlatil1
tetreaming stop_gradient (tulicontext_preee
Camputa total loss as gun of full-context, strcaming and distillation losses.
tullcontext_ streaming_loss distill_loss loss. .asckawwk))d))) Update weights.
Dual-mode Normalization. Moreover, following Yu ct al. (0018). we also find the normalization statistics like means and variances are different in streaming and full-context modes. Thus, for nor- malization layers including BatchNorm (loffe & Szegedy 2015) and LayerNorm (Ba et al., (016) in Dual-mode ContextNet and Dual-mode Conformer, instantiate two separate norm layers ded- icated to streaming and full-context mode respectively.
3.2 TRAINING DUAL-MODE ASR NETWORKS
The training algorithm of Dual-mode ASR networks is outlined in Algorithm I In this section, we discuss two important training techniques: joint training and inplace knowledge distillation.
Joint Training. To train Dual-mode ASR networks, given a batch of data in each training iteration, we can either randomly sample one from two modes to train, or train both modes and aggregate their losses. In the former approach, referred as randomly sampled training, we can control the importance of streaming and full-context modes by setting different sampling probabilities during training. In the latter approach, referred as joint training, importance can also be controlled by assigning different loss weights to balance streaming and full-context modes. Empirically we find joint training leads to better model qualities overall thus is adopted in all of our experiments. We will show an ablation study comparing randomly sampled training and joint training. In all of our experiments, treat streaming and full-context mode to be equally important by assigning equal importance during training.
Inplace Knowledge Distillation. Additionally we propose distill knowledge from the full-context mode (teacher) into the streaming mode (student) on the fly within the same Dual-mode ASR model, by encouraging consistency of the predicted token probabilities. Since in each iteration we always compute predictions of both modes, the teacher prediction comes for free (no additional computation or memory cost), as shown Algorithm[] We use the efficient knowledge distillation introduced by Panchapagesan et al.) which is based on the KL-divergence between full-context and streaming over the probability three parts: Plabet- Potank and Plabet Potant- We note that the prediction of full-context mode (teacher) usually has lower latency (since it has no incentive to delay its output), thus we can control the target emission latency of streaming mode (student) by shifting the prediction of full-context mode, before applying distillation loss. We do small-scale hyper-parameter sweep from -2 to 2 frames to shift for ContextNet and Conformer in our experiments.
4 EXPERIMENTS
4.1 MAIN RESULT
Measuring Latency. Latency measurement itself challenging for streaming ASR systems. Mo- tivated by Prefetching (Chang al. 2020) technique, we measure latency as the difference of two

---

Published as a conference paper at ICLR 2021
timestamps: 1) when the last token is emitted in the finalized recognition result; 2) the end of the speech when a user finishes speaking. We find this is especially descriptive of user experience in real-world ASR applications like Voice Search. ASR models that capture stronger contexts can emit the full hypothesis even before they are spoken, leading to a negative latency. Moreover, instead of naively averaging latency over all utterances, we report both median and 90th percentile of all utterances in test set, denoted as Latency 50 and Latency @90, to better characterize latency by excluding outlier utterances. To evaluate the model quality, we report WER only for full-context models and both WER and latency for streaming models (full-context latency is meaningless).
Datasets. We conduct our experiments on two datasets: public widely used dataset Lib- riSpeech (Panayotov et 2015) (1,000 hours of English reading speech) and a large-scale dataset MultiDomain (413,000 hours speech, 287 million utterances of mixture across multiple domains including Voice Search, YouTube, and Meetings). Table [ summarizes the information and statis- ties of two datasets. For LibriSpeech, we report our evaluation results on TestClean and TestOther (noisy) sets and compare with other published baselines. For MultiDomain, we report our evaluation results on Voice Search test set and compare with our reproduced baselines. For fair comparisons, on each dataset train and report our models and baselines with the same settings (number of training iterations, parammeate,,sss optimizer, regularization, etc.). We note that these inherited from previous work HHnn et al. (2020): Gulati et al. (2020) and not specifically tuned for our dual-mode models.
Table 1: Summary of datasets we used in our experiments.
Dataset Name
# Hours
# Utterances
Speech Domain
LibriSpeech
970
281, 000
Single domain of English reading speech.
Panayotov et al. 2015)
MultiDomain
Multiple domains including: Voice Search, Farfield Specch, YouTube and Mcctings.
2018 413, 000
287, 000, 000
Narayanan et al.
ASR Networks. We use two recent state-of-the-an art ASR networks to demonstrate the effectiveness of our proposed methods, ContextNet (Han et al.] (2200) and Conformer (Gulati al.] 2020). The encoder of ContextNet is based on depthwise-separable convolution (Chollet, 2017) and squeeze- and-excitation modules (Hu et al. 2018). In depthwise-sepaaabee
convolution of Dual-mode Con-
textNet, the weights of 1 x convolutions are fully shared between streaming and full-context mode, whereas for temporal-wing wise convolution we follow the design of Dual-mode Convolution proposed in Section 3.1 Note that in ContextNet, temporal-wise convolutions only take less than 1% of the model size thus the parameters overhead of full-context mode is negligible compared with steaming mode. In squeeze- qqueeze-and-extee and-excitation modules, use dual-mode average pooling layers (Section 3.1 to support both streaming and full-context mode without additional parameters.
Conformer (Gulati al. 2020) combines convolution and transformer model both local and global dependencies of speech sequences in a parameter-efficient way. In Dual-mode Conformer, we replace all convolution and transformer layers with their dual-mode correspondents (Section 3.1] Moreover, for normalization layers including BatchNorm (Toffe & Szegedy) 2015) and Layer- Norm (Ba et al. 2016) in Dual-mode ContextNet and Dual-mode Contormer, we instantiate two separate norm layers for streaming and full-context mode respectively.
Training Details and Results. We train our models exactly following our baselines Con- textNet (Han et al. 2020) and Conformer (Gulati et al.] 2020). using Adam optimizer (Kingmal & Ba 2014), SpecAugment (Park et al. 2019) and a transformer learning rate schedule (Vaswani 2017) with warm-up (Goyal et al. 2017). Our main results are summarized in Table 2nd Ta-
et al.
We also add a streaming ContextNet Look-ahead baseline (6 frames, 10ms per frame, totally 60ms look-ahead latency) in Table 3 by padding additional frames at the end of the input utterances. As shown in the tables, the streaming mode in Dual-mode ASR models has significantly better la- tency and similar higher WER results, surpassing other baselines including conventional models, LSTM based transducers (Sainath et al. 2020), seaformerr-rrassducee (Zhang et al., 2020) and some others.
ble 3
7

---

Published as a conference paper at ICLR 2021
Table Summary our results on MultiDomain dataset (Narayanan et al., 2018). We report WER on Voice Search test set. Compared with standalone ContextNet and Conformer models, Dual-mode ASR models have slightly higher accuracy and much better streaming latency. ASR models that capture stronger contexts can emit the full hypothesis even slightly before they spoken, leading to negalive latency.
Method
Mode
# Params (M)
VS Test WER(%)
Latency @5 (ms)
Latency @ 90 (ms)
ContextNet
Full-context
133
5.1
Conformer
Full-context
142
5.2
LSTM (Sainath al. 2020
Streaming
179
6.4
190
350
ContextNct (Han et al.
2020
Streaming
133
6.1
160
Conformer GGulati et al
2020
Streaming
142
6.1
160
300
Full-context Streaming
4,9
Dual-mode ContextNet
133
6.0 (4.1)
(-150)
220
Full-context
5.0
Dual-mode Conformer
142
Streaming
6.0 (40.1)
-50 (-210)
(-210
130 (-170)
Table 3: Summary of our results on Librispeech dataset (Panayotov et al.
2015)) We report WER on TestClean and TestOther (noisy) set. Compared with standalone ContextNet and Conformer models, Dual-mode ASR models have both higher accuracy in average and better streaming latency.
Method
Mode
Params (M)
Test Clean/Other
Latency @5 Latency
WER(%)
(ms)
(ms)
LSTM-LAS
Full-context
360
2.6 / 6,0
QuartzNet-CTC
Full-context
19
3.9 / 11.3
Transformer
Full-context
29
3.1 / 7,3
Transformer
Full-context
139
2.4 / 5.6
ContextNet
Full-context
31.4
2.4 /
Conformer
Full-context
30.7
2.3 / 5.0
Transformer
Streaming
18.9
5.0 / 11.6
80
190
ContextNet
Streaming
31.4
4.5 / 10.0
70
270
Conformer
Streaming
30.7
4.6 / 9.9
140
280
ContextNet Look-ahead
Streaming
31.4
4.1 / 9.0
150
420
Full-context
3.1 / 7.9
Dual-mode Transformer
29
Streaming
4.4 (-0.6) 11.5 (0.1)
-50 (-130)
(-160)
Full-context Streaming
2.3 / 5.3
Dual-mode ContextNet
31.8
3.9 (-0.6) 8.5 (-1.5
40 (30)
160 (-110)
Full-context Streaming
2.5 / 5.9
Dual-mode Conformer
30.7
3.7 (0.9) 19.2 (4.7)
10 (-130)
90 (-190)
4.2 ABLATION STUDY
In this section, we perform various ablation studies to support and understand the effectiveness of each technique in Dual-mode ASR. We train Dual-mode ContextNet on LibriSpeech training set with exactly same settings and report WER, Latency@55 50 and Latency@ (irency@99 90 on TestOther set of streaming mode. We specifically study three techniques and their combinations including weight shuring.join training and implace knowledge distillation during the training.
During the training we distill knowledge from full-context mode (teacher) into streaming mode (student) on the fly within the same dual-mode model. Inplace distillation during the training comes for free as shown in training Algorithm]] I But what we simply share weights and jointly train them without distillation? As shown in the second row of Table the model without inplace distillation during the training has worse results compared to the baseline.
Given batch of data for each training iteration, train both modes and aggregate their losses. We also show results randomly sampled training in the third row of Table which leads to even worse performance. Note that with randomly sampled training, cannot apply inplace distillation
8

---

Published as a conference paper at ICLR 2021
easily either because in each training iteration there is only prediction from either streaming mode or full-context mode.
Weight sharing reduces the model size which is of the major motivation of Dual-mode ASR. However, what if we simply train two individual models and use knowledge distillation with full- context model as the teacher? As shown in the last row of Table 4 the results are better than other ablation but still worse than the Dual-mode ASR baseline. It might indicate that weight sharing itself encourages learning better deep representation for streaming ASR. Weight sharing has been shown empirically to improve Multilingual ASR (Kannan et al. 2019), Model Pruning (Wa et al.) 2020), Endpointing (Hochreiter & Schmidhuber 1997) and some Computer Vision problems Yu et al. 2018 and this intriguing property need to be studied in more details as a future work.
Table 4: Ablation studies of weight sharing, joint training and inplace distillation. We report WER on TestOther (noisy) set (Panayotov al., 2013) using ContextNet with same training settings.
Weight Sharing
Joint Training
Inplace Distillation
TestOther WER(%)
Lateney @50
Latency@ 90
(ms)
(ms)
8.5
40
160
x
10.2 (+1.7)
120 (+80)
310 (+150)
x
10.6 (+2.))
90 (-50)
290 (+130)
9.9 (41.4)
(-10)
210 (450)
Further, we visualize the emission lattices dual-mode ASR models trained with and without in- place knowledge distillation. We randomly sampled two audio sequences LibriSpeech TestOther set and plotted their emission lattices of streaming mode in Figure 4 X-axis represents the speech input frames while Y-axis represents the text output labels (tokens). Figure 4soww that with knowl- edge distillation from full-context mode in Dual-mode ASR, streaming mode emits faster and has much less latency, which is very critical for product datasets like MultiDomain presented in work.
Dxmpple tovening Oualemoce witeust Dacikation
xo
d
bample Straainng Dau-mode suimed with Dtatitation
so
46
Dumple Staaringg Dusirmode ASR sained witeuu Obstilatic
Exanple 2, Steaking Hode Daier takned with Diliaiin
Figure 4: Two speech-text pair comparison of Dual-model ASR models trained with and without inplace distillation by visualization of their streaming emission lattices. X-axis represents the speech input frames while Y-axis represents the text output labels (tokens). Inplace distillation significantly reduces emission latency of streaming mode in Dual-mode ASR models which is critical in real- world applications.
5 CONCLUSION
In this work, have proposed a unified framework, Dual-mode ASR, unify and improve stream- ing ASR by joint full-context modeling. We hope our exploration will inspire streaming models in other fields such simultaneous machine translation and video processing.

---

Published as a conference paper at ICLR 2021
REFERENCES
Naveen Arivazhagan, Colin Cherry, Wolfgang Macherey, Chung-Chen Chiu, Semih Yavuz, Ruom- ing Pang. Wei Li, and Colin Raffel. Monotonic infinite lookback attention for simultaneous machine translation. In ACL, 2019.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint orXiv:1607.064900 2016.
Dzmitry Bahdanau, Jan Chorowski, Dmitriy Serdyuk, Philemon Brakel, and Yoshua Bengio. End- to-end attention-based large vocabulary speech recognition. In 2016 IEEE international confer- ence on acoustics, speech and signal processing (ICASSP), pp. 4945-4949. IEEE, 2016.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. arXiv preprint arXiv:/505.00544, 2015.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium security and privacy (sp). pp. 39-57. IEEE, 2017.
William Chan, Navdeep Jaitly, Quoe Le, and Oriol Vinyals. Listen, attend and spell: A neural network for large vocabulary conversational speech recognition. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4960-4964. IEEE, 2016.
Shuo- Shin Chang, Rohit Prabhavalkar, Yanzhang He, Tara N Sainath, and Gabor Simko. Joint endpointing and decoding with end-to-end models. In ICASSP 2019-2019 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 5626-5630. IEEE, 2019.
Shuo-Yio Mhio Chang, Bo Li, David Rybach, Yanzhang He, Wei Li, Tara Sainath, and Trevor Strohman. Low latency speech recognition using end-to-end prefetching. In Interspeech. ISCA, 2020.
Tianqi Chen, Ian Goodfellow, and Jonathon Shlens. Net2net: Accelerating learning via knowledge transfer. arXiv preprint
2015.
Chung-Cheng Chiu and Colin Raffel. 1rXiv: 2017.
Monotonic chunkwise attention.
arXiv preprint
Chung-Choe Chiu and Colin Raffel. Monotonic chunkwise attention. In International Conference on Learning Representations, 2018.
Chung-Chenne Chiu, Wei Han, Yu Zhang, Ruoming Pang, Sergey Kishchenko, Patrick Nguyen, Arun Narayanan, Hank Liao, Shuyuan Zhang, Anjuli Kannan, Rohit Prabhavalkar, Zhifeng Chen, Tara Sainath, and Yonghui Wu. A comparison of end-to-end models for long-form speech recognition. In ASRU, 2019.
François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1251-1258, 2017.
Jan Chorowski and Navdeep Jaitly. Towards better decoding and language model integration in sequence sequence models. arXiv preprint arXiv: 1612.02699, 2016.
Jan Chorowski, Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. End-to-end con- tinuous speech recognition using attention-basse recurrent nn: First results. arXiv preprint urXiv:/4122 1602, 2014.
Jan K Chorowski. Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. Attention- models for speech recognition. In Advances in neural information processing systems, pp. 577-585, 2015.
Ruchao Fan, Pan Zhou, Wei Chen, Jia Jia, and Gang Liu. An online attention-based model for speech recognition. arXiv preprint arXiv:1811.0522 2018.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. Convolutional sequence sequence learning. arXiv preprint arXiv: 170500300 2017.
10

---

Published as a conference paper at ICLR 2021
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint urXiv:1706002677 2017.
Alex Graves. Sequence transduction with recurrent neural networks.
arXiv preprint
arXiv:1211.3711,22 2012.
Alex Graves.
Generating sequences with recurrent neural networks, 2013.
Anmol Gulati, James Qin, Chung-Chemn Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. Conformer: cnnvolution-augmentt transformer for speech recognition. arXiv preprint arXiv:2005.08100, 2020.
Wei Han, Zhengdong Zhang, Yu Zhang, Jiahui Yu, Bhung--heeg Chiu, James Qin, Anmol Gulati, Ruoming Pang, and Yonghui Wu. Contextnet: Improving convolutional neural networks for automatic speech recognition with global context. arXiv preprint arXiv:2005.03191, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Yanzhang He, Tara N Sainath, Rohit Prabhavalkar, Ian McGraw, Raziel Alvarez, Ding Zhao, David Rybach, Anjuli Kannan, Yonghui Wu, Ruoming Pang, et al. Streaming end-to-end speech recog- nition for mobile devices. ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6381-6385. IEEE, 2019.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv: 150300233,,, 2015.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitttie networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141, 2018.
Gao Huang. Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 4700-4708, 2017.
Wenyong Huang, Wenchao Hu, Yu Ting Yeung, and Xiao Chen. Cony-transformer trans- ducer: Low latency, low frame rate, streamable end-to-end speech recognition. arXiv preprint arXiv:2008.05750, 2020.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint /502.03 2015.
Navdeep Jaitly, Quoe Le, Oriol Vinyals, llya Sutskever, David Sussillo, and Samy Bengio. An online -equencc-to-sequenee model using partial conditioning. In Advances in Neural Information Processing Systems 29, pp. 5067-5075. 2016.
Anjuli Kannan, Arindrima Datta, Tara Sainath, Eugene Weinstein, Bhuvana Ramabhadran, Yonghui Wu, Ankur Bapna, Zhifeng Chen, and Seungji Lee. Large-scale multilingual speech recognition with streaming end-to-end model. arXiv preprint 1909:: 90330, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Bo Li, Shuo-yiin Chang, Tara N Sainath, Ruoming Pang, Yanzhang He, Trevor Strohman, and Yonghui Wu. Towards fast and accurate streaming end-to-end asr. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6069-6073. IEEE, 2020a.
Wei Li, James Qin, Chung-Cheng Chiu, Ruoming Pang, and Yanzhang He. Parallel rescoring with transformer for streaming on-device speech recognition. arXiv preprint arXiv:2008. 13093, 2020b.
11

---

Published as a conference paper at ICLR 2021
Haoran Miao, Gaofeng Cheng, Changfeng Gao, Pengyuan Zhang, and Yonghong Yan. Transformer- based online etc/attention end-to-end speech recognition architecture. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6084- 6088. IEEE, 2020.
Niko Moritz, Takaaki Hori, and Jonathan Le Roux. Triggered attention for end-to-end speech recog- nition. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5666-5670. IEEE, 2019.
Niko Moritz, Takaaki Hori, and Jonathan Le. Streaming automatic speech recognition with the transformer model. In ICASSP 2020-2020 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 6074-6078, IEEE, 2020.
Arun Narayanan, Ananya Misra, Khe Chai Sim, Golan Pundak, Anshuman Tripathi, Mohamed Elfeky, Parisa Haghani, Trevor Strohman, and Michiel Bacchiani. Toward domain-inwe -nvaarian speech recognition via large scale training. In 2018 IEEE Spoken Language Technology Workshop (SL7), pp. 441 447. IEEE, 2018.
Arun Narayanan, Tara N Sainath, Ruoming Pang. Jiahui Yu, Chung-Cheng Chiu, Rohit Prab- havalkar, Ehsan Variani, and Trevor Strohman. Cascaded encoders for unifying streaming and non-streaming ast. arXiv preprint arXiv:2010. 14606, 2020.
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalehbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.0000 1609.03499 2016.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5206-5210. IEEE, 2015.
Sankaran Panchapagesan, Daniel S Park, Shuug-Chhng Chiu, Yuan Shangguan, Qiao Liang, and Alexander Gruenstein. Efficient knowledge distillation for mn-transducer models. arXiv preprint arXiv:2011.061 2020.
Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as defense adversarial perturbations against deep neural networks. In 2016 IEEE Symposium on Security and Privacy (SP), pp. 582-599 597. IEEE, 2016.
Daniel S Park, William Chan, Yu Zhang, Chung- Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le. Specaugment: A simple data augmentation method for automatic speech recognition. 2019.
arXiv preprint 1rXiv:1904.000
Rohit Prabhavalkar, Tara N Sainath, Yonghui Wu, Patrick Nguyen, Zhifeng Chen, Chung-Cheng Chiu, and Anjuli Kannan. Minimum word error rate training for attention-based sequence-to- sequence models. In 2018 IEEE International Conference on Acoustics, Speech and Signal Pro- cessing (ICASSP). pp. 4839-4843. IEEE, 2018.
C. Raffel, M. Luong, P. J. Liu, R.J. Weiss, and D. Eck. Online and Linear-Time Attention by Enforcing Monotonic Alignments. In Proc. ICML, 2017.
Prajit Ramachandran, Barret Zoph, and Quoe Le. Searching for activation functions. arXiv preprint arXiv: 1710.05941 2017.
Tara N Sainath, Ruoming Pang, David Rybach, Yanzhang He, Rohit Prabhavalkar, Wei Li, Mirkó Visontai, Qiao Liang, Trevor Strohman, Yonghui Wu, et al. Two-pass end-to-end speech recogni- tion. arXiv preprint arXiv: 1908. 10992, 2019.
Tara N Sainath, Yanzhang He, Bo Li, Arun Narayanan, Ruoming Pang, Antoine Bruguier, Shuo- yiin Chang, Wei Li, Raziel Alvarez, Zhifeng Chen, et al. A streaming on-device end-to-end model surpassing sirver-side conventional model quality and latency. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6059-6063 IEFE, 2020.
12

---

Published as a conference paper at ICLR 2021
Jonathan Shen, Patrick Nguyen, Yonghui Wu, Zhifeng Chen, Mia X Chen, Ye Jia, Anjuli Kannan, Tara Sainath, Yuan Cao, Chung-Cheng Chiu, et al. Lingvo: a modular and scalable framework for segueece-to-sequeeee modeling. arXiv preprint arXh::1902.08295, 2019.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1958 2014.
Emiru Tsunoo, Yosuke Kashiwagi, Toshiyuki Kumakura, and Shinji Watanabe. Towards online end-to-end transformer automatic speech recognition. arXiv preprint arXiv: 2019.
Emiru Tsunoo, Yosuke Kashiwagi, and Shinji Watanabe. Streaming transformer asr with blockwise synchronous inference. arXiv preprint arXiv:2006. 14941, 2020.
Eric Tzeng, Judy Hoffman, Trevor Darrell, and Kate Saenko. Simultancous deep transfer across domains and tasks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4068-4076, 2015.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, pp. 5998-6008, 2017.
Chengyi Wang, Yu Wu, Shujie Liu, Jinyu Li, Liang Lu, Guoli Ye, and Ming Zhou. Low latency end-to-end streaming speech recognition with a scout network. arXiv preprint arXiv:2003. 10369, 2020.
Shinji Watanabe, Takaaki Hori, Suyoun Kim, John Hershey, and Tomoki Hayashi. Hybrid etc/at- tention architecture for end-to-end speech recognition. IEEE Journal of Selected Topics in Signal Processing, (8): ((8))12400) 240-1253, 2017.
Ronald Williams and David Zipser. learning algorithm for continually running fully recurrent neural networks. Neural computation, 1(2):270-25) 280, 1989.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's neural machine trans- lation system: Bridging the gap between human and machine translation. arXiv preprint 17Xvv:160.00814 2016.
Zhaofeng Wu, Ding Zhao, Qiao Liang, Jiahui Yu, Anmol Gulati, and Ruoming Pang. Dynamic sparsity neural networks for automatic speech recognition. arXiv preprint arXiv:2005.1062. 10627, 2020.
Ching-Feng Yeh, Jay Mahadeokar, Kaustubh Kalgaonkar, Yongqiang Wang, Duc Le, Mahaveer Jain, Kjell Schubert, Christian Fuegen, and Michael L Seltzer. Transformer-transdueer End-to- end speech recognition with self-attention. arXiv preprint 1910v:1910.10 12977. 2019.
Jiahui Yu and Thomas Huang. Autoslim: Towards one-shot architecture search for channel numbers. arXiv preprint arXiv: 1903../903.11 1728, 2019a.
Jiahui Yu and Thomas S Huang. Universally slimmable networks and improved training techniques. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1803-1811, 2019b.
Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks.
In International Conference on Learning Representations, 2018.
Jiahui Yu, Pengchong Jin, Hanxiao Liu, Gabriel Bender, Pieter-Jan Kindermans, Mingxing Tan, Thomas Huang, Xiaodan Song, Ruoming Pang, and Quoc Le. Bignas: Scaling up neural archi- tecture search with big single-stage models. arXiv preprint arXiv:2003.111422 2020.
Qian Zhang, Han Lu, Hasim Sak, Anshuman Tripathi, Erik McDermott, Stephen Koo, and Shankar Kumar. Transformer transducer: A streamable speech recognition model with transformer en- coders and mn-t loss. In /CASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7829-7833 IEEE, 2020.
13


Reviews:
[{"review_id": "n7jxTnuqllz", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an unified framework for both streaming and non-streaming ASR and the knowledge transfer between them. The results show that both latency and performance are improved. The benefit of training full-context and streaming together are two folds: 1) Current full-context and streaming ASR are trained separately. Since usually the performance of streaming ASR is inferior to the full-context version, the unified training scheme could enforce the model to fit both tasks well, thus could serve as some kind of regularization. 2) The weight sharing proposed in this paper could make it more efficient for deploying both streaming and non-streaming ASR at the same time.\n\nCons:\n1. Though conformer and context achieved SOTA performance as full-context models, they are pretty new and not widely acknowledged and studied compared to Transformer. Adding results of Transformer could better support the claims and make the contribution in this paper more accessible to the community.\n2. The source of improvement on latency is not well explained.\n3. “It might indicate that weight sharing itself encourages learning better deep representation for streaming ASR.”, this claim should be further validated.\n\nQuestion and comments:\n1. Adding some related work on knowledge distillation could make it a more complete story.\n2. Seems the statistics of MultiDomain dataset are not consistent in the paper (413,000 hours vs 163,000 hours)\n3. \"For fair comparisons, on each dataset we train and report our models and baselines with the same settings (number of training iterations, hyper-parameters, optimizer, regularization, etc.).\" Could it be possible different models perform the best with different hyperparameter settings?\n4. It could be interesting if the visualization of knowledge transferred from full-context model to streaming model is investigated.", "extra_scores": {}, "extra_text": {}}, {"review_id": "jlYHsTcg_Mw", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a pragmatic approach to unifying end-to-end speech-recognition models for whole-utterance and streaming models. Streaming models are defined as not using any future audio, while whole-utterance (or \"full context\") models can look at the entire audio recording. In short, the paper shows that multi-task training is effective, where each minibatch uses each utterance twice: Once as using the whole-utterance model structure, and once using a modified model structure where all model weights referring to future information are forced to be zero.\n\nThe idea is not revolutionary, but it is interesting to see that it works at no loss of accuracy. Interestingly, the results of the proposed multi-task model are actually a little better compared to training distinct models on the two modalities. This is shown to be the case for two end-to-end model architectures, ContextNet and Conformer. The result is of high practical relevance.\n\nI do disagree with the naming \"Universal ASR.\" The word \"universal\" is, well, quite universal and can refer to many dimensions, such as whole-utterance vs. streaming, distant vs. close talking, wideband vs. narrowband, various domains, various tasks, multi-linguality, etc. While I understand that in today's crowded ML world, one wants to occupy powerful marketing names for ones own work, I strongly believe that they should not be over-stating. For this paper to be acceptable for publication, I believe the authors should tone down the \"Universal\" moniker, and choose a different name that delineates the dimension more accurately.\n\nFurther, I would like to note that although this is supposed to be a double-blind review, the non-blind reference for the internal corpus gives away that this is a Google paper.\n\n**Quality**: The results have been obtained on two quality data sets, one public generally accepted corpus (LibriSpeech), and one Google-internal set of a whopping 413 thousand hours. I consider the results both convincing and of interest to speech-interested members of the ICLR community.\n\nAs one feedback point, the paper does not seem to mention a simple and common technique of incorporating at least some future information, by pre-rolling a fixed number of speech frames, e.g. 5, and incorporating those e.g. by stacking or limited-forward looking lower-layer convolution(s) or self-attention. I think it is important that the authors comment on this, since there is still a one-point WER gap due to streaming. To what degree would a simple lookahead like this trade latency for accuracy? Is it possible to build a model that is universal over different numbers of lookahead frames, as to allow for an adaptive lookahead (e.g. pre-roll 0.5 seconds at sentence start, but less later; maybe depending on context such as the words recognized so far)?\n\nAs another feedback point, I find the references somewhat Google-heavy. Going through the references from the start, it takes a while to find a non-Google reference. Exaggerated self-referencing that gives the incorrect impression that Google invented everything is not appreciated by all members of the community. I suggest that the authors rebalance, or at least adjust the order. In some cultures, it is considered polite to mention yourself last.\n\n**Clarity**: The paper generally well-written and easily understandable, although I have a few feedback items.\n\nThe term \"inplace knowledge distillation\" is a little confusing. Initially I thought this was done during inference, and was curious how that may be accomplished. However, it is a training-only technique. Please add that to where this introduced (\"at training time\").\n\nI kindly ask the authors to be more precise in the results table. What is the unit of Latency? I presume ms, but it could also be speech frames. This is not stated, the string \"ms\" does not occur once. Also, I find it confusing that the table headings for the WER columns are the corpus names, while for latencies, it is the actual meaning of the column (\"Latency@X\"). So please (1) use WER for these columns, and (2) add units for both, e.g. by saying \"WER [%]\" and \"Latency@50 [ms]\". Further, the readability of Tables 3 and 4 could be improved by horizontally aligning the decimal points.\n\nI also kindly ask the authors to have the text proof-read by a native speaker of English, and/or an automated grammar checker which Google certainly has. I noticed a few missing articles.\n\n**Originality**: With the hindsight of having read the paper, the basic idea does not *seem* original at all, but rather straight-forward. The originality of the paper lies in asking the right question, which in this case then naturally leads to the proposed method.\n\n**Significance**: The method allows to reduce the number of models to train by half, which has tremendous practical impact for anyone operating an ASR service. While there is a clear reduction of overhead of model validation and deployment process, it is not clear from the paper whether training the joint model is cheaper than training two individual models (since each utterance is trained twice per minibatch). It would be great if the authors could share that information somewhere.\n\n**Pros**: An interesting paper that provides a simple, well-working solution to a real practical problem, evaluated on a massive real-life corpus.\n\n**Cons**: Some room for improvement of clarity as listed above. Lack of consideration of simple approach of pre-rolling a few frames of speech.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "cnbGWQAJQeH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This submission proposes a framework for training online and offline ASR models. Experimental results suggest that at least on Librispeech this approach provides tangible benefits for online ASR models. \n\nQuality: The quality of this submission suffers from (a) mostly verbal presentation (e.g. Figure 1 is a very inefficient way to show that prediction at time t is either dependent on past only input or all of the input) and (b) limited benefits observed on the challenging MultiDomain data set.  Regarding (a), you work would have been significantly stronger if you would have provided more technical descriptions of changes that you are proposing to ensure that the elements you are using are online \"friendly\". This might have helped you to link what you are proposing to other work done in the past and further strengthen your submission. Regarding (b), it seems that MultiDomain data set is very challenging or not enough tuning was performed to illustrate the benefit of your approach. \n\nClarity: The clarity of this submission suffers from a mostly verbal presentation of very technical operations. \n\nOriginality: This submission offers limited originality. \n\nSignificance: Given experimental results, the significance of this particular submission is minor. \n\nPros: This submission presents what I believe are generally useful techniques but the presentation is verbal rather than mathematical which makes establishing connections significantly harder than it needs to be. At least on one of the data sets the results appear to be good. \n\nCons: Technical elements are described in a very verbal fashion which may lead to misinterpretation. The results on more challenging MultiDomain data set do not make a convincing case that the proposed solution or its tuning is more generally useful. \n\nPost author response stage: Given the response from the authors and the input from other reviewers I increased the score from 4 to 6.", "extra_scores": {}, "extra_text": {}}, {"review_id": "7hnTiMSOh64", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a unified single neural network architecture to realize both streaming and full-context ASR systems. The idea is simple but very efficient. It uses the same model for both streaming and full-context ASR systems, but when we use the streaming mode, some of the network operations that use the future context are ignored (e.g., ignore the future context kernel in CNN, self-attention, or pooling is performed only with the current and history frames). The two modes interact with each other by teacher-student training, where the teacher is the full-context mode while the student is the streaming mode. The paper is well-written overall, but it requires some clarification (see my comments below). \n\nOne of the drawbacks of this approach is that the current technique is too specific to the ASR topic, and it may not get much attention from the general machine learning audience. The authors mention that we can apply the technique to simultaneous machine translation or video processing. However, I'm not sure because the method seems to depend highly on RNN-T structure (assuming the monotonicity). We could not directly use this for simultaneous machine translation. Also, the paper does not have some novel machine learning algorithms, and it would also not gain much attention from the general machine learning audience. Finally, the result looks very impressive, but it requires some clarifications (see my comments below). Especially, I could not fully understand why this method has better latency than the other methods (sometimes becomes negative). Is that related to some search strategy (end-point detection)? I want to have more clarifications about it.\n\nOther detailed comments:\n1. \"Universal ASR\" is not the correct word to represent the proposed method. Please re-consider this name.\n2. The paper claims the state-of-the-art streaming ASR results on the  LibriSpeech task, but I saw some papers (Moritz et al., 2020, Tsunoo et al., 2020) show better performance than this proposed method. Please confirm it and explain more differences. One possible reason is the use of LM. The methods in these papers use an LM, while this paper's method does not seem to use an LM. Please clarify it.\n3. \"Listen, Attend, and Spell\" isa a great study, but the original attention-based encoder-decoder for ASR was proposed by the following papers earlier. The authors should respect these pioneering studies and cite them. \n  - Chorowski, Jan, et al. \"End-to-end continuous speech recognition using attention-based recurrent NN: First results.\" NIPS 2014 Workshop on Deep Learning, December 2014. 2014.\n  - Chorowski, Jan K., et al. \"Attention-based models for speech recognition.\" Advances in neural information processing systems. 2015.\n4. \"Weight Sharing for Multi-tasking.\": How about discussing Watanabe, Shinji, et al. \"Hybrid CTC/attention architecture for end-to-end speech recognition.\" IEEE Journal of Selected Topics in Signal Processing 11.8 (2017): 1240-1253? Although the motivation is different, the methodology is very similar.\n5. Section 3 is well summarized. It well describes dual modes' design by making a clear distinction between the full-context and streaming operations.\n6. How much is more/less training time required when we use the dual-mode than single-mode or separately training both modes?\n7. \"Latency@50 and Latency@90\": I think it's reasonable to exclude the outlier result, but at the same time, this outlier would be more critical for the user satisfaction perspective. Thus, I'm curious about the outlier, and it would be great if the authors discuss the outlier more.\n7. I could not find the detailed machine spec information when computing the latency. Please clarify it.\n7. Section 4.1 has a clear relationship with the discussion in Section 3, and it is easy to follow.\n8. The table should have the unit for the latency. Millisecond?\n9. Compared with the other prior studies (Moritz et al., 2020), the streaming mode's degradation from the full-context mode seems to be larger. Please explain it. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an approach to unifying both full-context and streaming ASR in a single end-to-end model.   Techniques such as weight sharing, joint training and teacher-student knowledge distillation are used to improve the training.  The so-called dual-mode ASR is evaluated under the ContextNet and Conformer networks on Librispeech and MultiDomain datasets. The performance is good.  While the technical novelty is not overwhelmingly significant, all reviewers agree that it may have impact to the speech machine learning community as high-performance streaming ASR is of great importance in real-world deployment of ASR systems.  The authors have meticulously addressed the reviewers' comments and, in particular, changed the title from "universal ASR" to "dual-mode ASR" as suggested by some of the reviewers.  After the rebuttal, all reviewers are supportive on accepting the paper.  
Decision: Accept (Poster)