Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a unified single neural network architecture to realize both streaming and full-context ASR systems. The idea is simple but very efficient. It uses the same model for both streaming and full-context ASR systems, but when we use the streaming mode, some of the network operations that use the future context are ignored (e.g., ignore the future context kernel in CNN, self-attention, or pooling is performed only with the current and history frames). The two modes interact with each other by teacher-student training, where the teacher is the full-context mode while the student is the streaming mode. The paper is well-written overall, but it requires some clarification (see my comments below). 

One of the drawbacks of this approach is that the current technique is too specific to the ASR topic, and it may not get much attention from the general machine learning audience. The authors mention that we can apply the technique to simultaneous machine translation or video processing. However, I'm not sure because the method seems to depend highly on RNN-T structure (assuming the monotonicity). We could not directly use this for simultaneous machine translation. Also, the paper does not have some novel machine learning algorithms, and it would also not gain much attention from the general machine learning audience. Finally, the result looks very impressive, but it requires some clarifications (see my comments below). Especially, I could not fully understand why this method has better latency than the other methods (sometimes becomes negative). Is that related to some search strategy (end-point detection)? I want to have more clarifications about it.

Other detailed comments:
1. "Universal ASR" is not the correct word to represent the proposed method. Please re-consider this name.
2. The paper claims the state-of-the-art streaming ASR results on the  LibriSpeech task, but I saw some papers (Moritz et al., 2020, Tsunoo et al., 2020) show better performance than this proposed method. Please confirm it and explain more differences. One possible reason is the use of LM. The methods in these papers use an LM, while this paper's method does not seem to use an LM. Please clarify it.
3. "Listen, Attend, and Spell" isa a great study, but the original attention-based encoder-decoder for ASR was proposed by the following papers earlier. The authors should respect these pioneering studies and cite them. 
  - Chorowski, Jan, et al. "End-to-end continuous speech recognition using attention-based recurrent NN: First results." NIPS 2014 Workshop on Deep Learning, December 2014. 2014.
  - Chorowski, Jan K., et al. "Attention-based models for speech recognition." Advances in neural information processing systems. 2015.
4. "Weight Sharing for Multi-tasking.": How about discussing Watanabe, Shinji, et al. "Hybrid CTC/attention architecture for end-to-end speech recognition." IEEE Journal of Selected Topics in Signal Processing 11.8 (2017): 1240-1253? Although the motivation is different, the methodology is very similar.
5. Section 3 is well summarized. It well describes dual modes' design by making a clear distinction between the full-context and streaming operations.
6. How much is more/less training time required when we use the dual-mode than single-mode or separately training both modes?
7. "Latency@50 and Latency@90": I think it's reasonable to exclude the outlier result, but at the same time, this outlier would be more critical for the user satisfaction perspective. Thus, I'm curious about the outlier, and it would be great if the authors discuss the outlier more.
7. I could not find the detailed machine spec information when computing the latency. Please clarify it.
7. Section 4.1 has a clear relationship with the discussion in Section 3, and it is easy to follow.
8. The table should have the unit for the latency. Millisecond?
9. Compared with the other prior studies (Moritz et al., 2020), the streaming mode's degradation from the full-context mode seems to be larger. Please explain it. 
review_title: Powerful and practical techniques by unifying streaming and full-context ASR systems

final_score: rating: 7: Good paper, accept
review_content: The paper proposes a pragmatic approach to unifying end-to-end speech-recognition models for whole-utterance and streaming models. Streaming models are defined as not using any future audio, while whole-utterance (or "full context") models can look at the entire audio recording. In short, the paper shows that multi-task training is effective, where each minibatch uses each utterance twice: Once as using the whole-utterance model structure, and once using a modified model structure where all model weights referring to future information are forced to be zero.

The idea is not revolutionary, but it is interesting to see that it works at no loss of accuracy. Interestingly, the results of the proposed multi-task model are actually a little better compared to training distinct models on the two modalities. This is shown to be the case for two end-to-end model architectures, ContextNet and Conformer. The result is of high practical relevance.

I do disagree with the naming "Universal ASR." The word "universal" is, well, quite universal and can refer to many dimensions, such as whole-utterance vs. streaming, distant vs. close talking, wideband vs. narrowband, various domains, various tasks, multi-linguality, etc. While I understand that in today's crowded ML world, one wants to occupy powerful marketing names for ones own work, I strongly believe that they should not be over-stating. For this paper to be acceptable for publication, I believe the authors should tone down the "Universal" moniker, and choose a different name that delineates the dimension more accurately.

Further, I would like to note that although this is supposed to be a double-blind review, the non-blind reference for the internal corpus gives away that this is a Google paper.

**Quality**: The results have been obtained on two quality data sets, one public generally accepted corpus (LibriSpeech), and one Google-internal set of a whopping 413 thousand hours. I consider the results both convincing and of interest to speech-interested members of the ICLR community.

As one feedback point, the paper does not seem to mention a simple and common technique of incorporating at least some future information, by pre-rolling a fixed number of speech frames, e.g. 5, and incorporating those e.g. by stacking or limited-forward looking lower-layer convolution(s) or self-attention. I think it is important that the authors comment on this, since there is still a one-point WER gap due to streaming. To what degree would a simple lookahead like this trade latency for accuracy? Is it possible to build a model that is universal over different numbers of lookahead frames, as to allow for an adaptive lookahead (e.g. pre-roll 0.5 seconds at sentence start, but less later; maybe depending on context such as the words recognized so far)?

As another feedback point, I find the references somewhat Google-heavy. Going through the references from the start, it takes a while to find a non-Google reference. Exaggerated self-referencing that gives the incorrect impression that Google invented everything is not appreciated by all members of the community. I suggest that the authors rebalance, or at least adjust the order. In some cultures, it is considered polite to mention yourself last.

**Clarity**: The paper generally well-written and easily understandable, although I have a few feedback items.

The term "inplace knowledge distillation" is a little confusing. Initially I thought this was done during inference, and was curious how that may be accomplished. However, it is a training-only technique. Please add that to where this introduced ("at training time").

I kindly ask the authors to be more precise in the results table. What is the unit of Latency? I presume ms, but it could also be speech frames. This is not stated, the string "ms" does not occur once. Also, I find it confusing that the table headings for the WER columns are the corpus names, while for latencies, it is the actual meaning of the column ("Latency@X"). So please (1) use WER for these columns, and (2) add units for both, e.g. by saying "WER [%]" and "Latency@50 [ms]". Further, the readability of Tables 3 and 4 could be improved by horizontally aligning the decimal points.

I also kindly ask the authors to have the text proof-read by a native speaker of English, and/or an automated grammar checker which Google certainly has. I noticed a few missing articles.

**Originality**: With the hindsight of having read the paper, the basic idea does not *seem* original at all, but rather straight-forward. The originality of the paper lies in asking the right question, which in this case then naturally leads to the proposed method.

**Significance**: The method allows to reduce the number of models to train by half, which has tremendous practical impact for anyone operating an ASR service. While there is a clear reduction of overhead of model validation and deployment process, it is not clear from the paper whether training the joint model is cheaper than training two individual models (since each utterance is trained twice per minibatch). It would be great if the authors could share that information somewhere.

**Pros**: An interesting paper that provides a simple, well-working solution to a real practical problem, evaluated on a massive real-life corpus.

**Cons**: Some room for improvement of clarity as listed above. Lack of consideration of simple approach of pre-rolling a few frames of speech.

review_title: An effective, practically relevant approach to unifying end-to-end speech-recognition models for whole-utterance and streaming models

final_score: rating: 7: Good paper, accept
review_content: This paper proposes an unified framework for both streaming and non-streaming ASR and the knowledge transfer between them. The results show that both latency and performance are improved. The benefit of training full-context and streaming together are two folds: 1) Current full-context and streaming ASR are trained separately. Since usually the performance of streaming ASR is inferior to the full-context version, the unified training scheme could enforce the model to fit both tasks well, thus could serve as some kind of regularization. 2) The weight sharing proposed in this paper could make it more efficient for deploying both streaming and non-streaming ASR at the same time.

Cons:
1. Though conformer and context achieved SOTA performance as full-context models, they are pretty new and not widely acknowledged and studied compared to Transformer. Adding results of Transformer could better support the claims and make the contribution in this paper more accessible to the community.
2. The source of improvement on latency is not well explained.
3. “It might indicate that weight sharing itself encourages learning better deep representation for streaming ASR.”, this claim should be further validated.

Question and comments:
1. Adding some related work on knowledge distillation could make it a more complete story.
2. Seems the statistics of MultiDomain dataset are not consistent in the paper (413,000 hours vs 163,000 hours)
3. "For fair comparisons, on each dataset we train and report our models and baselines with the same settings (number of training iterations, hyper-parameters, optimizer, regularization, etc.)." Could it be possible different models perform the best with different hyperparameter settings?
4. It could be interesting if the visualization of knowledge transferred from full-context model to streaming model is investigated.
review_title: A study that could benefit the ASR community a lot

final_score: rating: 6: Marginally above acceptance threshold
review_content: This submission proposes a framework for training online and offline ASR models. Experimental results suggest that at least on Librispeech this approach provides tangible benefits for online ASR models. 

Quality: The quality of this submission suffers from (a) mostly verbal presentation (e.g. Figure 1 is a very inefficient way to show that prediction at time t is either dependent on past only input or all of the input) and (b) limited benefits observed on the challenging MultiDomain data set.  Regarding (a), you work would have been significantly stronger if you would have provided more technical descriptions of changes that you are proposing to ensure that the elements you are using are online "friendly". This might have helped you to link what you are proposing to other work done in the past and further strengthen your submission. Regarding (b), it seems that MultiDomain data set is very challenging or not enough tuning was performed to illustrate the benefit of your approach. 

Clarity: The clarity of this submission suffers from a mostly verbal presentation of very technical operations. 

Originality: This submission offers limited originality. 

Significance: Given experimental results, the significance of this particular submission is minor. 

Pros: This submission presents what I believe are generally useful techniques but the presentation is verbal rather than mathematical which makes establishing connections significantly harder than it needs to be. At least on one of the data sets the results appear to be good. 

Cons: Technical elements are described in a very verbal fashion which may lead to misinterpretation. The results on more challenging MultiDomain data set do not make a convincing case that the proposed solution or its tuning is more generally useful. 

Post author response stage: Given the response from the authors and the input from other reviewers I increased the score from 4 to 6.
review_title: Review by AnonReviewer3

Meta-review: This paper proposes an approach to unifying both full-context and streaming ASR in a single end-to-end model.   Techniques such as weight sharing, joint training and teacher-student knowledge distillation are used to improve the training.  The so-called dual-mode ASR is evaluated under the ContextNet and Conformer networks on Librispeech and MultiDomain datasets. The performance is good.  While the technical novelty is not overwhelmingly significant, all reviewers agree that it may have impact to the speech machine learning community as high-performance streaming ASR is of great importance in real-world deployment of ASR systems.  The authors have meticulously addressed the reviewers' comments and, in particular, changed the title from "universal ASR" to "dual-mode ASR" as suggested by some of the reviewers.  After the rebuttal, all reviewers are supportive on accepting the paper.  
Decision: Accept (Poster)