Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper addresses the differences between the training and inference conditions of streaming models. They first present a thorough analysis of the problem by finding correlation between representations during offline and streaming modes and plot that against the model performance in terms of BLEU score. They show that the deterioration happens from the last representations of the input. 

In order to fix this problem they propose a simple strategy during inference to match the conditions that wav2vec 2.0 based models are trained towards. They propose two strategies FINE-Wait and FINE-Mask and also discuss a combination of it called FINE-Hybrid. These techniques essentially create a pseudo longer context using mask tokens (matching the wav2vec 2.0 training criterion) or using actual speech tokens for better contextualization and then dropping it during inference. 

Overall, the technical contribution is low but the idea is neat and simple with a good motivational analysis. The drawback is that it's quite constrained towards models trained on wav2vec 2.0 criterion. 
strength_and_weaknesses: **Strengths**
1) Strong motivation with supported analysis for the motivation and a simple solution towards this problem. 
2) The paper is well written and clear to understand. 

**Weaknesses**
1) It seems like the approach is limited to wav2vec type models from its motivation. But in my opinion this approach should work for any kind of models, specially the FINE-Wait. A study or discussion on this would have made the paper stronger. 
2) The paper is also missing details/analysis on how does number of fine-tuning steps affects the FINE-Mask strategy. If the model is reliant on the MASK tokens that are used during self-supervised pre-training I would be curious to learn if amount of fine-tuning causes a change in the performance.
3) Similarly, the approach could have been stronger if the authors also studied how these model training could be modified to match the inference conditions. For example, maybe providing future biased masking during fine-tuning would allow the models to match the inference conditions. 
clarity,_quality,_novelty_and_reproducibility: 1) **Clarity and Quality** : Very Clear and Well Presented 
2) **Novelty** : Low, yet a simple and effective approach
3) **Reproducibility** : Low, no details on the model was trained

summary_of_the_review: I think the paper is well motivated, simple yet strong, if the authors are able to comment on the weaknesses I pointed out, this paper can be quite insightful.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new algorithm for streaming speech translation by improving the wait k policy-based approach with a mask trick. The paper first starts with the streaming speech translation issue caused by the end-position speech feature representations with various analyses. This analysis naturally motivates their proposed method of using mask tokens or extra speech tokens to avoid dealing with unstable end-position speech feature representations. The experiment is based on a public speech translation corpus (MustC). It shows the effectiveness of the proposed method based on the latency and translation performance trade-off with several ablation studies. 
strength_and_weaknesses: Strength
- Streaming speech translation becomes an essential technique for conversational AI.
- Clear analysis of the streaming issue in speech translation by focusing on the end position. I like their formulation of the w2v2 output representation for streaming and offline based on two-time indexes $a _{t, t}$ (but it requires a bit more explanation). This makes it easy to understand the feature representation in the end position.
- FINE-mask/wait are simple but powerful
- The paper is well written overall.

Weaknesses
- Although this would be a strength, I think the method is a bit too simple and does not have as much algorithmic/theoretical novelty as a machine learning conference.
- I want to have more relationships with streaming speech recognition techniques. I know that some methods (e.g., RNN-T) cannot be straightforwardly used for speech translation due to the reordering issues, but there are several attention-based streaming ASR methods (e.g., monotonic attention-based approaches like mocha https://arxiv.org/abs/1712.05382). 
- It is better to have more challenging language pairs with more serious reordering issues, e.g., En-Ja is recently supported in Must-C.
clarity,_quality,_novelty_and_reproducibility: Clarity
- The paper is written in general. 
- Since I have more ASR backgrounds (and many other people interested in this work would be the same backgrounds), I think it is better to mention that this paper deals with an attention-based method and briefly describes other streaming speech processing based on RNN-T and CTC in the beginning.
- The end of the introduction discusses performance improvement, but since it does not clearly mention the baseline, I could understand the significance of this improvement. It is better to briefly explain the baselines for the readers to understand the importance of this method.
- MoSST seems to be an important base model of paper, but it does not explain it clearly. Also, it is better to explain why the paper adopts MoSST as a base model.
- Section 3, 3rd paragraph suddenly uses "a factor 320," which is confusing. I expect the paper to assume 16kHz sampling and 320 samples (corresponds to 20 ms frameshift) in W2V2. It is better to clarify it.

Quality
- This work has a solid analysis to motivate the proposed method. This study's quality is sufficient with the proposed method's apparent effectiveness.

Novelty
- I'm not entirely convinced of this method in terms of the significant novelty for the machine learning community since the method attaches special tokens or waiting schemes. They are very logical based on their analysis, but the techniques are not significantly novel.
- If there are more technical novelty, solid theoretical background, etc., for the FINE-Mask/FINE-wait, I would recommend the authors spend more space explaining them in more detail.

Reproducibility
- The database and evaluation metrics are publically available. The reproducibility is high.
summary_of_the_review: Although the paper is clearly written with its rationale for solving the end-position issue by simple speech token augmentations, I feel that the technical novelty of this paper is not significant due to the above reasons.  

Several other suggestions and questions:
- How about applying this technique to steaming ASR? This will prove the generalization of this work and would attract more machine learning researchers
- I thought that $t'=t$ in Fig.2 (c) has the highest cosine similarity compared with the other two examples since this streaming condition is the same as the offline case (I think I misunderstood something). Can you explain it?
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper authors present their work on analyzing speech representation mismatch between non-streaming and streaming setting, for speech translation. Based on it, authors propose to leverage future-aware inference (FINE) (including FINE-Mask and FINE-Wait) for mitigation, and experimental results show the effectiveness of the proposed approach.
strength_and_weaknesses: Strength: The proposed methodology is technically sound. Overall, this paper clearly presents the proposed approach, with well designed analysis and experiments presented in both main paper and appendix.

Weakness: The proposed FINE-Mask and FINE-Wait are mostly based on existing technics, with limited novelty. For example, FINE-Wait appears to be a standard practice to incorporate future contexts. Also I think the experimental section may need clarification on several items (see questions in the summary section below).
clarity,_quality,_novelty_and_reproducibility: This paper is clearly written overall, and it is reproducible. It has limited originality.
summary_of_the_review: This paper clearly presents authors' work to improve streaming speech translation, including mismatch analysis, the proposed FINE-Mask and FINE-Wait approaches, experimental design and ablation studies etc. While there are good thoughts and experiments put in it, I find the proposed approach is mostly based on existing techniques or standard practice. I'd also suggest authors consider addressing the following questions:

1. For the results shown in Figure 5, do authors have comments on model size and computation cost, when comparing the proposed methods vs. baselines?
2. On page 7, it's said "The length of future context tokens (m) is 50 and 10 for FINE-Mask and FINE-Wait, respectively." How to select the value of m, and why is it different for both?
3. Do authors think when training is conducted in a streaming manner (e.g. similar as FINE-Mask and FINE-Wait), accuracy could be further improved with streaming inference? If so, is there a way to train speech translation models using streaming inputs, and generalize it for different streaming inference configurations?
4. In the last section of conclusion, it's said "In the future, we will experiment with other methods to improve the accuracy of predicting future information." This is not informative. It would help to list examples for methods/directions authors think worth exploring, if applicable.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper investigates the difference is speech representations towards end for streaming input vs when complete utterance input is available. Based on this investigation the work proposes future aware streaming ST which introduces few frames of future speech signals. Extensive experiments verify the improvements compared to other recent works. 
strength_and_weaknesses: ### Strengths
- Both FINE mask and FINE wait methods are simple modifications for streaming speech translation
- FINE Mask is a clever strategy to make the output representations similar to offline full context speech and streaming speech.
- Results compared to MoSST baseline are quite significant in the low latency region.
- Paper is well written and supports the hypothesis with extensive experiments and analysis.


### Questions
- FINE-wait approach is not entirely novel, and is also mostly known as a latency/performance trade-off, is there anything different that I missed?
- As I understand the authors reimplement the MOSST baseline. From Fig 5 RealTrans and MU-ST perform better on low latency region. Can FINE mask and FINE wait extended to other existing methods that perform better in the low latency region?
- Recently Conformer architecture has shown improvements over transformer for speech related tasks. Did the authors consider trying better encoder layers like conformer as well as better pretraining methods for the encoder like w2vBERT? Will that make applying FINE-mask approach difficult?

clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and detailed. The experiments are thorough and detailed and the ideas are simple and novel. Code and work should be reproducible. 
summary_of_the_review: Overall the work is novel and interesting, well supported by experiments and analysis. Hence my high rating.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper investigates the streaming property in ST and propose new streaming methods.
The two proposed methods: FINE-mask and FINE-wait are proven to be effective and achieve better trade-offs between translation quality and latency.
Furthermore, these methods can effectively alleviate the mismatch problem between offline training and online inference.
strength_and_weaknesses: Strength:
The paper propose two different kinds of method and good results are obtained.


Weaknesses:
Although the investigation may draw a lot of interest, the conclusion is not surprising.
In 3.1, the investigation shows that the representations at the end position has a higher gap with the full input case. This is not surprising as   frames near the boundary has no future context comparing with the full input case.
By the way, if the end position is actually the end position of the whole sentence, the representations should have no difference with the full input case.

I also think the authors should compare their methods with the idea in "Low latency end-to-end streaming speech recognition with a scout network"

The FINE-wait method proposed by the paper is not new.
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is clearly written.

Quality: Standard.

Novelty: Marginal. The novelty is mainly contributed by the FINE-mask method. For the FINE-wait method, it is not new and is just slightly modified from existing wait-k method.

Reproducibility: The codes are provided in the paper and others can replicate the results.


summary_of_the_review: This paper is motived by an observation that representations at the end position are usually very different from the full input case.
The authors try to solve this problem by proposing two different methods and good results are obtained.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a new algorithm for streaming speech translation model by utilizing the information from future speech frames. The algorithm starts from an offline pre-trained wav2vec 2.0 model, and uses FINE-Mask and FINE-Wait to improve the translation quality. The major contribution of this paper is to show the importance of future speech frames with wav2vec 2.0 as a baseline offline model.  All reviewers believe this is a very well written paper. The analysis part is also very informative. 

However, the novelty is the biggest concern from reviewers. Utilizing future frames is not new in streaming speech tasks. There are lots of works in the speech recognition area using future frames with streaming Transformers. Here are some examples
[1] Zhang, Qian, et al. "Transformer transducer: A streamable speech recognition model with transformer encoders and rnn-t loss." ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020.
[2] Shi, Yangyang, et al. "Emformer: Efficient memory transformer based acoustic model for low latency streaming speech recognition." ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021.
[3] Kim, Kwangyoun, et al. "Multi-mode Transformer Transducer with Stochastic Future Context." arXiv preprint arXiv:2106.09760 (2021).
[4] Mahadeokar, Jay, et al. "Flexi-Transducer: Optimizing Latency, Accuracy and Compute for Multi-Domain On-Device Scenarios." in Proc. Interspeech, 2021.

The authors argued that the traditional methods can only have fixed N or fixed K in stride-N or stride-K to match training and testing. However, methods in [3][4] can have arbitrary N during inference for the same model. There are also streaming speech translation works that don’t have fixed policy by leveraging RNN-T, such as CAAT or 
[5] Xue, Jian, et al. "Large-Scale Streaming End-to-End Speech Translation with Neural Transducers." in Proc. Interspeech, 2022.
In [5], future speech frame is also used to improve speech translation quality. 
Look-ahead is also used in some related works mentioned by the authors:
[6] Chen, Junkun, et al. "Direct Simultaneous Speech-to-Text Translation Assisted by Synchronized Streaming ASR." Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. 2021.
[7] Zeng, Xingshan, et al. "RealTranS: End-to-End Simultaneous Speech Translation with Convolutional Weighted-Shrinking Transformer." Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. 2021.
As a conclusion, utilizing future speech frames is a natural choice for streaming speech recognition and translation tasks. Therefore, the novelty is really limited.

The FINE-Mask is a very interesting idea that applies to wav2vec 2.0, which uses masked tokens. However, this may not be applicable to other pretraining methods, like standard ASR pretraining. Ideally, the proposed method should not be restricted to only one pretraining method in order to have larger impact. 

For a paper published in ICLR, ideally the paper should have larger audience. This paper focuses too specifically for speech. If the paper is submitted to ICASSP or Interspeech, it may attract more attention, especially if the authors can strengthen the novelty by generalizing the proposed method not only on the pre-trained wav2vec 2.0 but also to other popular pre-trained speech models without using masked tokens.

justification_for_why_not_higher_score: As noted in the summary, the proposed future-aware method is a widely used method in speech processing, for both automatic speech recognition and speech translation. Although the FINE-MASK method has its novelty, it is not generalized to other pretraining models. Furthermore, the proposed method is too speech-specific, not generalized to other areas.
Decision: Reject