FINE: Future-Aware Inference for Streaming Speech Translation
Authors: Biao Fu
Kai Fan
Minpeng Liao
Zhongqiang Huang
Boxing Chen
Xiaodong Shi
Yidong Chen
Venue: Submitted to ICLR 2023 (2023)
Keywords: Streaming Speech Translation
Future-Aware Inference

Abstract: A popular approach to streaming speech translation is to employ a single offline model together with a \textit{wait-$k$} policy to support different latency requirements. It is a simpler alternative compared to training multiple online models with different latency constraints. However, there is an apparent mismatch in using a model trained with complete utterances on partial streaming speech during online inference. We demonstrate that there is a significant difference between the speech representations extracted at the end of a streaming input and their counterparts at the same positions when the complete utterance is available. Built upon our observation that this problem can be alleviated by introducing a few frames of future speech signals, we propose \textbf{F}uture-aware \textbf{in}ferenc\textbf{e} (FINE) for streaming speech translation with two different methods to make the model aware of the future. The first method FINE-Mask incorporates future context through a trainable masked speech model. The second method FINE-Wait simply waits for more actual future audio frames at the cost of extra latency. Experiments on the MuST-C EnDe, EnEs and EnFr benchmarks show that both methods are effective and can achieve better trade-offs between translation quality and latency than strong baselines, and a hybrid approach combining the two can achieve further improvement. Extensive analyses suggest that our methods can effectively alleviate the aforementioned mismatch problem between offline training and online inference.

Under review as a conference paper at ICLR 2023
FINE: FUTURE-AWARE INFERENCE FOR STREAMING SPEECH TRANSLATION
Anonymous authors Paper under double-blind review
ABSTRACT
A popular approach to streaming speech translation is to employ a single offline model together with a wait-k policy to support different latency requirements. It is a simpler alternative compared to training multiple online models with differ- ent latency constraints. However, there is an apparent mismatch using a model trained with complete utterances partial streaming speech during online infer- ence. We demonstrate that there a sigmificant difference between the speech representations extracted at the end of streaming input and their counterparts at the same positions when the complete utterance is available. Built upon our observation that this problem can be alleviated by introducing few frames of future speech signals, we propose Future-aware inference (FINE) for streaming speech translation with two different methods make the model aware of the future. The first method FINE-Mask incorporates future context through a train- able masked speech model. The second method FINE-Wair simply waits for more actual future audio frames at the cost of extra latency. Experiments on the MuST- C EnDe, EnEs, and EnFr benchmarks show that both methods are effective and can achieve better trade-off's between translation quality and latency than strong baselines, and hybrid approach combining the two can achieve further improve- ment. Extensive analyses suggest that our methods can effectively alleviate the aforementioned mismatch problem between offline training and online inference.
INTRODUCTION
Streaming speech translation (ST) systems consume audio frames incrementally and generate real- time translations, unlike their offline counterparts which have access to the complete utterance before starting to translate. Because of the streaming nature, streaming ST models commonly use uni- directional encoders (Ren et al. 2020 Ma et al. 2220b; Zeng et al. 202T and are trained with some wait-k & policy (Ma et al.] 2019) 9 that determines whether to wait for more speech frames or emit target tokens. In real-world applications, however, it is costly effort to train and maintain multiple models to satisfy different latency requirements (Zhang & Feng. (222). Recently, some works (Papi et al.) 2022 Dong et al. 2022) show that offline models be adapted to streaming scenarios with wail- k policies to meet different latency requirements and achieve comparable or better performance, partially due their use of more powerful bidirectional encoders. However, there an inherent mismatch in using model trained with complete utterances on incomplete streaming speech during online inference (Ma al. 2019).
Intuitively, speech representations extracted from streaming inputs (Figure ((b) are less informative than in the case with full speech encoding (Figure I(a].. Two questions arise naturally: how much is the difference in speech representations between the two inference modes, and is it significant enough to cause problems? We analyze the gap in speech representations, measured by cosine similarity, at different positions in the streaming input compared to using the full speech (Section 3 We find that there is a significantly greater gap for representations closer to the end of a streaming segment, with average similarity score low 0.2 for the last frame, and the gap quickly narrows for frames further away. Moreover, we observe more degradation in translation quality for utterances with the greatest gap in speech representations between online and offline inference.
Based on the above findings, we hypothesize that the lack of future contexts at the end of streaming inputs could be detrimental to streaming speech translation. To this end, we propose novel

---

Under review as a conference paper at ICLR 2023
wailing for more frames
1000 |+
4000
-0000
Training:
Full speech encoding
(b) Testing: Streaming encoding
FINE-Mask
(d)
Figure I: (a) and (b) represent the input mismatch between offline training and streaming testing. (c) and (d) denote the proposed methods FINE-Mask and FINE- FINt-Wait, respectively. "M" in (c) denotes the mask token. Our methods introduce more informative future context to mitigate the mismatch.
Future-aware inference (FINE) strategies for streaming speech translation: FINE-Mask and FINE- Wait, as shown in Figure I(c) and I(d) In FINE-Mask, we append a few mask embeddings to the end of the current streaming speech tokens additional input to the acoustic feature extractor, which based on its masked modeling capability can implicitly estimate and construct future contexts in the corresponding hidden representations and extract more accurate representations for the frames in the streaming imput. Since we find that only the speech representations of the last few positions in the streaming input are severely affected by the mismatch problem, the closest future context could provide the most improvement. Thus in FINE-Wai Wait, we simply wait for few extra speech tokens during streaming encoding and use them as the future context extract improved representations for the frames in the original streaming segment. FINE-WA incurs additional latency as the strategy requires waiting for more oracle future context, but it achieves significant improvement in translation quality and leads to a better trade-off.
We conduct experiments on the MuST-C EnDe, EnEs, and EnFr benchmarks. Experimental results show that methods outperform several strong baselines the trade-off between translation quality and latency. In particular, in the lower latency range (when AL is less than 1000ms), we achieve improvements of 8 BLEU in EnDe, 12 BLEU in EnEs, and 6 BLEU in EnFr. Extensive analyses demonstrate that introducing future context reduces the representation gap between the full speech encoding and the partial streaming encoding.
2 BACKGROUND
Speech translation systems can be roughly categorized into non-streaming (offline) and stream- ing (online) depending the inference mode. Regardless of the inference mode, speech trans- lation models typically employ the encoder-decoder architecture and are trained on an ST corpus (x1,,..., sT) denotes audio sequence, = (y1,-.. yJ) the corresponding source transcription and target translation respectively.
D ((x, y)). where
(zı,...,zr) and
Non-Streaming Speech Translation For the non-streaming ST task, the encoder maps the entire input audio x to the speech representations h, and the decoder generates the j-th target token yj conditional the full representations and the previously generated tokens yej- The decoding process of non-streaming ST is defined as:
ply x) =
II p p x,y.)).
(I)
A significant amount of work has focused non-streaming ST, including pre-training (Wang al.
2020aa Dong et al. 202 Tang et al. 2022 Ao et al. [002)) multi-task learning (Liu et al. 2020 Indurthi et al. 2020 2021 ,dta augmentation (Pino et al. 2019 Di Gangi et al. 2019b Mc
et al. 2020) knowlee ledge distillation (Dong et al. 2021b 2021
Zhao et al. 2021 Du et al. 2022).am and
Prodaitydality representation learning (Tang et al.
Fang al. 2022 Ye al. (0222)
Streaming Speech Translation A streaming ST model generates the j-th target token Yj based streaming audio prefix xzg() and the previous tokens Yęj where g()) is monotonic
2

---

Under review as a conference paper at ICLR 2023
decreasing function representing the ending timestamp of the audio prefix that needs to be consumed to generate the j-th word. The decoding probability is calculated as:
J ply x) II (v xca)))
Thus, a streaming ST model requires a policy to determine whether to wait for more source speech or emit new target tokens. Recent studies (Ma al. 2020b Ren et al. 20.0 Zeng et al. 2021] Dong 2022 on streaming ST make read/write decisions based on a variant of the waitk policy (Ma et al.
al. 2019) that was initially proposed for streaming text translation, which alternates write and read operations after reading the first & source tokens. Because there is no explicit word boundaries streaming audio, several works attempt to detect word boundaries the audio sequence using methods such as fixed length (Ma et al., 2020b), Connectionist Temporal Classification (Ren et al., Papi et al.) 2022). ASR outputs (Chen et al., 2021), and integrate-and 2022). The wair-k policy is applied based detected words rather than audio frames. In other words, g(j) in Eq. represents the length of audio segment corresponding to the first j k 1 detected words in the streaming ST. Moreover, some studies (Arivazhagan et al.,
2020 Zeng et al.
202I;
fire (Dong al.
2020c Zhang et al. |0200 Schneider & Waibel [2020] Miao et al. 2021 Zhang & Zhang al. 2022 Chang & Lee 2022 Liu al. 2021) explore adaptive policies to dynamically decide when to read or write for streaming text and/or streaming speech translation.
2019
Ma et al.
Feng
2022a
Several works (Zhang et al., 2021: Zhang & Feng 2022c) apply knowledge distillation and fill future source positions with positional encoding to inroduuee future information during training for simultaneous machine translation within the prefix-to-prefi framework. In this paper, we focus on a matter less attended to to-too how to alleviate the mismatch between offline training and online inference.
3 PRELIMINARY ANALYSIS
In this section, analyze the major mismatch in Transformer-based (Vaswani al. 2017) ST architecture between offline training and online decoding. In full-sentence ST, the speech represen- tation of each frame is obtained by attending to all unmasked frames by the multi-head attention in the transformer encoder layers. However, if directly applied to the streaming inference with the model trained offline, the speech representation of the current last frame will deteriorate because it can only attend to its previous frames. Recently, a common approach in speech translation is to stack a pre-trained Wav2Vec2.0 (Baevski et al. 2020) as the acoustic encoder with semantic NMT encoder-decoder, and achieves SOTA performance in the ST task (Han et al.) 2021 Dong et al.)
2022 Fang et al. 2022 Ye al. 2022). ) because has been shown that a better speech representa-
tion can be learned via Vev2 (Baevski et al. 2020).
To explore the preeise effects of streaming inputs, we first follow MoSST (Dong al., 2022) train an offline ST model on the MuST-C EnDe training set, where the acoustic encoder trainable. After the offline ST training, we conduct an analysis on the MuST-C EnDe tst-OMMMN set. We remove the outliers and the noisy data, and select audios with duration between 2s and 10s, resulting in a total of 1829 examples.
is
For an input sequence of audio frames x Wav2Vec2.0 0 shrinks the length of the raw audio by a factor 320 and outputs the full speech repre- sentation sequence In other words, every 320 elements in x become vector in a. For readability reasons, we uniformly use the notation 7 to denote the sequence length of a, i.e., a = (aI..... .T)). This simplified notation does not undermine any of our conclusions while at the same time mak- ing the equations for readable") For streaming input Vt < T, = (I),, Wav2Vec2.0 will output the representation å, (ā,,1..... ).,)).
(ei,...,TT) the convolutional subsampler of
3.1 WHICH PART OF STREAMING SPEECH REPRESENTATION IS WORSE?
To measure the gap of the speech representations between the offline and online inputs, calculate the cosine similarity 8t,5' between the speech representation at the 1'tth (t' & t) position in the 1-th
Because we can always define (IIrT) such that J, represents consecutive 320 audio frames.
3

---

Under review as a conference paper at ICLR 2023
Costne Sintkaity
01.8
0.8
0.8
10.6
0.0
0.8
0.4
0.4
0.4
0.2
Stremmin encoding step?
Streaming enooding step
Streaming eneoding
S of the first three positions
St,e/ of middle three positions
(c) Sa,er of the last three positions
Figure 2: The average cosine similarity St.e' of the first three (L' = 1,2,)), middle three (!'
Ltt 1, 1tt 1 1), and last three (t' - 2, 1, 1) positions for each encoding step
30
Offline
Stresmnn
25.69
Cosime Similarit
24.13
28.42
21.91
BLEU
20
19.75
0.6
14.86
Chimera
0.4
12.01
MoSST
10
9.5
0.2
40
30
-20
110
Best
Better
Medium
Worse
Worst
Position (beginning-+onmg
Degree (small->large
(a) Last 40 positions
(b) Degree of deterioration of last representations
Figure 3: (a). The average cosine similarity of the forty representations at the end positions (position= -1 denotes last position) in the streaming speech. (b). Performance with degree of deterioration of the representation at the last position of the streaming speech.
streaming audio input &, and the speech representation the same position in the full encoding. Then we average the cosine similarities over the sentences in dataset B to obtain robust statistics.
For l' L, 81,6
(x,(())
Σ
cos(ar, (e))
B2
B.
XEB.
xEB,
where B, = (x: [x] 2 t} contains the audio inputs with length no shorter than t.
We empirically compare the averaged cosine similarity at the beginning, middle, and end positions of the speech representations. Figure 2 shows št, of the first three (t' = 1, 1,,,3), middle three (t' -t'= [+ 1, 1, 1 L 1+t 1), and last three (t' - 2, - 1, positions for each encoding step At the beginning and middle positions, the averaged cosine similarity št,'' is greater than 0.8 except = 1, indicating that the representations at such positions the partial streaming input are close those in the full speech. Note that L' 1 with slightly lower similarity won't hurt the performance much, because in practice it is almost impossible to apply wait-1 policy in streaming ST. However, the št, declines significantly for the end positions, especially for the last one. In addition, we observe that as f becomes larger, the streaming input will gradually approximate the full speech input, then the gap of the speech representation between the offline and the online input becomes smaller. We conclude that the representations of the end position in the streaming speech are particularly inferior.
We also average the cosine similarity over both dataset and time dimension with reversed index.
__
$1,-141(x)
B+
x
xEB,
We calculate the metric 8 8_, e of the representations at the last 40 positions in the streaming speech for different methods: Chimera (Han al. 2021), STEMM (Fang al. 2022) and MoSST, and
4

---

Under review as a conference paper at ICLR 2023
Tminsaaton
mask
oraele speech token
f
Translation Decoder
Semantic Encoder
Transformer Encoder
Transformet Encoder fo
Word Boundary Detector
waiking for tokens
Acoustic Enceder (Wav2Vec 2.0)
CNN ft
CNN fc
Streaming audio
(a) Offline Model Structure
(b) FINE-Mask
(c) FINE-Wait
Figure 4: Illustration of offline ST model and proposed methods FINE-Mask and FINE-Wait.
report the results in the Figure 3(a) The consistent results verify that the conclusion above holds and decide that the low-quality representations at the last 10 positions cannot be ignored.
3.2 DOES THE POOR REPRESENTATION AT THE LAST POSITIONS OF STREAMING SPEECH AFFECT STREAMING ST PERFORMANCE?
To answer this question, we only calculate the average cosine similarity in the last position for each sample.
VX, 8-1
Σ cos cos (ūt,t, at);
(5)
(x) reflects the degree of deterioration of the representation the last position of the streaming speech. We sort the dataset by the value of the degree and divide them evenly into 5 groups to ensure enough samples in each group. The translation quality of each group is shown in Figure 3(b] The performance of streaming ST drops close to 10 points as the representation at the last position of the streaming speech becomes worse, while the full-sentence ST fluctuates less than points. In addition, the performance gap between the streaming ST and the full-sentence ST becomes larger as the representation at the last position gets worse. In the worse group, the streaming ST is 12.41 points lower than the full-sentence ST. Therefore, we conclude that the poor representation at the end position of the streaming speech has a strong effect on the translation quality.
4 FINE: FUTURE- INFERENCE
Based on these analyses, we find that only necessary for the offline ST model to aware of short future during streaming encoding. Thus, we propose two Future-aware INferencE strategies, FINE-Mask and FINt-Wait, to enhance the representations of streaming speech in Figure
4.1 FINE-MASK
In this strategy, them to the speech tokens generated from the already consumed speech frames. The mask token em- bedding is trainable when pre-training Wave2Vec2.0. Particularly, Wav2vec2.0 applies span masks to the speech tokens and reconstructs 2 the corresponding latent features based on unmasked con- text. By default, the pre-training results in approximately 49% of all time steps being masked with a mean span length of 14.7 (300ms). This pre-training strategy makes the Wav2vec2.0 able to extract better speech representations offline ST task (Han al. 202 Dong et al. 2022 Ye al. 2022).
the mask tokens of Wece2Vec2.0 as the pseudo future context and append
Vec2 consists of multi-layer convolutional subsampler fa and Transformer encoder Se- during online inference, the Sc first outputs
Concretely, for each audio prefix &, = streaming speech tokens é (Ci. C, where è E =x and d is the dimension of model and is the sequence length after convolutional subsampling. Then, we concatenate the streaming speech tokens è and mask token embeddings E Rd along the time dimension, resulting in longer
Strictly speaking, the task is identify the quantized latent audio representation rather than reconstruction.
5

---

Under review as a conference paper at ICLR 2023
Algorithm Pseudocode of FINE-Mask inference strategy in PyTorch-like style.
model: an offline-trainee ST model consists of acoustic encodez Wav2vec2. O, token boundairy dotector, semantic encodar, and decoder mask length, K: wait lagging, audio: aucle wavetorm
mask_enb: cre-trained maak embedding in Rav2vec
tthe number of source text tokens
X - [] ssteemming audio prefix
y transsatioon
eebg baas__mmb.reppee tm, () mask embeddings:
while y] !- *<eoa>";
if == audio: audio has been read
y model (2,y) write new torges token
elit N lyn Ki wait K decected source tokens
x (auctol incremental ly read audic waeell.wvvvee..cm
audio tokens rxd
concatenate audia
emboddings, (r mm) x cat (1c, maak_embs) dim-0)
model www.wwwvwwccccccmmc www.eee2..nccderree audio representations, mr)
al:a.ahape m, :] diacard the predicted representacions, r.x
segiel,token_eeteen
(a): source text boundary detected
claa: model. wwwaemmiceeenaeeteeeeet
y model. decoder th, y) write new target token
sequence speech tokens E ( The new speech tokens are then fed into the Transformer encoder fe, but only the first encoder outputs (i.e., speech features) will be kept for the decoder because, as discussed in Section 3.1 the last m speech features are of poor quality and adversely affect translation quality. The FINk-Mask inference strategy is outlined in Algorithm T
4.2 FINE-WAR WAIT
In the previous section, conclude that the speech representations of only the last few positions in the streaming input are inferior. Therefore, a straightforward method is to discard the poor rep- resentation the end positions. In the FINE- Wait strategy, read more audio frames until the fo can output m extra speech tokens, resulting in new sequence of speech features with length t m. We then discard the last speech features in the FINE-mask strategy above. The FINE-Wair in- ference strategy is outlined as Algorithm 2 jn Appendix. The FINE-W Wait strategy incurs additional latency as the model waits for more actual audio frames during each streaming encoding. However, this strategy results in significant improvement, and motivates us to reconsider the trade-off between translation quality and latency. The detailed analysis is given in Section 5.3
5
EXPERIMENTS
5.1
EXPERIMENTAL SETTINGS
Datasets We evaluate our approach on MuST-C Inglish-German (EnDe), English-Spanie (EnEs) datasets (Di Gangi et al. 2019a). Because limited previous works discussed the MuST-C English- French streaming ST with BLEU-latency curve, we present the EnFr results in Appendix. All the corpora contain source audios, source transcriptions, and target translations, and the results reported are conducted on the corresponding tst- COMMON set. For speech data, we normalize the raw audio wave to the range of [-1, 1). For text data, we keep punctuation and remove non-printing characters, and remain sese-sensi For each translation direction, the unigram Sentence Piece model (Kudo & Richardson 2018 is used to learn shared vocabulary of size 10k.
2022 to train the offline model. For the Model Configuration We follow MoSST (Dngg et al.
acoustic encoder, which use a self-supervised learning framework to pre-train on large-scale audio data from the Lib- riSpeech (Panayotov al. 2015) corpus. We use the continuous integrate-and-fire (CIF) module
Wav2vec2. Baevski et al. 2020) following the base configurations,

---

Under review as a conference paper at ICLR 2023
28
24
20
20
B1.50
16
12
offline (greedy)
offline (gresdy)
SimutSpeech
RealThans
Mo8ST
RealThans
MoSST f
FINE-Wait
FINE-Mask
FINE-
FINE-Mask
8
1,000
2,000
3,000
4,000
5,000
1,000
2,000
3,000
4,000
Average Lagging (ms)
Average Lagging (ms)
(a) EnDe
(b) EnEs
Figure 5: The translation quality (BLEU) against the latency metries (AL) on the Ist-COMMON set of MuST-C EnDe and EnEs dataset. denotes that the results are obtained from corresponding papers. denotes that the results are from our improved MoSST inference method.
28
26
BLEU
offiie (groedy)
offline (gready)
18
Fait-Wair
FINF-Wair
FINE Mask
FINE Mask
22
FINE Hybrid
16
1,000
2,000
3,000
1,000
1,500
2,000
2,500
3,000
3,500
Averuge Lugging (ms)
Average Lagging (ms)
(a) En
(b) EnEs
Figure 6: The translation quality (BLEU) against the latency metrics (AL) on the Ist-COMMON set of MuST-C EnDe and EnEs dataset.
Yi al. 2021 as the word boundary detector. We use 8 and 6 Transformer layers for the semantic encoder and the translation decoder respectively, with 4 attention heads and 768 hidden units.
Inference We use the offline-trained ST model to perform streaming-testing with the wait-k policy. here means & detected source text tokens by the CIF word boundary detector. The length of future context tokens (m) is 50 and 10 for FINE-Mask and Fiit-Wait, respectively. For consistency with previous works (Zeng al., 2021 Dong et al. 2022 we do rewrite the tokens that have already been generated during inference. All hyper- peeerparameters are tuned on EnDe and applied to other language pairs. In all experiments, we use our re-implemented MoSST inference method for a better performance (see Appendix B.2 for a detailed explanation).
Evaluation Metrics We use SacreBI.EU measure the translation quality. The latency evalu-
ated with Average Latency (AL) (Ma et al. 2019), Average Proportion (AP) (Cho & Esipova 2016 and Differentiable Average Lagging (DAL) Cherry Foster 2019) in the SimulEval toolki MMa
ct al. 2020a).
Baselines We compare our method with several strong end-to and-to-end streaming ST approaches. (i) SimulSpeech (Ren et al., 2020) and RealTranS (Zeng et al. (021) uni-directional encoder rather than bidirectional to simulate streaming inputs. (ii) MoSST (Dong et al.] (2022) applies an
offline-trained model with monotonic segmentation module for streaming testing and achieves

---

Under review as a conference paper at ICLR 2023
22
18
14
m
14
m 50
BLEL
10
m 30
m
50
-
m1
40
m 20
30
-
15
2
m0
m
m
500
1,000
1,500
2,000
500
1,000
1,500
Average Lagging((ms)
Average Lagging(ms)
(a) FINE-Waa Wait
(b) FINE-Mask
Figure 7: Effect from different lengths of future context. The observed points of FINE-Wai and FINE-Mask in the plots represent wait-k policy with k = 5,
competitive performance. For fair comparisons, we use MoSST as our baseline model. (iii) MU-ST (Zhag et al. 2022) learns an adaptive segmentation policy to detect meaningful units, which makes read/write decisions.
5.2 MAIN RESULTS
We presents the results on the MuST-C EnDe and EnEs Ist-COMMNN set in Figure 5 Compared with the online models SimulSpeech and RealTranS,, the offline model MoSST achieves higher translation quality with high latency as it encodes bidirectional context information during training. But in the low latency region, performs poorly offline-training and online-decoding. With the ability to reduce this mismatch, FINE-Mask and FINE-Wait achieve higher BLEU in all latency regions. In particular, most compatible base model MoSST by large margins lower latency regions (when AL less than 1000ms), with improvements over 8 BLEU in EnDe and 12 BLEU in EnEs. This indicates that FINE-Mask and FINE-Wait can effectively mitigate the input mismatch between offfine-training and online-decoding. FINE-Wait Wait only requires waiting for a small amount of oracle future context but brings higher translation quality. In addition, our strategies achieve comparable translation quality with full-speech translation at middle latency (at AL around 2000ms), especially for EnEs. Compared with FINE-Wait Wait, FINE-Mask achieves better trade-off as it can introduce more future context without additional latency. Moreover, we find that FINE- Wait and FINE-Mask can be further combined into new strategy FINE- Hybrid. Specifically, given streaming speech with length т, we first wait for m extra oracle speech tokens and add m mask tokens to end of the streaming speech. The total length of the streaming speech tokens will be 7 7 + m'. but like FINE-MEN MaNk and FINE- Wait, only the first 7 encoder outputs will be kept for decoding. We set m
reason for which is the input mismatch between
methods outperform
and m' = 50 for FINE-Hybrid, which are the optimal settings of FINE- and FINE-Mask, respectively. The experimental results are shown in Figure We observe that FINE-Hybrid achieves a better trade-off.
5.3 ABLATION STUDY
In this section, we describe experiments to evaluate the effectiveness our methods from various aspects. All ablation results are obtained from the MuST-C EnDe tst-COMMON set.
5.3.1 How MUCH FUTURE CONTEXT IS NEEDED?
To answer this question, compare FINE-Mask and FaNt-Wait with different lengths of future context m. Figure shows the results. The system that inherits the mismatch problem, i.e. uses the offline model directly for online decoding, is shown by setting m 0. For FINE-Wait, increasing m obtains better translation quality, but generally brings higher latency. Our results in Figure 7(a) shows that it achieves the better trade-off between quality and latency when m {10, 15, 20
If
Thee extended results for other latency metrics (AP and DAL) are described in Appendix B.6
'The results are reported with m 10 for FINE-Wait only corresponds 200ms oracie audio.
8

---

Under review as a conference paper at ICLR 2023
Cosine Similarity
0.8
Cosine Similarity
0.8
0.0
0.6
FINE-Wait
FINE- Wail
FINE Mask
0.2
Mask
0.2
MoSST
6
10
20
30
60
70
30
-2
-10
(a) Future Context Length
(b) Last 40 positions of consumed speech
Figure 8: (a). Effect of future context length on the average cosine similarity St, at the last position of streaming speech. (b). Effect on the average cosine similarity 3-r' of the forty representation at the end position in the streaming speech. After applying FINE-Mask and FINE-Wait, the represen- tation at the end position is improved.
m 20, the large latency becomes prohibitive. Unlike TINE-Wait, FINE-Mask can attend to more future context without waiting for future audio. It achieves the best trade-off between quality and latency at m = 50 (Figure ((b).. Since the predicted context further into the future will likely introduce more noise, FINE-Mask obtains similar performance when is increased from 30 to 70. We also investigate the impact of various future context lengths on the representation of the last position by calculating the average cosine similarity in Eq. (4). The results shown in Figure 8(a) We observe that 1) as m increases, the representation of the last position in the streaming speech becomes better. 2) the curves of the average cosine similarity for Fait-Wait and FINE- Mask becomes flattened when m > 10 and m > 50, respectively. This is consistent with the results Figure Therefore, we future context length 10 and m 50 respectively for FINE-War and FINE-Mask in our other experiments.
5.3.2 WHY DOES FINE WORK?
Figure plott plots the changes average cosine similarity 8 (Eq. (4)) of the last 40 positions the streaming speech after applying the FINE strategies. FINE-Mask and FINE-Wait achieve at least 0.6 and 0.8 cosine similarity at the last position. In MoSST, however, the cosine similarity is less than 0.6 for the last 4 positions and only 0.2 for the last position. Thus, the speech representations with FINE- Wait and FINE-Mask are closer to those of the full speech input, especially at the end positions. The cosine similarities in FINE-Mask are lower than those in FINE-Wair because the predicted future context is less accurate by consuming mask tokens. However, FINE-Mask achieves better balance (Figure (5) as does not incur additional latency. In sum, introducing future context significantly reduces the representation gap between full and partial speech input, improves streaming speech representations, and achieves better balance between quality and latency.
More ablation studies are included in the Appendix. In Appendix B.3 we find the least monotonic examples are mostly improved by our strategies. Appendix [B.4 analyzes the difference of the pre- trained and fine-tuned Wav2Vec2.0 with respect to future context representations. Appendix B.5 demonstrates why all predicted features should be discarded in FINE-Mask.
CONCLUSION
In this paper, we examine streaming speech translation from a new perspective. We investigate the effects of the input mismatch between offline-taining and online-decoding. We find that the representations at the end positions the streaming input are particularly poor, directly impacting the translation quality. We propose FINE-Mask and FINE-Wait to improve these representations by introducing, respectively, predicted and real future context. Experiments and analysis demon- strate their effectiveness in bridging the representation gap between full speech encoding and partial streaming encoding. Furthermore, our strategies can be generally beneficial to streaming speech translation models that are based on Wec22. In the future, we will experiment with other methods to improve the accuracy of predicting future information. We hope the work and perspec- tive presented in this paper can engender further innovations in general streaming translation.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Junyi Ao, Rui Wang. Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, Zhihua Wei, Yao Qian, Jinyu Li, and Furu Wei. SpeechTS: Unified-modal encoder- decoder pre-training for spoken language processing. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5723-5738, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.1 18633/11//2222. acl-long.393, URL
org/ http:22022.acclooooo 393
Naveen Arivazhagan, Colin Cherry, Wolfgang Macherey, Chung-Chenn Chiu, Semih Yavuz, Ruom- ing Pang, Wei Li, and Colin Raffel. Monotonic infinite lookback attention for simultaneous ma- chine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1313-1323, Florence, Italy, July 2019. Association for Computational Linguis- tics. doi: 10.18653/ URL hctps: http://///acaatthhooooooooo///
Alexei Bacvkki, Yuhao Zhou, AbdeIrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural In- formation Processing Systems, volume 33, pp. 12449-12460. Curran Associates, Inc., 2020. URL Inttps://proceddingg weweeeeiiinnnnnnneieeeeeeeeeeeee neurips. cc/paperr/220//1eee http:///wwsrr.rrree../.o//////// wtw.eeebccofffffaaaee
.ddf
Chih-Chiang Chang and Hung-yi Lee. Exploring continuous integrate-and-fire for efficient and adaptive simultaneous speech translation. arXiv preprint arXiv:2204.09595 2022.
Junkun Chen, Mingbo Ma, Renjie Zheng, and Liang Huang. Direct simultaneous speech-to- text translation assisted by synchronized streaming ASR. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. pp. 4618-4624 Online, August 2021. Associ-
ation for Computational Linguistics. doi: 10. 10.1186311/2222 1iin....nng.. aaddings-acl.406
URL https:
7/acl Exclanthology wwwpy/21.gg22020/ firlllffiningg. acl 406
Colin Cherry and George Foster. Thinking slow about latency evaluation for simultaneous machine translation. arXiv preprint arXiv: 1906:9900000 00048, 2019.
Chung-Cheng Chiu and Colin Raffel Monotonic chunkwise attention. In International Confer- ence Learning Representations, 2018. URL httpss:/openneereew.eeeet fermm
[kko85pICM
Kyunghyun Cho and Masha Esipova. Can neural machine translation do simultaneous translation?
arXiv preprint arXiv: 1606. 160600006 2016.
Mattia A. Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi. MuST- C: a Multilingual Speech Translation Corpus. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long and Short Pupers), pp. 2012-2017, 7, Minneapolis, Minnesota, June 2019a. Association for Computational Linguistics. doi: 10. 11853/v1/N19-11111 102.. URL https:
//aclanthology o
Mattia A. Di Gangi, Matteo Negri, Viet Nhat Nguyen, Amirhossein Tebbifakhr, and Marco Turchi. Data augmentation for end-to-end speech translation: FBK@IWSLT '19. In Proceedings of the 16th International Conference on Spoken Language Translation. Hong Kong. November 2-3 2019b. Association for Computational Linguistics. URL. https: http//aaaannoooooooooooooooooooa org/
2019 9. ws 1 1tlllt...11. 14
Qian Dong, Yaoming Zhu, Mingxuan Wang, and Lei Li. Learning when to translate for streaming speech. Proceedings of the 60th Annual Meeting of the Association for Computational Linguis- fics (Volume 1: Long Papers), pp. 680-694, Dublin, Ireland, May 2022. Association for Compu- tational Linguistics. doi: 10. 00.18653/11/022.aad ao0.oog.... URL https: httpss///aalaathllog...
org/2022 /01/2222.ccc1122 -ong..lon.... .50
Qianqian Dong, Mingxuan Wang, Hao Zhou, Shuang Xu, Bo Xu, and Lei Li. Consecutive decoding for speech-to-texx translation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 12738-12748, 2021a.
10

---

Under review as a conference paper at ICLR 2023
Qianqian Dong, Rong Ye, Mingxuan Wang, Hao Zhou, Shuang Xu, Bo Xu, and Lei Li. Listen, understand and translate: Triple supervision decouples end-to-end speech-to-text translation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 12749-12759, 2021b.
Yichao Du, Zhirui Zhang, Weizhi Wang, Boxing Chen, Jun Xie, and Tong Xu. Regularizing end- to-end speech translation with triangular decomposition agreement. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 10590-10598, 2022.
Qingkai Fang, Rong Ye, Lei Li, Yang Feng, and Mingxuan Wang. STEMM: Self-learning with speech-text manifold mixup for speech translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 7050-7062, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 10.11653/w1/20222acl--oom 486.
A. Graves. Sequence transduction with recurrent neural networks. Computer Science, 58(3):235- 242, 2012.
Chi Han, Mingxuan Wang, Heng Ji, and Lei Li. Learning shared semantic space for speech-to-text translation. In Findings of the Association for Computational Linguistics: ACI-IJCNIPP 2021, pp. 2214-2225, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/ v1/2021 findiingiidinsseee 19.. URL 195
wttp:yyy.rgg/20001011 wwtp/////.....innn.eeeeeeeeeeeee acl.
Sathish Indurthi, Houjeung Han, Nikhil Kumar Lakumarapu, Beomseok Lee, Insoo Chung, Sangha Kim, and Chanwoo Kim. End-end speech-to-text translation with modality agnostic meta- learning. In ICASSP 2020-20222 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 7904-7908. IEEE, 2020.
Sathish Indurthi, Mohd Abbas Zaidi, Nikhil Kumar Lakumarapu, Beomseok Lee, Hyojung Han, Seokchan Ahn, Sangha Kim, Chanwoo Kim, and Inchul Hwang. Task aware multi-task learning for speech to text tasks. In ICASSP 2021-202 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7723-7721 IEEE, 2021.
Taku Kudo and John Richardson. SentencePiece: A simple and language independent sub- word tokenizer and detokenizer for neural text processing. Proceedings of the 2018 Con- ference on Empirical Methods in Natural Language Processing: System Demonstrations, pp- 66-71, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10.18653/v//D18-22 2012. URL
Dan Liu, Mengge Du, Xiaoxi Li, Ya Li, and Enhong Chen. Cross attention augmented transducer networks for simultaneous translation. Proceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pp. 39-55, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 00.18653/11/2221.oo org. /tt.or////00//2 2021. www.epmm........
main.
4. URL /RRL:LLtt/////aaatttollll
Yuchen Liu, Jiajun Zhang, Hao Xiong, Long Zhou, Zhongjun He, Hua Wu, Haifeng Wang, and Chengqing Zong. Synchronous speech recognition and speech-to-texx translation with interactive decoding. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 8417- 8424, 2020.
Mingho Ma, Liang Huang, Hao Xiong, Renjie Zheng, Kaibo Liu, Baigong Zheng, Chuanqiang Zhang, Zhongjun He, Hairong Liu, Xing Li, Hua Wu, and Haifeng Wang. STACL: Simultaneous translation with implicit anticipation and controllable latency using prefix-to-prefxx framework. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5025-3036, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/ v1/P19- 1289. URL https: a antho logy. www.ooo...oooo.....oo..o.....oo. 1289
Xutai Ma, Mohammad Javad Dousti, Changhan Wang, Jiatao Gu, and Juan Pino. SIMULEVAL: An evaluation toolkit for simultaneous translation. Proceedings of the 2020 Conference on Empiri- cal Methods in Natural Language Processing: System Demonstrations, pp. 144-150, Online. Oc- tober 2020a. Association for Computational Linguistics. doi: 10.18653/v 1/20200emmlp-demmm..
19. URL https: / ac ant www.thooolly..../22222222.2 emn lmmnll-ddmos.....mm 19

---

Under review as a conference paper at ICLR 2023
Xutai Ma, Juan Pino, and Philipp Koehn. SimulMT SimulST: Adapting simultaneous text trans- lation to end-to-end simultaneous speech translation. In Proceedings of the 1st Conference of the Asia-Pacific Pacific Chapter of the Association for Computational Linguistics and the 10th Inter- national Joint Conference on Natural Language Processing, pp. 582-587, Suzhou, China, De- cember 2020b. Association for Computational Linguisties. URL Sxg/2020..o aacl main.58 58
Xutai Ma, Juan Miguel Pino, James Cross, Liezl Puzon, and Jiatao Gu. Monotonic multihead attention. In International Conference on Learning Representations, 2020c. URL https:// openreview net forum//formeee (ttes/oooommidd-yyyooooo HygBKyS
Arya D McCarthy, Liezl Puzon, and Juan Pino. Skinaugment: Auto-encoding speaker conversions for automatic speech translation. In ICASSP 2020-2020 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP), pp. 7924-7928, IEEE, 2020.
Yishu Miao, Phil Blunsom, and Lucia Specia. A generative framework for simultaneous ma- chine translation. In Proceedings of the 2021 Conference on Empiricul Methods in Natural Language Processing, pp. 6697-6706, Online and Punta Cana, Dominican Republic. Novem- ber 2021. Association for Computational Linguistics. doi: 1/200mmmm// URL
org/ 2021. www.lmmnlpp..... meo-mainn.. 536
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 5206-5210, IEEE, 2015.
Sara Papi, Marco Gaido, Matteo Negri, and Marco Turchi. Does simultaneous speech translation need simultaneous models? arXiv preprint arXiv:2204.03788, 2022.
Juan Pino, Liezl Puzon, Jiatao Gu, Xutai Ma, Arya D. McCarthy, and Deepak Gopinath. Harnessing indirect training data for end-to-end automatic speech translation: Tricks of the trade. In Proceed- ings of the 16th International Conference on Spoken Language Translation, Hong Kong, Novem- ber 2-3 2019. Association for Computational Linguistics. URL https: /ttts::/aaclatto... org/ orq/20133 2019. iwsltwl....
Yi Ren, Jinglin Liu, Xu Tan, Chen Zhang, Tao Qin, Zhou Zhao, and Tie- Liu. Simul- Speech: End-to-end simultaneous speech to text translation. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics. pp. 3787-3796, Online, July 2020. Association for Computational Linguistics. doi: 10.18653//1/2020..a acl-000..cmmmmi..335 URL https://aclaatthin http://hooooyyr///22000//0 wttp:/20.....m.....n....
Felix Schneider and Alexander Waibel. Towards stream translation: Adaptive computatioo time for simultancous machine translation. In Proceedings of the 17th International Conference on Spo- ken Language Translation, pp. 228-236, Online, July 2020. Association for Computational Lin- guistics. doi: 10.18653/v11 11888//11/2200.vv/// 1.28. URI. https: / acl antho 1 wochhlloo.ooooo org//020.. wswl t 28
Yun Tang, Juan Pino, Xian Li, Changhan Wang, and Dmitriy Genzel. Improving speech translation by understanding and learning from the auxiliary text translation task. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4252 4261, Online, August 2021. Association for Computational Linguistics. doi:
1/203//11/222.ac/ ao1-.ong..28.... URL https://acaaathoooyy .rg. wwtp:o/0.//21.1111111 -ong. 328
Yun Tang, Hongyu Gong, Ning Dong, Changhan Wang, Wei-Ning Hsu, Jiatao Gu, Alexei Baevski, Xian Li, Abdelrahman Mohamed, Michael Auli, and Juan Pino. Unified speech-text pre-training for speech translation and recognition. In Proceedings of the 60th Annual Meeting of the Asso- ciation for Computational Linguistics (Volume I: Long Papers), pp. 1488-1499) Dublin, Ireland, May 2022, Association for Computational Linguistics. doi: 10.18853/v wttp://w//22...oo acl- long. 105. URL antho httpa//atnooooooooooo..222222..2 acl long. 105
https: / aat
12

---

Under review as a conference paper at ICLR 2023
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceed- ings of the 31st International Conference Neural Information Processing Systems, pp- 6000-6010, 2017. URL http:s::/pprreeeednggee neurips httpp//wweeeaoo....eeeereeppp/ap
Chengyi Wang, Yu Wu, Shujie Liu, Ming Zhou, and Zhenglu Yang. Curriculum pre-training for end-to-end speech translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 3728 3738, Online, July 2020a. Association for Computational Linguistics. doi: 10. 10.18653/v1/2020..o acl- aal-aaci..3a33 main.344. URL 2020. ma maain... 344
Chengyi Wang, Yu Wu, Liang Lu, Shujie Liu, Jinyu Li, Guoli Ye, and Ming Zhou. Low Latency End-to-End Streaming Speech Recognition with a Scout Network. In Proc. Interspeech 2020, pp. 2112-2116, 2020b. doi: 10.21437/Iterxpeccc.22222 1292.. BRLhhtt::/// //tp:///xx.o doi. org/ 21437/Int (.437/Interspecch. 1220
Rong Ye, Mingxuan Wang, and Lei Li. Cross-modal contrastive learning for speech translation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Compu- tational Linguistics: Human Language Technologies, pp. 5099-5113, Seattle, United States, July 2022. Association for Computational Linguistics. URL 2022. www.aaaacmmminn11..
org/
Cheng Yi, Shiyu Zhou, and Bo Xu. Efficiently fusing pretrained acoustic and linguistic encoders for low-resource speech recognition. IEEE Signal Processing Letters, 28:788-792, 2021.
Xingshan Zeng, Liangyou Li, and Qun Liu. RealTranS: End-to-end simultaneous speech trans- lation with convolutional weighted-shrinking transformer. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 2461-2474, Online, August 2021. Associ-
ation for Computational Linguistics. doi: 10. 10.18653/v1/2222 1/2...////............-.---///- mandings-acl.218
URL https:
lalclannhhlogy.orgrr org/2021. findiniiiii www-gggs-lll. 218
Ruiqing Zhang, Chuanqiang Zhang, Zhongjun He, Hua Wu, and Haifeng Wang. Learning adaptive segmentation policy for simultaneous translation. In Proceedings of the 2020 Conference on Em- pirical Methods in Natural Language Processing (EMNLP), pp. 2280-2289, Online, November 2020. Association for Computational Linguisties. doi: 10. 1tt.33/11122..mmmpppll main, 178. URL wwcpoog..orrg//// 20200 emnlp-man 178
Ruiqing Zhang, Zhongjun He, Hua Wu, and Haifeng Wang. Learning adaptive segmentation pol- icy for end-to-end simultaneous translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 7862-7874, Dublin, Ire- 1on..l.....2...
land, May 2022. Association for Computational Linguisties. doi: 10.
URL https: /sttss:/aalaattoo.... httpethhooooy..or///000
org 2022
542
Shaolei Zhang and Yang Feng. Universal simultaneous machine translation with mixture-of-experrs. wait-k policy. In Proceedings of the 2021 Conference Empirical Methods in Natural Lan- guage Processing, pp. 7306-7317, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.1 18653/11/202211 http://2022..mmmmmmma...5.3. URL
581
Shaolei Zhang and Yang Feng. Gaussian multi-head attention for simultancous machine transla- tion. In Findings of the Association for Computational Linguisties: ACL. 2022, pp. 3019-3030, Dublin, Ireland, May 2022a. Association for Computational Linguistics. doi: 10. 10.18653/v1/2022. findings-acl.233, URL httpt::t////etee /BRLLttpe///aeleenhooooooyyo org/202 wtt.02......niioi ind acl.22... 238
Shaolei Zhang and Yang Feng. Modeling dual read/write paths for simultaneous machine transla- tion. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2461-2477. Dublin, Ireland, May 2022b. Association for Compu- tational Linguistics. doi: 18.18653/v acl.oon.cl-oon11100 URL https /aw.aaanannoooy.....
org/2022.co at2.2..ccll-a.... 1onl. 176
13

---

Under review as a conference paper at ICLR 2023
Shaolei Zhang and Yang Feng. Reducing position bias in simultaneous machine translation with length-aware framework. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6775-6788, Dublin, Ireland, May 2022c. Association for Computational Linguistics. doi: 10. 10.18663/1/20222.ddloong.g...
URL.
logy. www.ogg..ooooooooooo 2022 atl. 467
Shaolei Zhang, Yang Feng, and Liangyou Li. Future-guided incremental transformer for simultane- ous translation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp- 14428-14436, 2021.
Jiawei Zhao, Wei Luo, Boxing Chen, and Andrew Gilman. Mutual-learning improves end-to-end speech translation. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pp. 3989-3994, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguisties. doi: 10. 18653/11/202/ http:///02..mmmmmma....... 325.. URL emnlp-ma in. 325
14

---

Under review as a conference paper at ICLR 2023
Algorithm 2 Pseudocode of FINE-Wait inference strategy in PyTorch-like style.
medel: an offline-trainee ST model consists of acoustic encodez Wav2vec2. O, token boundary datector, semantic encodar, and decoder mask length, Kt wait lagging, audios auclo wavetorm
N 0 the number of source
tokens
x [] streaming aucio prerix
y - [] Eranslations
while y[-1] !- <<oos>";
audio: audio been read
elif y momeel (a, 1 write new target token
wait detected source tokens x (aucioo incrementally read
(ee
model.wavzver ceeec22.nnnk (X) & audio tokens c. <= waiting for aucio takens
audio representations, m) shap.shape[[] m, dissard
represantations, 7
1f model .toeen.toennnnee brokke_ddttsttorii (a): source text taken boundary is detected
else:
madel
paadel.scmantiie www.sscmanicccc. (n
y model decoder th, y) write
A ALGORITHM OF FINE-WAID
The pseudo code FINE-War are described in Algorithm
B ADDITIONAL EXPERIMENTS
B. WIYY WE USE AL RATHER THAAN k??
In our presented results, we plot the BLEU v.s. AL rather than k. We argue that is not fair metric to evaluate the latency. In text streaming translation, different tokenization (e.g., different number of BPE operations) will lead to different token boundaries for the same sentence. It indicates the k tokens do not necessarily represent the same partial sentence for different BPE methods. This situation becomes even severer for speech streaming translation. As we have a source text token boundary detector in our model, the first k detected text tokens will represent different lengths of audio frames for different input audios. To be precise, the wait-k policy used in our streaming speech translation is actually wait-k detected tokens policy. Therefore, we prefer use AL rather than & as the latency metric in our experiments.
B.2 WHY WB USED OUR IMPLEMENTED MOSST?
For streaming speech translation, when the al- lowed AL is increasing, the performance of streaming ST will gradually converge the of- fline model. However, in the original imple- mentation of MoSST, there is still nonnegli- gible gap between the SST and offline ST for the large AL. We re-implement the MoSST in- ference method and build our inference strate- gies on top of the improved MoSST inference. The difference of the inference performance be- tween the original MoSST and our improved MoSST can be seen from Figure 9 We can see the inference BLEU of our improved MoSST approximate the offline model as the AL in- creases. The detailed implementation can refer the submitted code.
24
29
BLEU
16
offline (groody)
imporvad-MoST
M
1,000
2,000
3,000
4,000
Average Lagging (ms)
Figure 9: proved MoSST.
Difference between original and im-
15

---

Under review as a conference paper at ICLR 2023
Table 1: Performance (BLEU) on different monotonic levels test of MuST-C EnDe.
Monotonic Level
Easy
Medium
Hard
AL
# example
668
1013
895
Offline (greedy)
29.36
23.32
22.38
MoSST
24.29
15.32
9.22
1295
FINE- INNt-Wait
26.61 +2.32
19.36+1.00
17.24 17.24+8.02
1251
FINE-Mask
26.03 + 1.74
19.35 +-.03
17.40 48.18
1143
(189)
(114
(105:
(649)
(145)
(203)
(71
1910)
(450)
(060)
(1545)
(283)
(565)
(700)
(a) EnDe
EnEs
(c) EnFr
Figure 10: The source-to-target alignment position shift MuST-C EnDe, EnEs, and EnFr tst- COMMON set.
B.3 WHAT EXAMPLES ARE IMPROVED BY OUR STRATEGIES?
For Ist-COMMON on MuST-C EnDe, we use fast-aligr 9 to identify the token-level alignment be- tween source transcription and target translation following /Zhang & Feng (2022c). First, we define the source-to-target alignment position shift as i where the ith source token is aligned to the jth target token. If i j is large, means in order translate the jth target token, the model may need to read more until seeing the ith source token. Then we calculate the monotonic level of each example as the averaged alignment position shift over the number of aligned tokens, i.e.,
movotonic.level aaligned-pain
Σ
max(0,
(6)
(t.f)ealigned.pairs
We divide the test set into groups according to different monotonic levels: easy (= 0). medium (< 3) and hard (2 3). For each group, we evaluate different inference methods and the results in Table As explained in B.I it is almost impossible guarantee the same AL for different inference methods. For fair comparison, we try our best to set the AL of different methods to be approximately equal. We can our inference strategies show significant advantage on the non-monotonic examples (hard group).
B.4 How IMPORTANT OF THE WAV2VEC2.0?
As we mentioned in the main text, the special audio token "mask" in Wav2Vec2.0 is pre-trained on the Librispeech dataset to reconstruct the corresponding feature conditional on unmasked context via the contrastive task. our experiments, we didn't include contrastive learning as the auxiliary task in the downstream ST training. And in FINE-Mask inference, directly leverage the mask embeddings as the future context by appending them to the streaming input. However, we found the speech representations after ST training becomes even better. Particularly, we calculate the cosine similarity between every predicted future representation and full speech representations the same position, and the results illustrated in Figure TT On either the Librispeech the MuST-C audio test set, the fine-tuned Wav2Vec2.0 can produce better speech representations from the masking inputs.
https: www..hh......lll.. fast_align
16

---

Under review as a conference paper at ICLR 2023
0.8
Innetund-w2222 pretrained-whh
Ninetuned-w222
wetrained-ww222
Cosine Similarity
Cosine Similacity
0.0
0.2
0.2
10
0
10
30
40
10
10
20
30
Distance from consumed speech
Distance from consumned speech
(a) MuST-C
LibriSpeech
Figure 11:
We measure the accuracy of predicted context by calculating the cosine similarity be- tween every predicted future representation and full speech representations at the same position.
B.5 WHY ARE ALL PREDICTED FEATURES DISCARDED?
In FINE-Mask strategy, all the output represen- tations corresponding to the m 50 mask- ing tokens will discarded, because we have demonstrated that the representations at the ending positions are inferior. However, as shown in II] the first 10 predicted represen- tations are not as bad as the next 40. There- fore, the EnDE test set, also conduct another streaming ST inference by appending different numbers of predicted context the original speech representations. We use dis- card rate p to measure the number append- ing features. When = 1.0, all predicted fea- are discarded and it reduces the stan- dard FINF-Mask inference. In Figure [2. we compare the streaming speech translation qual- ity between regular LINE-Mask and its variant. is concluded that the predicted future context is too noisy and harmful to the performance.
20
BLEU
4
1,000
2,000
3,000
Average Lagging(ms)
Figure 12: BLEU AL different p.
B.6 ADDITIONAL RESULTS ON ENDE/ES AND ENFR
In this section, we evaluate our inference meth- ods with other latency metrics AP and DAL.. The AP-BLEU and DAL-BLEU curves on the MuST-C EnDe, EnEs, and EnFr IsI-COMMON sets are shown in Figure 14 For both EnDe and EnEs, our proposed interence strategies can consistently improve the baseline by large margin.
30
24
BLEU
18
offline (greedy)
12
MoSST
EnFr In general, we found very limited pre- vious works evaluating on the MuST-C EnFr dataset with BLEU-latency curves. We apply our strategies on the EnFr datasets on different latency metrics and present the results in Fig- ure 13 and Figure 14(e) and 14(f)) The FINE- Mask strategy
FINE Wair
6
FINE-Mack
1,000
2,000
3,000
4,000
Average Lagging (ms)
achieve significant improve- ment. Note that the y-axis of EnFr has a larger measuring scale than EnDe EnEs, the gap
Figure 13: BLEU v.s. AL the tst-COMMON set of MuST-C EnFr dataset.
17

---

Under review as a conference paper at ICLR 2023
24
20
offiine (greedy)
offline (grecdy)
12
t
FINE-Mack
FINE-Mask
0.3
0.6
0.7
0.8
0.9
1,000
2,000
3,000
4,000
5,000
Average Proportion
Differentiable Average Lagging
(a) EnDe
(h) En
28
28
24
24
20
20
offiine (groody)
offline (groddy)
McSST
McSST
8
16
4
0.6
0.7
0.8
0.9
1,000
2,000
3,000
4,000
Average Proportion
Differentiable Lagging
(c) EnEs
(d) EnEs
BLEU
18
18
offline (gready)
offline (gready)
Ma8ST
12
MoSST
FINE: Wait
FINE Wait
FINE-Mask
FINE Mask
0.1
0.6
1,000
2,000
3,000
4,000
Average Proportion
Differentiable Avenge Lagging
(e) EnFr
(I) EnFr
Figure 14: The translation quality (BLEU) against the latency metries (AP, DAL) the tst- COMMON set of MuST-C EnDe, EnEs and EnFr dataset. & denotes that the results are from our improved MOSST.
is not visible the other two language pairs, e.g., the absolute BLEU gain AL (1000ms) from MoSST to FINE-Mask about 6 BLEUs. We observe that FINE-Wait does not bring as signifi- cant improvement over the baseline improved MOSST. We also count the distributions of monotonic levels of EnFr test set as shown in Figure [0(c) and find the majority (94.8%) the averaged po- sition shift is 3. Most of the EnFr examples fall into the easy and medium groups According to the finding in Table [ should common that there less significant difference between our strategy and the baseline MoSST.
18

---

Under review as a conference paper at ICLR 2023
B.7 How Do FINE-TUNING EPOCHS AFFECT THE FINE-MASK STRATEGY?
For FINE-Mask strategy, we use a mask token that well-trained during pre-training of the Wav2Vec 2.0 model to predict future speech features. Now we investigate the performance of different fine- tuning steps on FINE-Mask strategy. We evalu- ate the FINE-Mask inference and regular stream- ing inference (MoSST) at different checkpoints with epoch e = {23, 25, 27, 29,31] after train- ing converges, as illustrated in Figure 15 Al- though different checkpoints achieve different per- formances, FINE-MAN Mask consistently brings im- provement (about 6 BLEU at low latency regimes) the baseline. Moreover, we can observe that the performance gap between different checkpoints in MoSST is larger than that in FINE-Mask. Specif- ically, when AL E [1000, 1500], the performance gap between the best and the worst checkpoints in MoSST is 3 BLEU, while the gap in FINE-Mass is only 2 BLEU. All these indicate that the number of fine-tuning steps is not sensitive to the performance improvement of FINE-Mask.
MoSSI 23)
FINE-Mask ie 25)
MuSST = 25)
FINE- URE-MMikic ie 27)
MuSST = 27)
FINE TINE-Misk (e: 29)
MoSST (e 29)
FINE-Misk (e 31)
MoSST (E 31)
18
14
12
10
8
1,000
1,500
2,000
2,500
3,000
Averuge Lagging (ms)
Performance of different fine- Figure 15:
tuning epochs on FINE-Mask strategy.
B.8 FUTURE WORK: UPPER BOUNDER OF FINE-MASK
If we infer with FINE-Wair without counting the latency, the result will be the upper bound of the FINE-Mask, as illustrated in Figure [6 We can observe that the translation quality is further improved all latency regimes when the accuracy of the predicted contexts becomes fully correct. In particular, the translation qual- ity remains 16 BLEU at a very low latency regime (AL about only 200mss) exceeding the FINE- Mask by about 10 BLEU. Thus it mo- tivates us to predict the future contexts or rep- resentations as accurate as possible and we will explore this direction in the future work.
20
BLEU
16
12
(greedy)
(cracle)
50 (mark)
200
500
1,000
1,500
2,000
2,500
Average Lagging (mns)
Figure 16: Upper Bound of FINE-Mask.
C DIFFERENCES FROM STREAMING SPEECH RECOGNITION TECHNIQUES
We briefly describe the differences between work and other streaming speech processing tech- niques for other readers from other backgrounds to understand our work. Most streaming recognition techniques still fall into the category to train the model in a streaming manner, c.g., MoChA (Chiu & Raffel 2018). Similarly, CAAT (Liu et al. 2021) follows the RNN-T (Graves) 2012) method to build streaming speech translation model. However, what are researching completely differ- ent from these streaming recognition techniques. We want to reuse the pre- vee-triine offline ST model for streaming inference without further streaming- lem for two reasons. First, the streaming inference takes the partial speech sequence as input, while the offline model always consumes the full speech sequence. Second, the reordering issue prevents partial speech sequence from seeing future information. Our proposed method attempts relieve such mismatches via the FINE-Mask strategy.
training. In general, this is difficult prob-
Moreover, the word boundary detector in our model is similar to that in SN model (Wang et al. 20206) However, there are two significant differences. First, in our pseudo word detector, we use CIF module. For detector architecture, SN and CIF are almost identical with linear layer and the sigmoid activation. The only difference is the label used. The label for CIF is the total number of words in the transcription. During inference, the CIF module will segment the speech sequence
19

---

Under review as a conference paper at ICLR 2023
into several possibly overlapped segments, whose number should be roughly equal to the number of words if the model is well-trained. However, for SN, it uses actual word-frames alignment in- formation the labels to learn the real word boundary. So CIF is almost unsupervised or weakly supervised learning, while SN is supervised learning. Second, our methods are mainly used to adapt single pre-trained offline ST' model to streaming inference without further streaming training, and alleviate the input mismatch between offline training and streaming inference. However, for SN, it trained conditioned the previous states. It indicates SN is greammnn-trraied model with unidirectional encoder.
In summary, our method is different from traditional streaming techniques.
D WHY MOSST AS A BASE MODEL
MoSST (Dong al. 2022 consists of acoustic encoder, monotonic segmentation module (i.e., CIF module), semantic encoder, and a translation decoder. First, the acoustic encoder is pre-trained Wev2Oec2.0. The mask speech embedding is well-trained and is helpful for our FINE-Mask strategy. Moreover, the CIF module can be trained to roughly segment the speech sequence into words level with very weak signal (the total number of words). After training, the number of resulted segments is roughly equal to the number of words. So it can be used as a pseudo word detector for streaming inference. Thus, we use MoSST as our base model.
Details of Offline Training The offline ST model is first trained for about 20 epochs by multi-task learning, including ASR and ST tasks. A language identity tag is prepended to the target sentence for indicating which task is learned. In this stage, the CIF module which is used for monotonic segmentation deactivated, in other words, the CIF module not trained. The main purpose is to leamn better decoder, i.e., well-trained language model. Then, activate the CIF module such that its parameters are trainable, and continue to train for another several epochs. In this stage, only the ST task is learned. Usually, it will only take 6 epochs converge.
E NUMERIC RESULTS FOR THE FIGURES
Table 2: Numeric results for Figure 5
13
EnDe
EnEs
EnFr
Model
AL
AP
DAL
BLEU
AP
DAL
BLEU
AL
AP
DAL
BLEU
867
0.51
1032
7.79
802
0.38
981
3.54
743
0.4
866
4.59
1295
0.65
1531
13.31
1327
0.64
1606
16.99
1376
0.66
1576
21.07
1939
0.78
2234
18.72
1832
0.76
2189
23.30
1886
0.78
2150
28.59
MoSST
2505
0.85
2788
20.67
2571
0.86
2944
26.30
2532
0.87
2798
31.47
3312
0.92
3559
22.33
3188
0.91
3543
27.40
3083
0.91
3341
32.54
4410
0.97
4576
23.16
4047
0.95
4327
27.58
3811
0.95
4017
33.08
0.63
1223
16.10
0.57
1123
17.15
593
1143
0.70
1559
19.19
1101
0.68
1499
21.15
1009
0.59
1205
17.88
1534
0.76
1928
21.15
1532
0.77
1954
25.43
1499
0.72
1724
26.69
FINE-Mask
2109
0.83
2476
22.23
1990
0.82
2404
27.24
1937
0.8
2200
30.89
2647
0.88
2974
23.15
2647
0.88
3035
27.45
2554
0.87
2821
32.78
3404
0.93
3678
23.65
3228
0.92
3568
27.44
3087
0.92
3335
33.41
4457
0.97
4625
23.42
4017
0.96
4287
27.60
3797
0.95
3996
33.70
1045
0.68
1455
18.43
965
0.65
1368
20.40
841
0.49
957
10.07
1453
0.75
1849
20.39
1409
0.74
1808
24.15
1390
0.68
1558
22.93
1857
0.81
2227
21.43
1895
0.81
2291
26.45
1892
0.79
2116
28.83
FINE-Wai
2414
0.86
2750
22.45
2357
0.86
2734
27.19
2314
0.85
2563
31.26
2926
0.90
3220
22.91
2975
0.91
3323
27.49
2884
0.9
3119
32.70
3628
0.94
3875
23.33
3516
0.93
3815
27.57
3374
0.93
3594
33.20
4590
0.98
4737
23.39
4232
0.96
4466
27.95
4017
0.96
4197
33.57
20

---

Under review as a conference paper at ICLR 2023
Table 3: Numeric results for Figure 5
The results of MU-ST obtained from (Zhang al, The results of SimulSpeech and RealTrans are obtained from (Zeng et al. 2021).
2022
MU-ST
AL
1023
1424
1953
2642
3621
4453
5089
5754
BLEU
17.94
20.85
22.78
24.3
24.82
24.99
25.05
25.9
EnDe
RealTrans
AL
1355
1838
2290
2720
3106
BLEU
16.54
18.49
19.84
20.05
20.41
FINE Ayb--d
AL 1056
1396
1778
2328
2840
3557
BLEU
18.66
20.94
22.26
22.76
23.29
23.60
SimulSpeech
AL 694
1336
2169
2724
3331
BLEU
15.02
19.92
21.58
22.42
22.49
EnEs RealTrans
AL 1047
1554
2043
2514
2920
BLEU
18.54
22.74
24.89
25.54
25.97
FINE-Hybrid
AL
1108
1468
1779
2234 27.35
2854
3399
BLEU
22.53
24.14
26.62
27.37
27.70
Table 4: Numeric results for Figure 7
FINE-Mask
FINE- Wait
m
AL
BLEU
m
AL
BLEU
77
AL
BLEU
m
AL
BLEU
178
0.10
442
12.38
178
0.10
882
17.00
483
1.68
30
785
16.06
0
483
1.68
20
1286
19.79
867
7.79
1146
18.83
867
7.79
1686
21.22
198
1.66
475
12.65
484
10.88
1098
17.99
577
5.98
50
796
16.10
5
891
16.19
30
1504
20.18
977
12.58
1143
19.19
1295
18.87
1890
21.59
324
6.39
473
12.65
641
14.59
1316
18.06
10
706
11.29
60
811
16.23
10
1045
18.43
1710
20.48
1088
16.61
1152
19.09
1453
20.39
2082
21.81
409
10.99
494
12.33
768
16.11
1511
18.62
20
783
14.61
70
801
16.28
15
1166
19.31
50
1893
20.92
1144
18.36
1148
18.77
1575
20.67
2253
21.95
We also provide the numeric results for Figures and for Figures and 6 in Table 4
and in Tables
for Figure
Table
21


Reviews:
[{"review_id": "ylXEtI4wTs", "reviewer": "Reviewer_2wmu", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper authors present their work on analyzing speech representation mismatch between non-streaming and streaming setting, for speech translation. Based on it, authors propose to leverage future-aware inference (FINE) (including FINE-Mask and FINE-Wait) for mitigation, and experimental results show the effectiveness of the proposed approach.", "summary_of_the_review": "This paper clearly presents authors' work to improve streaming speech translation, including mismatch analysis, the proposed FINE-Mask and FINE-Wait approaches, experimental design and ablation studies etc. While there are good thoughts and experiments put in it, I find the proposed approach is mostly based on existing techniques or standard practice. I'd also suggest authors consider addressing the following questions:\n\n1. For the results shown in Figure 5, do authors have comments on model size and computation cost, when comparing the proposed methods vs. baselines?\n2. On page 7, it's said \"The length of future context tokens (m) is 50 and 10 for FINE-Mask and FINE-Wait, respectively.\" How to select the value of m, and why is it different for both?\n3. Do authors think when training is conducted in a streaming manner (e.g. similar as FINE-Mask and FINE-Wait), accuracy could be further improved with streaming inference? If so, is there a way to train speech translation models using streaming inputs, and generalize it for different streaming inference configurations?\n4. In the last section of conclusion, it's said \"In the future, we will experiment with other methods to improve the accuracy of predicting future information.\" This is not informative. It would help to list examples for methods/directions authors think worth exploring, if applicable.", "strength_and_weaknesses": "Strength: The proposed methodology is technically sound. Overall, this paper clearly presents the proposed approach, with well designed analysis and experiments presented in both main paper and appendix.\n\nWeakness: The proposed FINE-Mask and FINE-Wait are mostly based on existing technics, with limited novelty. For example, FINE-Wait appears to be a standard practice to incorporate future contexts. Also I think the experimental section may need clarification on several items (see questions in the summary section below).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "s8kuRRrdfZ", "reviewer": "Reviewer_JSnA", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates the streaming property in ST and propose new streaming methods.\nThe two proposed methods: FINE-mask and FINE-wait are proven to be effective and achieve better trade-offs between translation quality and latency.\nFurthermore, these methods can effectively alleviate the mismatch problem between offline training and online inference.", "summary_of_the_review": "This paper is motived by an observation that representations at the end position are usually very different from the full input case.\nThe authors try to solve this problem by proposing two different methods and good results are obtained.", "strength_and_weaknesses": "Strength:\nThe paper propose two different kinds of method and good results are obtained.\n\n\nWeaknesses:\nAlthough the investigation may draw a lot of interest, the conclusion is not surprising.\nIn 3.1, the investigation shows that the representations at the end position has a higher gap with the full input case. This is not surprising as   frames near the boundary has no future context comparing with the full input case.\nBy the way, if the end position is actually the end position of the whole sentence, the representations should have no difference with the full input case.\n\nI also think the authors should compare their methods with the idea in \"Low latency end-to-end streaming speech recognition with a scout network\"\n\nThe FINE-wait method proposed by the paper is not new.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "lSpKWi7PAx", "reviewer": "Reviewer_uAxq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses the differences between the training and inference conditions of streaming models. They first present a thorough analysis of the problem by finding correlation between representations during offline and streaming modes and plot that against the model performance in terms of BLEU score. They show that the deterioration happens from the last representations of the input. \n\nIn order to fix this problem they propose a simple strategy during inference to match the conditions that wav2vec 2.0 based models are trained towards. They propose two strategies FINE-Wait and FINE-Mask and also discuss a combination of it called FINE-Hybrid. These techniques essentially create a pseudo longer context using mask tokens (matching the wav2vec 2.0 training criterion) or using actual speech tokens for better contextualization and then dropping it during inference. \n\nOverall, the technical contribution is low but the idea is neat and simple with a good motivational analysis. The drawback is that it's quite constrained towards models trained on wav2vec 2.0 criterion. ", "summary_of_the_review": "I think the paper is well motivated, simple yet strong, if the authors are able to comment on the weaknesses I pointed out, this paper can be quite insightful. ", "strength_and_weaknesses": "**Strengths**\n1) Strong motivation with supported analysis for the motivation and a simple solution towards this problem. \n2) The paper is well written and clear to understand. \n\n**Weaknesses**\n1) It seems like the approach is limited to wav2vec type models from its motivation. But in my opinion this approach should work for any kind of models, specially the FINE-Wait. A study or discussion on this would have made the paper stronger. \n2) The paper is also missing details/analysis on how does number of fine-tuning steps affects the FINE-Mask strategy. If the model is reliant on the MASK tokens that are used during self-supervised pre-training I would be curious to learn if amount of fine-tuning causes a change in the performance.\n3) Similarly, the approach could have been stronger if the authors also studied how these model training could be modified to match the inference conditions. For example, maybe providing future biased masking during fine-tuning would allow the models to match the inference conditions. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XmNKv5-mtd", "reviewer": "Reviewer_3NNH", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper investigates the difference is speech representations towards end for streaming input vs when complete utterance input is available. Based on this investigation the work proposes future aware streaming ST which introduces few frames of future speech signals. Extensive experiments verify the improvements compared to other recent works. ", "summary_of_the_review": "Overall the work is novel and interesting, well supported by experiments and analysis. Hence my high rating.", "strength_and_weaknesses": "### Strengths\n- Both FINE mask and FINE wait methods are simple modifications for streaming speech translation\n- FINE Mask is a clever strategy to make the output representations similar to offline full context speech and streaming speech.\n- Results compared to MoSST baseline are quite significant in the low latency region.\n- Paper is well written and supports the hypothesis with extensive experiments and analysis.\n\n\n### Questions\n- FINE-wait approach is not entirely novel, and is also mostly known as a latency/performance trade-off, is there anything different that I missed?\n- As I understand the authors reimplement the MOSST baseline. From Fig 5 RealTrans and MU-ST perform better on low latency region. Can FINE mask and FINE wait extended to other existing methods that perform better in the low latency region?\n- Recently Conformer architecture has shown improvements over transformer for speech related tasks. Did the authors consider trying better encoder layers like conformer as well as better pretraining methods for the encoder like w2vBERT? Will that make applying FINE-mask approach difficult?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "-SPBq5Q4IK", "reviewer": "Reviewer_xgfh", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new algorithm for streaming speech translation by improving the wait k policy-based approach with a mask trick. The paper first starts with the streaming speech translation issue caused by the end-position speech feature representations with various analyses. This analysis naturally motivates their proposed method of using mask tokens or extra speech tokens to avoid dealing with unstable end-position speech feature representations. The experiment is based on a public speech translation corpus (MustC). It shows the effectiveness of the proposed method based on the latency and translation performance trade-off with several ablation studies. ", "summary_of_the_review": "Although the paper is clearly written with its rationale for solving the end-position issue by simple speech token augmentations, I feel that the technical novelty of this paper is not significant due to the above reasons.  \n\nSeveral other suggestions and questions:\n- How about applying this technique to steaming ASR? This will prove the generalization of this work and would attract more machine learning researchers\n- I thought that $t'=t$ in Fig.2 (c) has the highest cosine similarity compared with the other two examples since this streaming condition is the same as the offline case (I think I misunderstood something). Can you explain it?\n", "strength_and_weaknesses": "Strength\n- Streaming speech translation becomes an essential technique for conversational AI.\n- Clear analysis of the streaming issue in speech translation by focusing on the end position. I like their formulation of the w2v2 output representation for streaming and offline based on two-time indexes $a _{t, t}$ (but it requires a bit more explanation). This makes it easy to understand the feature representation in the end position.\n- FINE-mask/wait are simple but powerful\n- The paper is well written overall.\n\nWeaknesses\n- Although this would be a strength, I think the method is a bit too simple and does not have as much algorithmic/theoretical novelty as a machine learning conference.\n- I want to have more relationships with streaming speech recognition techniques. I know that some methods (e.g., RNN-T) cannot be straightforwardly used for speech translation due to the reordering issues, but there are several attention-based streaming ASR methods (e.g., monotonic attention-based approaches like mocha https://arxiv.org/abs/1712.05382). \n- It is better to have more challenging language pairs with more serious reordering issues, e.g., En-Ja is recently supported in Must-C.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject