Training Simultaneous Speech Translation with Robust and Random Wait-k-Tokens Strategy
Authors: Linlin Zhang
Kai Fan
Jiajun Bu
Zhongqiang Huang
Venue: EMNLP 2023 Main (2023)
Keywords: Simultaneous Speech Translation
Robust and Random Wait-k
Cross-modal alignment

Abstract: Simultaneous Speech Translation (SimulST) is a task focused on ensuring high-quality translation of speech in low-latency situations. 
Despite this, the modality gap (\emph{e.g.}, unknown word boundaries) between audio and text presents a challenge. 
This gap hinders the effective application of policies from simultaneous text translation (SimulMT) and compromises the performance of offline speech translation. 
To address this issue, we first leverage the Montreal Forced Aligner (MFA) and utilize audio transcription pairs in pre-training the acoustic encoder, and introduce a token-level cross-modal alignment that allows the wait-$k$ policy from SimulMT to better adapt to SimulST. 
This token-level boundary alignment simplifies the decision-making process for predicting read/write actions, as if the decoder were directly processing text tokens. 
Subsequently, to optimize the SimulST task, we propose a robust and random wait-$k$-tokens strategy. 
This strategy allows a single model to meet various latency requirements and minimizes error accumulation of boundary alignment during inference. 
Our experiments on the MuST-C dataset show that our method achieves better trade-off between translation quality and latency.

Training Simultaneous Speech Translation with Robust and Random Wait-k-Tokens Strategy
Linlin Zhang Kai Fan ,Jijuun Bu', Zhongqiang Huang
'Zhejiang University, Alibbaa DAMO Academy
(zhanglinlinlin, bij)ezuu. edu. (z. huang, k. faaann,k.ann)aallle
com
Abstract
There was
motorcade
back
there.
# word
Simultaneous Speech Translation (SimulST) is task focused on ensuring high-quality translation of speech in low-latency situa- tions. Despite this, the modality gap (e.g., un- known word boundaries) between audio and text presents a challenge. This gap hinders the effective application policies from simulta- neous text translation (SimulMT) and compro- mises the performance of offline speech transla- tion. To address this issue, first leverage the Montreal Forced Aligner (MFA) and utilize au- acous-
MFA
Hoppy
There was
motor
cade
back
there.
subword
Boundary Loss
E a
1
12,
EE mel
Prediction
Hopth
src src_len len 5 and 3 by sampling
Hinter
dio transcription pairs in pre-training tic encoder, and introduce token-level cross- modal alignment that allows the wait-k pol- icy from SimulMT to better adapt SimulST. This token-level boundary alignment simpli- process for predicting
Figure I: An example of our proposed robust and ran- dom wait-k-token strategy. #word 0 indicates that MFA aligns chunk of consecutive frames to no word. Og the CIF weight (details refer Sec. 3.1).
fies the decision-makee read/write actions, if the decoder di- rectly processing text tokens. Subsequently, to optimize the SimulST task, we propose ro- bust and random wait-k-tokens strategy. This strategy allows a single model to meet various latency requirements and minimizes error accu- mulation of boundary alignment during infer- ence. Our experiments on the MuST-C dataset show that our method achieves better trade- off between translation quality and latency.
reduce the discrepancy between training and infer- ence. An efficient wait-k: method was proposed by (Elbayad et al., 2020) to train multiple paths by ran- domly sampling i values, instead of training multi- ple models w.r.f. different It values. The progress in SimulMT has greatly influenced advancements in SimuIST. In particular, transformer-based end- to-end neural architectures have achieved perfor- mance levels comparable to cascade systems in both offline and simultancous ST tasks (Vila et al., 2018; Bentivogli et al., 2021; Fang et al., 2022a; Ren et al., 2020; Liu et al., 2021; Anastasopoulos et al., 2022).
1 Introduction
Simultaneous Speech Translation (SimulST) is task designed to generate real-time translations by incrementally consuming audio frames. A com- mon practice is cascade streaming Automatic Speech Recognition (ASR) system and a Simulta- neous Text Machine Translation (SimulMT) model (Oda et al., 2014; Dalvi et al., 2018). The latter has been significantly improved earlier, as evidenced by (Gu et al., 2017). Then, the prefix to to-prefix (P2P) framework (such as the wait-k policy) (Ma et al., 30; Le et al., 2020) has been developed to
Typically, an end-to-end SimulST model re- quires a high-quality speech translation model that can take as input partial audio and a decision pol- icy for controlling read/write actions. One sim- ple but effective approach is to borrow ideas from SimulMT. However, applying SimulMT strategies directly to SimulST poses challenge because speech signals continuous, while text tokens are discrete and have inherent meanings. There- fore, the strategies of SimulMT, which defines in- put granularity at the token level, cannot be directly applied to SimulST. Integration of a segmentation
Equal contribution. Work was done during Linlin Zhang's 's internship at DAMO Academy, Alibaba Group. Corresponding author.

---

module into SimulST has become common prac- tice, as it serves as prerequisite for resembling SimulMT policies. Notable examples include the connectionist temporal classification (CTC)-based approach (Ren et al., 2020; Yang et al., 2023) and the fixed-sized chunk method (Ma et al., 2021). However, CTC was not designed for simultane- ous scenarios and the fixed-sized chunk method lacks flexibility in representing semantic meanings. The Continuous Integrate- aantte--nn-FFrree (CIF) method (Dong and Xu, 2020), which is ideally suited for strcaming input, has recently attracted more atten- area (Dong et al., 2022a; Shu et al., 2023; Fu et al., 2023). Our work builds upon the research line of the CIF module.
(3) We carried out experiments across multiple language directions on the MuST-C dataset. Our simultaneous translation model achieved better trade-off between translation quality and various latency requirements with a single model, in both restricted and unrestricted scenarios.
2 Related Works
Speech translation can be roughly classified into offline and simultancous scenarios, with our dis- cussion primarily focusing on end-to-end models. Offline Speech Translation generates each tar- get token based on the full audio representation. Following the advent of the first end-to-end neu- ral network model for ST (Berard et al., 2016), significant portion of end-to-end ST works em- phasized offline scenarios. Weiss et al. (2017); Berard et al. (2018); Bansal et al. (2019); Aline- jad and Sarkar (2020); Dong et al. (2021) demon- strated the effectiveness of pre-training in improv- ing performance. Consequently, this has become the standard paradigm for current end-to-end mod- els. Other prevalent research areas include multi- task learning (Bahar et al., 2019; Liu et al., 2020; Indurthi et al., 2020; Han et al., 2021; Ye et al., 2021; Zhang et al., 2023), knowledge distillation (Liu al.. 2019; Gaido et al., 2020; Wang et al.. 2023), and curriculum learning (Kano et al., 2018;
tion in the streaming speech-to-ter
We propose fine-grained CIF module as the traditional CIF lacks clear and interpretable train- ing supervision, as shown in Figure 1. We use the Montreal Forced Aligner (MFA) (McAuliffe et al., 2017) to acquire frame-word or -phoneme alignments, and use them to pre-train an acoustic encoder with a fine-grained CIF loss. By aligning speech and transcription, we can readily adapt the read and write policy of SimulMT to SimulST.
An issue similar to the exposure bias (Ranzato et al., 2016), whereby the prediction error of seg- mentation may cause a gap between training and in- ference, rarely discussed in previous works. We propose a robust-alignment training strategy to min- imize this gap. Moreover, previous SimulST works based on Wav2Vec2.0 (Baevski et al., 2020) of- ten overlook its bidirectional attention, which isn't inherently streaming fashion. We discuss this by suggesting a random wait-k policy, making model compatible with a bidirectional encoder. Our robust strategy solely involves transccranniiptioon alignment and independent of the read/write pol- icy, thus making it feasible to combine with other aduptive policies. Our main contributions can be summarized as follows.
Wang et al., 2020).
Simultaneous Speech Translation was first re- alized through pipelined system featuring stream- ing ASR and SimulMT model. ASR played a nat- ural role in segmenting speech frames and aligning with transcriptions, albeit at the cost of increased la- tency. Recently, end-to-end SimulST has attracted more research attention. Thus, a SimulST model requires a policy that determines whether to wait for more speech frames generate new target to- kens. Previous works often used fixed size speech chunk (Ma al., 2020b; Nguyen et al., 2021; Ma et al., 2021; Liu et al., 2021).
(1) We designed a mooss-modal pre-training method for the acoustic encoder, based on our fine-grained CIF module. This approach allows for the segmen- tation and alignment of speech with text represen- tation in both offline and online scenarios.
Adaptive size policy has been thoroughly ex- plored in SimulMT (Arivazhagan et al., 2019; Ma et al., 2020a; Zhang et al., 2020). As to SimulST, the works of (Dong et al., 2022a; Zhang et al., 2022) most relevant to research. The for- mer first applied CIF SimulST, and the latter determined the read/write policy by learning segment audio frames into meaningful units. Our model achieves tross-modie token-level alignment through pre-training a novel fine- mne-grained CIF mod-
(2) We enhanced the robustness and introduced more randomness to the wait-A policy for real-time speech translation. Our robust-alignment training strategy can effectively minimize the gap between training and inference.

---

ule, thereby facilitating downstream SimulST. Con- sidering the frame length of each segmentation, our model also triggers an adaptive policy.
in Figure 1. Because Wav2 involves 320x
subsampling, the alignment between acoustic kens a and the word-level transcription is also available. We can express the acoustic segment sequence as,
3 Main Method
End-to-end simultaneous speech translation mod- els typically employ a sequence-to-sequene learn- ing framework built on an encoder-decoder ar- chitecture. For a standard ST training corpus, D - {ss, x,y) represents triplet dataset that includes audio, transcription, and translation se- quences. The SimulST model is generally defined probabilistic model:
(a1, a2,..., aT) (ab)e))) Abw:ew) (2)
where W is the number of words in x, and bw ew indicate the beginning and ending indices that align to the w-th word. Note that x is usually repre- sented as a subword sequence (I1,..., FI. accord- ing to the NMT tokenizer, so L * W in general. For our running example, b4, 64 are aligned to the word "motorcade", whereas "motorcade" in is represented subwords 24 "motor" and 2; "cade". Nevertheless, can easily determine how many subwords each segment corresponds and denote as (4,,..;ww) satisfying
p(y|s; ss; 0) P(wslssson
where 0 is the model parameters, and g()) monotonically non-decreasing function that indi- cates the ending timestamp of the audio required to generate the j-th target token.
>w' w- This quantity will serve as the fine-grane -ne-ggained supervision for our proposed CIF.
In our work, the aim of the pre-training stage is to embed audio features into the text represen- tation space (i.e., source embedding space) and to identify the candidate ending timestamp for the function g(j). The fine-tuning stage directly op- timizes the SimulST model as per Eq. (1). To achieve these objectives, we initially utilize the transcription-tranne
Following the traditional CIF, we obtain the CIF weights â (á1, âT) by inputting the acoustic tokens into a linear layer with sigmoid activation. Contrasting with the conventional loss in CIF, we utilize labels from forced alignment and propose a novel fine-grained CIF loss, defined as follows.
pairs in 'D to pre-train an offline machine translation model (NMT). In prac- tice, additional MT corpus is more readily avail- able, allowing us to pre-train the NMT using even larger data sets. The overall model architecture, comprising both stages, is displayed in Figure 2.
Loif
L
+
Σ
à
L
(3)
The first term our proposed fine-grained subword- level boundary loss, where the learning signal cor- responds to the number of subwords within a given timestamp interval. The second term is the original CIF loss that learns the subword boundary under a weak signal - the total number of subwords.
3.1 Semantic Alignment Pre-Training
In this stage, the primary objective is to achieve cross-modal semantic alignment between speech and text. We utilize only the mudio-transcription pairs (s, )) to train the parameters of the acoustic encoder, while temporarily discarding the NMT encoder and decoder.
3.1.2 Semantic Alignment Supervision
During early training that Eq. (3) does not con- verge, the & is ill-defined for boundary inference. typically necessitating length normalization.
3.1.1 Fine-grained CIF Supervision
L Σ 6
(4)
Similar to recent SimulST works with adaptive size
policy (Dong et al., 2022a; Zhang et al., 2022), we adopt Wav2Vec2.0 (Bacvski et al., 2020) the speech feature extractor. For input audio
With this update, fulfills the condition Σ, aq L. During inference, normalization is not required, as the value of is unknown. However, it is the- oretically reasonable, as the loss Lar in Eq. (3) converges to O, Σ, &r will also converge to the underlying L.
s, we denote the output features of Wav2Vec2.0 as acoustic tokens a (a1, .2,,..., aT).
We also utilize the Montreal Forced Aligner (MFA) to achieve word alignment, as shown

---

Hinter mir ...
Linear Layer (CE)
NMT Decoder
threshold based
weighted average
Linear (Emb)
NMT Encoder
CIF weights
NMT Source Embedcling
Linear Layer
3 x Transformer Layers
acoustic
takens
K, V
If a, + d2 < and a, + 22 + 2
Acoustic
9 = ==:a +0 (1-a; -
Layer
Encoder
acoustic tokens
ag - 41 +3 -
Compute starting with (az. (3))
Wav2Vec2.0
5
X There was no motor cade
back
there.
The overall model architecture of the proposed approach. The NMT is firstly pre-trained. the pre- training stage for the acoustic encoder, the NMT encoder and decoder is temporarily discarded. In the fine-tuning stage for SimulMT task, the model is trained end-to-end. In the left panel, an example illustrates the threshold based weighted average.
Figure 2:
Once we have the normalized weights, we pro- cess the acoustic tokens using threshold-based weighted averaging from left to right, suitable for the simultaneous scenario. An intuitive exam- ple featuring 0122,3 is depicted in the left panel of Figure. (for more details, refer to Appendix A.1). During training, the averaged acoustic to- kens should possess the same subword length as the transcription x, denoted as c (cI..... CL).
Cross Cross-Entropy (CE) loss.
Lemb = || Emb(x)ll2
(6)
Lee
CE (Linearce(z)) m4)
(7)
worth noting that the mixup (Chen et al., 2021; Fang et al., 2022b) between z and Emb(x) is ran- domly applied when calculating the CE loss. Thus, our final pre-training loss is the aggregation of Eq. (3,6,7),
Next, we incorporate several Transformer layers for semantic alignment learning. In contrast to the traditional multi-hcad attention (MHA) of the encoder, our approach is designed as follows.
-pt = Leif + A Lemb + Lec- (8)
where a tunable hyper-parament balance the scale of mean squared loss Lemb- Note that the embedding matrix is frozen, and the linear layer for CE will be discarded for ST inference.
is multiplied
h; MHA(Q = h K a, = a).
(5)
3.2 Robust and Random Wait-k-tokens
where hg is initialized as The output length for each layer will always be the same that of c.
After the previous stage, both the acoustic encoder and the NMT model are well pre-trained. The out- put of the acoustic encoder is expected to align with the source embedding space in terms of both semantic meaning and sequence length. During the fine-tuning stage, we will use the full ST data, de- noted as D, to train the end-to-end SimulST model.
For supervisory learning, propose two seman- tic objectives through two linear layers, as shown in Figure 2. The first linear layer maps h to the source embedding space of NMT via Linearembl (h). The corresponding loss aims align the acoustic features with subword embeddings. The second linear layer further maps the output z of the acous- tic encoder to the discrete vocabulary, employing
3.2.1 Random Wait-k
Our proposal draws inspiration from the efficient wait-k strategy (Elbayad et al., 2020), originally

---

designed for SimulMT with a unidirectional en- coder. However, the raw speech feature extrac- tor, Wav2Vec2.0, is a bidirectional encoder. Prior SimulMT studies (Dong et al., 2022a; Zhang et al., 2022) directly utilized it and overlooked the dis- crepancy between offline and streaming input. We propose robust and random training strategy to address this issue.
tation based on the MFA, otherwise, the sampled length from the latter distribution is already at the subword-level and the training uses the predicted segmentation based on a.
3.2.3 Overall Loss
The overall loss of the fine-tuning stage is defined to optimize all model parameters.
In partieular, we first sample source word-level length w from a uniform distribution (1, W), and sample & ( K2) as the number of wait- ing timestamps, where K1 < K2. According to the MFA alignment and word-level length w, we can readily derive the subword-level source length 1 C [1, I.]. The target length then becomes 4 11 k + 1. In our case, K1 could be negative number as long 4 does not exceed the transla- tion sentence length. Then, we propose the random wait-k-tokens loss.
Ln = Lof C + Cff + L + Sp
(10)
where Lort is the offline translation loss conditional on full speech transcription, and C is defined via p(u, X.M. Vet).
Experiments
4.1 Experimental Setting
4.1.1 Dataset
For a fair comparison with previous works, we conduct our experiments on the widely used MuST-C V1: (nglish--GGermann French, Span- ish) (En > (De, Fr, Es)) (Gangi et al., 2019). For our multitask ST model with auxiliary MT data, extract 20M En-Fr and En-Es 15M sentence pairs from the WMT14 and WMT13 data. For En- De, we use all of the WMT14 dataset. The data statistics are shown in Table 1.
C E, log Zgt. Io (9)
where S represents sampled set of source lengths. To improve efficiency, we use the same k for each example, but resample & across examples in batch. Note that (i) our approach permits bidirectional at- tention on the partial input, as fix is essentially optimizing the sampled prefix pairs in the wait-k decoding path; (ii) However, we also encounter the unresolved issue from (Dong et al., 2022a; Zhang et al., 2022): during inference, the encoder hidden states of Wav2Vec2.0 cannot be cached for stream- ing input. We will discuss potential solutions to this problem as future work in the limitations section.
corpus
ST(Hrs/Sentt)
Optional MT(Sents)
En-De
408/234K
4.5M(WMT14)
En-Fr
492/280K
20M(WMT14
En-Es
504/270K
15M(WMT13)
Table
The statistics for the three language pairs.
Following the preprocessing recipes of STEMM (Fang et al., 2022b), we use MFA (McAuliffe et al., 2017) to process ASR data to obtain speech-text alignment information. We also filter out the raw audio larger than 450000 frames. The text vocab- ulary consists of 10,000 subword units learned by SentencePiece (Kudo, 2018), shared between the source and target languages.
3.2.2 Robust Wait-k
As previously mentioned in Eq. (4), unless the loss Car is strictly equal to 0, the inferred will be suboptimal, and a discrepancy will exist between i = Σ ā and the actual transcription length. To address this issue, we propose robust training strategy. Essentially, by emulating the scheduled sampling (Bengio et al., 2015) that can mitigate exposure bias, we use the predicted alignment for the loss Eq. (9) with probability (e.g., 0.5).
4.1.2 Model Configuration and Evaluation In all experiments, our semantic alignment module comprises 3 transformer layers. The Neural Ma- chine Translation (NMT) model employs 6 trans- former encoder and decoder layers. Both have hidden dimension of 512 and utilize eight attention heads. We pre-train two variants of NMT models: one using the MuST-C data exclusively, and an- other incorporating an additional MT corpus. Dur- ing SimulST training, we sample k between and
Specifically, when sampling the source length in Cst. we randomly select the sampling rule from the following two methods.
1. Sample ~ (1, W', then derive 4;;
2. Directly sample 1, ~ (1, i.).
the former distribution is selected, it implics that the training can utilize the ground-truth segmen-

---

28
24
341
24
McSST
30
20
81.80
Realirans
McSST
20
RealTras
16
MU-ST1
22
Ous
16
12
MU
Ours
Uus"
Oun
offinne (greddy)
ofline (gready)
12
Ours
offlnne (groddy)
offine (gredyy)
1,000
2,000
3,000
4,000
1,000
2,000
3,000
4,000
Average Lagging (ms)
Average Lagsing (ms))
Average Lagging (ms)
(a) EnDe
(b) EnFr
(c) EnEs
Figure 3: The translation quality (BLEU) against the latency metrics (AL) on the 1st-COMMON set of MuST-C En-De, En-Fr and En-Es dataset. denotes that the results are obtained from corresponding papers. 4 denotes that the results are based on our re-implementation, where our implemented MoSST is slightly better than the original results and includes the results on large latency region. All dashed lines are models trained in constrained scenarios. denotes that the results of solid lines are from models that utilize external machine translation data.
10, while for evaluation, l ranges from 2 to 25. We conduct our training using 8 V100 GPUs, with a batch size per device of 3.2M audio frames.
boundaries based on CIF prediction.
Regarding En-De, which is the most researched language pair, our method significantly outper- forms all previous approaches in terms of trans- lation quality at high latency. In the constrained scenario (represented by dashed lines), our method scores at least 2 BLEU higher than MU-ST and MoSST, even outperforming MU-ST in the uncon- strained case. In the unconstrained scenario, our method can further enhance performance. low latency, it seems the curves of our approach and MU-ST are close. However, if we focus on the snapshot around latency 1000ms, our method sur- passes MU-ST by approximately 2 BLEUU. For En- Fr, fewer works have explored this language pair. Compared to existing baselines, model in strained scenario outperforms them by a significant margin. For instance, around low latency (1000ms), our method achieves a BLEU score exceeding 28. In the case of English to Spanish (En-Es), although our results are generally better, we found the addi- tional MT corpus appears to be less beneficial. The exact numerical results in Figure 3 can be found in Appendix A.7.
Model selection is based the corresponding development set, and we report final results the Ist-COMMON set. We utilize the detokenized case-sensitive BLEU score, calculated using sacre- BLEU!. Following the convention the SimulST, we use the average lagging (AL) evaluation model to measure the latency (Ma et al., 2019b) based on the toolkit SemiEval?.
4.2 Main Results of SimulST
We compared our model with recent simultancous ST models: SimulSpeech (Ren et al., 2020), Re- alTranS (Zeng al., 2021), MoSST (Dong al., 2022b), MU-ST (Zhang et al., 2022), MMA-SLM (Indurthi et al., 2022), and ITST (Zhang and Feng, 2022). Most of these works explored the SimulST task using the MuST-C dataset alone. MU-ST also conducted experiments with external data, us- ing both additional Automatic Speech Recognition (ASR) and Machine Translation (MT). In our case, we did not use additional ASR data.
Quality and latency trade-off by single model Figure 3, we plot the BLEU v.s. AL curve of single checkpoint our ST model for both offline and online scenarios across different latency. To achieve these results, we leverage our proposed wait-k-tokens strategy during inference, with token
Analysis of Pre-training Stage In the pre- training stage, the loss Epe primarily optimizes semantic alignment Icarning in terms of both se- quence length and semantic representations. We will illustrate the performance of these objectives.
First, we conducted a statistical analysis compar- ing the number of predicted segments by CIF to the actual subword lengths of the corresponding tran-

---

160
Model
En-De
En-Fr
En-Es
BealT (Zeng el al., 2021)
23.0
140
MoSST (Dong al., 2022b)
24.9
35.3
XSNET (Ye et al., 2021)
25.5
36.0
29.6
120
STEMM (Fang et al., 2022b)
25.6
36.1
30.3
100
ConST (Ye al., 2022)
25.7
36.8
30.4
Ours
26.8
37.7
30.4
80
XSNET (Ye al., 2021)
27.8
38.0
30.8
60
STEMM (Fang et al., 2022b)
28.7
37.4
31.0
ConST (Ye al., 2022)
28.3
38.3
32.0
40
STPT (Tang al., 2022)
39.7
33.1
20
Ours
29.2
39.7
31.0
0
Table The performance of the offline on the tst- COMMON dataset. The beam size of 5 is adopted in our evaluation. The top of the table represents the results using the constrained ST data, while the bottom represents the results using external data. The amount of external data varies lot between individual models.
50
100
150
Figure 4: A linear regression analysis using the ground- truth length (y) and the predicted length (x) on En-De Ist-COMMON dataset.
responding acoustic and text representations. If the transcription has two same tokens (such as "do" Figure 5), we can still observe semantic similar- ity for their corresponding acoustic output (23 and 25 in Figure 5). This phenomenon further verifies the effectiveness of semantic learning, Additional visualizations can be found in Appendix A.3.
Bhar
4A
0.14
6224
6.70
6.01
6.444
2277
do
6.92
6:99
6:17
6.10
6:11
Third, we explicitly visualize the audio-text alignment Figure 6. the provided example, we can observe there is a strong correlation between the values in the second row and the amplitude values of the speech in the first row. The segmen- tation activations in the third row are also highly correlated with the corresponding positions of the subwords. For meaningless blank characters with- out corresponding meaningful subword, the pre- dicted a almost 0, and no subword is generated. Additional examples can be found in Appendix A.4.
0/30
6308
6302
6:00
6.022
6214
1371
42
-Th
6872
279
6382
6:58
6.22
6.0
C317
1.03
Figure An example of the alignment for semantic representation via L2 distance. z is the acoustic encoder output after Linearemb as shown Figure 2.
scriptions. Figure demonstrates the En-De test set with the ground-truth length (y) and the predicted length (x) by our alignment module used to fit a lin- ear regression. The regression line closely aligns with the diagonal of the coordinate system, i.e., y = x, indicating the high accuracy of our length prediction. Furthermore, few outliers are distant from the regression line, suggesting that our length prediction exhibits low bias and variance. These re- sults collectively highlight the effectiveness of our speech pre-training in predicting sentence length. The analysis of length prediction for En-Fr and En-Es is provided in Appendix A.2.
4.3
Main Results of Offline ST
As the bidirectional architecture of Wav2Vec2.0. our ST model is naturally compatible with offline speech translation. Specifically, we compare our model, previously used for SimulST in Figure 3, with recently published offline ST methods: XS- NET, STEMM, ConST, and STPT. We also in- clude two earlier SimulST works (Real'rranS and MoSST) that evaluate both offline and online tasks.
Second, Figure illustrates the L2 distance be- tween the acoustic encoder output and the word em- beddings of its corresponding transcription. The el- ements along the diagonal generally exhibit smaller values, indicating a closer semantic distance for cor-
The overall results summarized in Table 2. In data-constrained scenarios, our offline ST model's BLEU evaluation matches the performance of these competitive offline models. In unconstrained sce- narios, only the performance of En-Es exhibits a

---

-5
0.2
0.0
80
20
40
60
80
Figure 6: First Row: word-level segmentation of the audio by MFA and the corresponding subwords in each segmentation. Second Row: predicted by fine- tne-ggained CIF module. Third Row: the accumulated summation of o, which will be reset once exceeding threshold 1.0.
R wait-k-
Fine gineggrained
Regarding the offline task, the R² wait-k-tokens training did not yield any notable difference when comparing model (a) and (b), however, in the
model
loss
pre- train
beam=5
tokens
CIF loss
(8)
29.15
(h)
29.00
(c)
V
28.70
SimulST task, the absence of this strategy changed the curve from an are shape to an almost straight line. When the fine-grained CIF loss was removed (model (c)), the offline performance decreased by about 0.5 BLEU, while the are curve also uniformly shifted downwards by 0.5 BLEU. Removing the entire semantic alignment pre-training (model (d)) resulted in SimulST performance at low latency that was on par with model (c), but the BLEU scores at high latency significantly decreased. summary, R² wait-k primarily enables streaming; the semantic alignment pre-training improves the performance at high latency; and the fine fine-grainee grained CIF loss is the final touch that further enhances performance at low latency. Another two loss lated ablation studies can be found in Appendix A.5. The analysis of how we design the current pre-training loss can refer to Appendix A.6.
(d)
V
27.95
(c)
v
28.62
(f)
v
28,
(
27.63
Table 3: The BLBU scores of offline ST from the abla- tion experiments conducted on tst-COMMON En-De. "R2 wait-A" means the robust and random wait-k to- kens training. "Fine-grained CIF" indicates the use of fine-grained CIF loss (first term in Eq. (3)) of both pre- traniigg and fine-tune tunitu. "CE loss" refers the use of CE loss during fine-tuning, "pre-train" indicates the use our proposed semantic alignment pre-training stage. The pre-trained NMT is always required.
significantly lower BLEU score. We hypothesize the main reason for this is the considerable varia- tion in the volume of external MT data across dif- ferent models. Our selected 15M En-Es MT corpus may also have a potential domain mismatch with MuST-C. Generally, these results demonstrate that our single checkpoint can maintain robust offline translation quality while also adapting to real-time translation requirements.
4.5 Stable Inference Low Lateney
In practice, during the inference stage, the first few target tokens often have low quality when the streaming input audio is short. Various strate- gies have been proposed by previous works to ad- dress this problem. E.g., RealTranS employs the Wait-K- --Stiide-t strategy, while MU-ST uses tail- truncation trick. In our approach, we empirically find that when the first few short speech segments are processed as streaming input, the CIF weights tend to bias towards larger values and require more
4.4 Ablation Studies
We conducted ablation experiments on the uncon- strained En-De dataset. The different settings and critical offline evaluations our control experi- ments are listed Table 3. The corresponding BLEU v.S. AL curves are shown in Figure 7.

---

28
resentation. The second step entails end-to-end training of the ST model, leveraging robust and random wait-k tokens policy. Remarkably, our model manages to satisfy the objectives of offline and streaming speech translation within a single checkpoint. Experimental results substantiate that our approach achieves a commendable trade-off between translation quality and latency. Given that semantic alignment pre-training is independent of the downstream policy in simultaneous trans- lation, we aim to explore opportunities to better accommodate an adaptive read/write policy in fu- ture work.
26
24
BLEU
22
model
20
model b
mudele
18
madi d
audel e
16
model g
14
1,000
2,000
3,000
4,000
Average Lagging (ms)
Figure 7: Ablation experiments conducted simulta- neous translation corresponding to Table 3.
Limitations
As discussed in Section 3, our training strategy can indeed mitigate some issues of the bidirectional Wav2Vec2.0, but it still encounters the same un- resolved issue in (Dong et al., 2022a; Zhang et al., 2022). During inference, the encoder hid- den states of Wav2Vec2.0 cannot be cached for streaming input. Thus, it is an efficient audio feature extractor in streaming scenario since the recalculation of audio features is necessary when- ever new streaming input arrives. A potential re- search direction could be replacing Wav2Vec2.0 with a streaming acoustic encoder. It's worth not- ing that Wav2Vec2.0's self-supervised learning is preitrained by masking 49% of all time steps with mean span length of 300ms. Therefore, fine- tuning the existing Wav2Vec2.0 with streaming input should feasible approach.
22
20
BLEU
18
origin
force
16
700
900
1,100
1,300
Average Lagging (ms)
Figure 8: origin means additional trick is applied. wait rmoor is to wait for more n extra segments before decoding the first target token, while the other decoding is normal. force is to force the model not to finish the hypothesis before the source audio is fully read.
Ethics Statement
Our work complies with the ACL Ethics Policy. Our experiments are based on the open-sourced dataset that is widely used in academia, and there is no violation of this dataset. Our writing is com- pletely based on the authors without plagiarism.
input frames to stabilize. Conversely, with longer speech inputs, the decoding process may end pre- maturely. Therefore, we propose strategy to wait for few extra segments before initiating the first write operation. After this, the read and write be- havior returns to normal. According to our analysis Figure& 8, our "wait-nmore" strategy effectively improves performance at low latency, and we upply the wait-2more strategy in our experiments.
A Acknowledgements
We would like to thank all the anonymous review- ers for their insightful and helpful comments. This work supported by Alibaba Group through Al- ibaba Research Intern Program.
5 Conclusion
We have proposed two-step training method for simultaneous speech translation. The first step in- volves performing token-level cross-modal align- ment between the audio and the transcription in terms of both sequence length and semantic rep-
References
Ashkan Alinejad and Anoop Sarkar. 2020. Effectively pretraining speech translation decoder with ma- chine translation data. In Proceedings of the 2020

---

Conference on Empirical Methods Natural Lan- guage Processing, EMNLP 2020, Online, November 16-20, 2020, pages 8014-8020. Association for Com- putational Linguistics.
Marco Turchi. 2021. Cascade versus direct speech translation: Do the differences still make a differ- ence? In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 2873-2887, Online, Association for Computa- tional Linguistics.
Antonios Anastasopoulos, Loic Barrault, Luisa Ben- tivogli, Marcely Zanon Boito, Ondicj Bojar, Roldano Cattoni, Anna Currey, Georgiana Dinu, Kevin Duh, Maha Elbayad. Clara Emmanuel, Yannick Estève, Marcello Federico, Christian Federmann, Souhir Gahbiche, Hongyu Gong, Roman Grundkiewiez, Barry Haddow, Benjamin Hsu, Dávid Javorsky, Vera Kloudová, Surafel Lakew, Xutai Ma, Prashant Mathur, Paul McNamee, Kenton Murray, Maria Nädejde, Satoshi Nakamura, Matteo Negri. Jan Nichues, Xing Niu, John Ortega, Juan Pino, Eliz- abeth Salesky, Jiatong Shi, Matthias Sperber, Se- bastian Stüker, Katsuhito Sudoh, Marco Turchi, Yo- gesh Virkar, Alexander Waibel, Changhan Wang, and Shinji Watanabe. 2022. Findings of the IWSLT 2022 evaluation campaign. Proceedings of the 19th International Conference on Spoken Language Translation (IWSLT 2022), pages 157, Dublin, Ireland (in-person and online). Association for Com- putational I Linguistics.
Alexandre Berard, Laurent Besacier. Ali Can Ko- cabiyikoglu, and Olivier Pietquin. 2018. End-to-end automatic speech translation of audiobooks. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2018, Calgary, AB, Canada, April 15-20, 2018, pages 6224-6228. IEEE.
Alexandre Berard, Olivier Pietquin, Christophe Servan, and Laurent Besacier. 2016. Listen and translate: A proof of concept for end-to-end speech-to-text trans- lation. CoRR, abs/1612200
Guandan Chen, Kai Fan, Kaibo Zhang, Boxing Chen, and Zhonggiang Huang. 2021. Manifold adversar- ial augmentation for neural machine translation. Findings of the Association for Computational Lin- guistics: ACL-IJCNLP 2021, pages 3184-3189.
Naveen Arivazhagan, Colin Cherry, Wolfgang Macherey, Chung-Cheng Chiu, Semih Yavuz, Ruom- ing Pang. Wei Li, and Colin Raffel. 2019. Monotonic infinite lookback attention for simultaneous machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1313-1323, Florence, Italy. Association for Computational Linguistics.
Fahim Dalvi, Nadir Durrani, Hassan Sajjad, and Stephan Vogel. 2018. Incremental decoding and training methods for simultancous translation in neural ma- chine translation. In Proceedings of the 2018 Con- ference of the North American Chapter of the As- sociation for Computational Linguisties: Human Language Technologies, NAACL-HLT, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 2 (Shori Pa- pers), pages 493-499. Association for Computational Linguistics.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2ver 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33: 12249-12460
Linhao Dong and Bo Xu. 2020. Cif: Continuous integrate- andegrate-and-foe and- fire for end-to- end-to-end speech recognition. In ICASSP 2020- 2020 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP), pages 6079-6083
Parnia Bahar, Tobias Bieschke, and Hermann Ney. 2019, A comparative study on end-to-end speech to text translation. 2019 IEEE Automatic Speech Recog- nition and Understanding Workshop (ASRU), pages 792-799. IEEE.
Qian Dong. Yaoming Zhu, Mingxuan Wang, and Lei Li. 2022a. Learning when to translate for streaming speech. In Proceedings of the 60th Anmual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Pupers), pages Ireland. Association for Computational Linguistics.
Sameer Bansal, Herman Kamper, Karen Livescu, Adam Lopez, and Sharon Goldwater. 2019. Pre-training on high-resouree speech recognition improves low- resource speech-to-text translation. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019. Minneapolis. MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 58-68. Association for Computational Linguistics.
Dublin,
Qian Dong, Yaoming Zhu, Mingxuan Wang, and Lei Li. 2022b. Learning when translate for streaming speech. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 680-694. Association for Computational Linguistics.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled sampling for sequence prediction with recurrent neural networks. Advances neural information processing systems, 28.
Qianqian Dong, Mingxuan Wang, Hao Zhou, Shuang Xu, Bo Xu, and Lei Li. 2021. Consecutive decoding for speech-to-text translation. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh
Luisa Bentivogli, Mauro Cettolo, Marco Gaido, Alina Karakanta, Alberto Martinelli, Matteo Negri, and

---

Symposiun on Educational Advances in Artificial In- telligence, EAAI 2021, Virtual Evens, February 2-9, 2021, pages 12738-12748, AAAT Press.
Sathish Reddy Indurthi, Houjeung Han, Nikhil Kumar Lakumarapu, Beomseok Lee, Insoo Chung, Sangha Kim, and Chanwoo Kim. 2020. End-end speech-to- text translation with modality agnostic meta-learning. In 2020 IEEE International Conference on Acous- fics, Speech and Signal Processing, ICASSP 2020, Barcelona, Spain, May 4-8, 2020, pages 7904-7908. IEEE.
Maha Elbayad. Laurent Besacier, and Jakob Verbeek. 2020. Efficient Wait-k Models for Simultameous Ma- chine Translation. In Proc. Interspeech 2020, pages 1461-1465.
Qingkai Fang, Rong Ye, Lei Li. Yang Feng, and Mingx- uan Wang. 2022a. STEMM: Self-learning with speech-text manifold mixup for speech translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pages 7050-7062, Dublin, Ireland. Association for Computational Linguisties.
Sathish Reddy Indurthi. Mohd Abbas Zaidi, Beomseok Lee. Nikhil Kumar Lakumarapu, and Sangha Kim. 2022. Language model augmented monotonic atten- tion for simultaneous translation. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 38-45, Seat- tle. United States. Association for Computational Linguisties.
Qingkai Fang, Rong Yc, Lei Li, Yang Feng, and Mingx- uan Wang. 2022b. STEMM: self-learning with speech-text manifold mixup for speech translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), ACL. 2022. Dublin, Ireland, May 22-27, 2022, pages 7050-7062. Association for Com- putational Linguistics.
Takatomo Kano, Sakriani Sakti, and Satoshi Nakamura. 2018. Structured- -baeed curriculum learning for end- to-end english-japanese speech translation. CoRR. abs/1802.06003
Taku Kudo. 2018. Subword regularization: Improv- ing neural network translation models with multiple subword candidates. In Proceedings of the 56th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66-75, Melbourne, Australia, Association for Computational Linguistics.
Biao Fu, Kai Fan, Minpeng Liao, Zhongqiang Huang, Boxing Chen, Yidong Chen, and Xiaodong Shi. 2023. Adapting offline speech translation models for streaming with future-aware distillation and infer- ence.
Marco Gaido, Mattia Antonino Di Gangi, Mat- teo Negri, and Marco Turchi. 2020. End-to- end speech-translation with knowledge distillation: Fbk@2ws12000. In Proceedings of the 17th Interna- tional Conference Spoken Language Translation. /WSLT 2020, Online, July 9- 10, 2020, pages 80-88. Association for Computational I Linguistics.
Hang Le, Juan Pino, Changhan Wang. Jiatao Gu, Didier Schwab, and Laurent Besacier. 2020. Dual-decoder transformer for joint automatic speech recognition and multilingual speech translation. In COLING 2020 (long paper).
Dan Liu, Mengge Du, Xiaoxi Li, Ya Li, and Enhong Chen. 2021. Cross attention augmented transducer networks for simultaneous translation. In Proceed- ings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 39-55, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Mattia Antonino Di Gangi, Roldano Cattoni, Luisa Ben- tivogli. Matteo Negri, and Marco Turchi. 2019. Must- c: multilingual speech translation corpus. In Pro- ceedings of the 2019 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies. NAACL- IITTT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pages 2012-2011 Association for Computational Linguisties.
Yuchen Liu, Hao Xiong, Jiajun Zhang, Zhongjun He, Hua Wu, Haifeng Wang, and Chengqing Zong. 2019. End-to-end speech translation with knowledge distil- lation. Interspeech 2019.
Jiatoo Gu, Graham Neubig, Kyunghyun Cho, and Vic- tor O.K. Li. 2017. Learning to translate in real-time with neural machine translation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume
Yuchen Liu, Jiajun Zhang, Hao Xiong, Long Zhou, Zhongjun He, Hua Wu, Haifeng Wang, and Chengqing Zong. 2020. Synchronous speech recog- nition and speech-to-ted text translation with interactive decoding. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Con- ference, TAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, FAAI 2020. New York, NY, USA, February 7-12, 2020. pages 8417-8424. AAAI Press
1. Long Papers. pages 1053-1062, Valencia, Spain. Association for Computational Linguistics.
Chi Han, Mingxuan Wang, Heng Ji, and Lei Li. 2021. Learning shared semantic space for speech-to-texx translation. Findings of the Association for Com- putational Linguistics: ACIAIJCNLP 2021, Online Event, August 1-6, 2021, volume ACL/IJCNLP 2021 of Findings of ACL, pages 2214-2225, Association for Computational Linguistics.
Mingbo Ma, Liang Huang, Hao Xiong. Renjie Zheng. Kaibo Liu, Baigong Zheng, Chuanqiang Zhang,

---

Zhongjun He, Hairong Liu, Xing Li, Hua Wu, and Haifeng Wang. 2019a. STACL: Simultaneous trans- lation with implicit anticipation and controllable la- tency using prefix-to-preet framework. In Proceed- ings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5025-3036, Flo- rence, Italy. Association for Computational Linguis- tics.
Marc' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016. Sequence level train- ing with recurrent neural networks. In 4th Inter- national Conference on Learning Representations, ICLR 2016.
Yi Ren, Jinglin Liu, Xu Tan, Chen Zhang. Tao Qin, Zhou Zhao, and Tie-Yan Liu. 2020, SimulSpeech: End-to-end simultaneous speech to text translation. In Proceedings of the 58th Anmual Meeting of the As- sociation for Computational Linguistics, pages 3787- 3796, Online. Association for Computational Lin- guistics.
Mingbo Ma, Liang Huang, Hao Xiong, Renjie Zheng, Kaibo Liu, Baigong Zheng, Chuanqiang Zhang, Zhongjun He, Hairong Liu, Xing Li, Hua Wu, and Haifeng Wang. 2019h. STACL: simultaneous trans- lation with implicit anticipation and controllable latency using prefix-to-preee framework. In Pro- ceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Pa- pers, pages 3025-3036. Association for Computa- tional Linguisties.
Yuchun Shu, Haoneng Luo, Shiliang Zhang, Longbiao Wang, and Jianwu Dang. 2023. A cif-based speech segmentation method for streaming e2c asr. IEEE Signal Processing Letters, 30:344 348.
Yun Tang, Hongyu Gong. Ning Dong, Changhan Wang, Wei-Ning Hsu, Jiatao Gu, Alexei Bacvski, Xian Li, Abdelrahman Mohamed, Michael Auli, and Juan Pino. 2022. Unified speech-text pre-training for speech translation and recognition. In Proceedings of the 60th Annual Meeting of the Association for Compulational Linguistics (Volume I: Long Papers), pages 1488-1499, Dublin, Ireland. Association for Computational Linguistics.
Xutai Ma, Juan Miguel Pino, James Cross. Liezl Pu- zon, and Jiatao Gu. 2020a. Monotonic multihead attention. In International Conference on Learning Representations.
Xutai Ma. Juan Miguel Pino, and Philipp Koehn. 2020b.
Simulmt to simulst: Adapting simultaneous text translation to end-to-end simultaneous speech trans- lation. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Com- putational Linguistics and the 10th International Joint Conference on Natural Language Processing, AACL/ICNLP 2020, Suzhou, China, December 4-7, 2020, pages 582-587. Association for Computational Linguistics.
Laura Cross Vila, Carlos Escolano, José AR Fonollosa, and Marta R Costa-Jussa. 2018. End-to-end speech translation with the transformer. IberSPEECH. pages 60-63.
Chengyi Wang. Yu Wu, Shujie Liu, Ming Zhou, and Zhenglu Yang. 2020. Curriculum pre-training for end-to-end speech translation. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 3728-3738. Association for Computa- tional Linguistics.
Xutai Ma. Yonggiang Wang, Mohammad Javad Dousti, Philipp Koehn, and Juan Pino. 2021. Streaming si- multaneous speech translation with augmented mem- ory transformer. In ICASSP 2021- 2021 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7523-7527.
Shushu Wang. Jing Wu, Kai Fan, Wei Luo, Jun Xiao, and Zhongqiang Huang. 2023. Better simultaneous translation with monotonic knowledge distillation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pages 2334-2349.
Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. 2017. Montreal forced aligner: Trainable text-speech align- ment using kaldi. In Interspeech, volume 2017, pages 498-502.
Ron J. Weiss, Jan Chorowski. Navdeep Jaitly, Yonghui Wu, and Zhifeng Chen. 2017. Sequence-te to to-sequence models can directly translate foreign speech. In In- terspeech 2017, 18th Annual Conference of the Inter- national Speech Communication Association, Stock- holm, Sweden, August 20-24, 2017, 7, pages 2625-2629. ISCA.
Ha Nguyen, Yannick Estève, and Laurent Besacier. 2021. An empirical study of end-to-end simultaneous speech translation decoding strategics. In IEEE Inter- national Conference on Acoustics, Speech and Sig- nal Processing, ICASSP 2021, Toronto, ON, Canada, June 6-11, 2021, pages 7528-7532. IEEE
Jichen Yang, Kai Fan, Minpeng Liao, Boxing Chen, and Zhonggiang Huang. 2023. Bridging the gap between cascade and end-to-end cross- mrosl translation mod- els: A zero-shot approach.
Yusuke Oda, Graham Neubig. Sakriani Sakti, Tomoki Toda, and Satoshi Nakamura. 2014. Optimizing seg- mentation strategies for simultancous speech transla- tion. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, ACI. 2014, June 22-27, 2014, Baltimore, MD, USA, Vol- ume 2: Short Papers, pages 551-556. The Associa- tion for Computer Linguistics.
Rong Ye, Mingxuan Wang, and Lei Li. 2021. End-to- end speech translation via cross-modal progressive training. In Interspeech 2021, 22nd Annual Con- ference of the International Speech Communication

---

Association, Brno, Czechia, 30 August September 2021, pages 2267-2271. ISCA.
Algorithm Threshold-based Weighted Averaging in Continuous Integrate-and-Fine (CIF)
Rong Ye, Mingxuan Wang, and Lei Li. 2022. Cross- modal contrastive learning for speech translation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, NAACI 2022, Seattle, WA, United States, July 10-15, 2022, pages 5099-5113 13. Association for Computa- tional Linguistics.
Input: The output features of Wav2Vec2.0 acoustic tokens a a2,, aT). The nor- malized CIF weights a = (011, 02; .... 0T) sat- isfying Σ' at = Threshold = be default.
Output: The averaged acoustic tokens after CIF module have the same subword length as the transcription x, denoted as -ci,.... CI.). function MAIN
Xingshan Zeng, Liangyou Li, and Qun Liu. 2021. Re- altrans: End-to-end simultaneous speech translation with convolutional weighted-shrintee In Findings of the Association for Computational Lin- guistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021, volume ACL/MCNLP 2021 Findings of ACL. pages tional Linguisties.
transformer.
// Initialize i 1 and will end with L, initial accumulated weight og 0, initial ac- cumulated state ag = 0:
Association for Computa-
3:
i == 1, ajj = 0, aj =
4:
for = 1 do
Linlin Zhang. Kai Fan, Boxing Chen, and Luo Si. 2023. simple concatenation can effectively im- prove speech translation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1793- 1802.
5:
// calculate accumulated weight;
6:
of = 08-1 +o
if o < 5 then
// boundary is located;
af a;-1 + at X at
10:
else
Ruiqing Zhang, Zhongjun He, Hua Wu, and Haifeng Wang. 2022. Learning adaptive segmentation policy for end-to-ent simultaneous translation. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Pa- pers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 7862-7874. Association for Computational Linguistics.
11:
// a boundary is located;
12:
Il on is divided into two part, the
first part 0t,, is used to fulfill the integration of current
13:
0t; 1 -1-af-1
14:
C, = ag. + an ai
15:
i
Ruiqing Zhang, Chuanqiang Zhang, Zhongjun He, Hua Wu, and Haifeng Wang. 2020. Learning adaptive segmentation policy for simultaneous translation. In Proceedings of the 2020 Conference Empirical Methods Natural Language Processing (EMNLP), pages 2280 2289, Online. Association for Computa- tional Linguisties.
16:
// The other part Oxz is used for the next integration;
17:
af = 0ttz = att Ct1l
18:
a 012 x a
19:
end if
20:
end for
Shaolei Zhang and Yang Feng. 2022. Information- transport-based policy for simultaneous translation.
21:
22: end function
In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 992- Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
23: return e (cI, CL)
A Appendix
length for different language pairs. From Figure 9, it can be seen that our method exhibits high pre- diction accuracy during both the cross cross-modal pre- training and ST fine-tuning phases. All models can fit the linear regression equation y The data is primarily concentrated on this line. From the results, our model can maintain high level of accuracy in predicting length whether or not fine- tuning is done on the ST. At the same time, there is little variation across different language pairs.
A.1
Threshold-Based Weighted Averaging of CIF
In the section, we formulate the threshold-basee weighted averaging of CIF into Algorithm 1.
A.2
Visualization of Length Prediction for CIF
We conducted linear regression analysis on the pre- dicted length of CIF and actual transcript token

---

160
160
160
140
140
120
120
100
100
100
80
60
60
40
40
20
20
0
150
50
100
150
50
100
150
(a) En-De Pre-training
(b) En-Fr Pre-training
(c) En-Es Pre-training
160
160
140
140
120
120
100
100
100
80
80
60
60
20
20
20
0 50
100
150
50
100
150
50
100
150
(d) En-De Fine-tuning
(c) En-Fr Fine-tuning
En-Es Fine-tuning
Figure 9: A linear regression analysis using the ground-truth length (y) and the predicted length (x) En-De, En-Fr and En-Es IN-COMMON dataset. The first row corresponds to the cross-modal pre-training phase, while the second row represents the ST fine-tuning phase.
A.3 Visualization of Cross-modal Alignment
randomly selected examples, including complete speech sentences and some truncated sections of the speech. In general, can observe early activation for subwords, i.e., the accumulation sum of tends to reach threshold 1.0 earlier than the boundary detected by MFA. We hypothesize that the boundary from MFA is usually located the middle of consecutive silence frames, and the cor- responding a for silence frame is almost 0, thus, the larger a values are usually predicted earlier.
We illustrate the L2 distance between the acoustic encoder output and the word embeddings of its corresponding transcription in Figure 10. From Figure 10, it can be observed that regardless of the accuracy of length prediction, all examples exhibit lighter-colored areas along the diagonal, indicating that method aligns speech and text distances more closely in that region, while distances greater in other areas.
Additionally, subwords such as "y" in Fig- ure 10(b), "to" in Figure 10(c), "_the" in Fig- ure 10(d), and "ddo" in Figure 10(e) are repeated, with their corresponding speech output being very similar. This suggests that method not only simply brings speech and text closer in input posi- tion order, but truly surpasses input order align them in semantic space.
A.5 Additional Ablation Loss
The three pre-training losses (CE loss, embed loss, and fine-grained CIF loss) are by default optimized in the fine-tuning stage, and the relevant ablation study is mainly conducted on the fine-tuning stage in Table 3. In this section, we conduct several new ablation studies of the auxiliary loss to supplement Table 3. In Table 4, if the loss is not marked V. it means the corresponding loss is removed in both the pre-training and fine-tuning stages. Re- moving both the CE loss and embed loss both the pre-training and fine-tuning stages results
A.4 Speech Segmentation and Alignment
Our method, after the cross-modal align pre- training, is effective in achieving segmentation and alignment at the token-level. Figure shows some

---

(8)
(b)
4
25
-0
270
C:00
110
.Ph
AXT
140
.007
144
144
640
(d)
Figure 10: Some examples of speech segmentation and alignment on dev En-De, where z is the output of the acoustic encoder after Linearamh shown in Figure 2. (Better view by zoom-in) When token appears than once in sentence, we can observe the lower valued elements may not appear on the diagonal, egg., "y" in (b). "to" (c), "the" in (d), and "do" (e).
a significant drop in performance. Based on the previous ablation experiment results, clude that the CE loss plays crucial role in the pre-training stage but has minimal impact in the fine-tuning stage. In Table 5, we conduct a new ablation study to explore whether pre-training loss Sp is still required the fine-tuning stage. In the bottom row of Table 5, after the pre-training stage, all three losses in Cpt are removed in fine-tuning As expected, the BLEU on the dev set significantly dropped. Theoretically, the fine-grained CIF loss in crucial because it provides the ability of audio segmentation for the streaming input. Without CIF loss, we found the length prediction becomes ex- tremely worse, and imposes big gap between the training and inference. In addition, the convergence becomes disordered. The convergence is usually achieved after 10+ epochs. Surprisingly, when Cep is removed, the highest BLEU on the dev set is ob- served in the first epoch, followed by decline in subsequent epochs. Therefore, our results indicate that Ept is indeed crucial during the fine-tuning stage. Actually, including the Epr in fine-tuning stage becomes common in speech translation, e.g.,
Finc-grained CIF loss
model
CH loss
Emh loss
5
con-
29.15
(ii)
28.77
(iii)
27.84
Table 4: The BLEU scores of offline ST from the abla- tion experiments conducted on tst- MI-COMMMN En-De. If not marked with V. the corresponding loss removed for BOTH pre-training and fine-tune fine-tuning stages.
pre-train loss
beam=5
beam=1
V
29.15
28.
24.49
23.32
Table 5: Ablation study pre-training loss in fine- tuning stage.
offline model STPT.
A.6 The Ablation of Alignment Pre-training
We demonstrate ablation experiments for cross- modal alignment pre-training. As shown in Table 6. given sufficient training, there is little difference in accuracy and model length between the speech recognition models. However, in order to bring

---

to
Today
energ
(a)
this
50
smmle
exercise
150
There
world
record
that
this
0.4
0.2
0.0
20
40
60
80
1.0
0.5
20
40
60
80
(c)
Figure 11: Some examples of speech segmentation and alignment on dev En-De. Each example divided into three rows. The first row displays the word-level segmentation of speech computed by MFA and the tokens/subwords corresponding to each segment. The second row shows the alphas from CIF. The third row displays the accumulated scores, which activates the generation of tokens (when they exceed the threshold of 1.0).

---

ce loss
ctc loss
embed
mixup
Accuracy
WER
Avg length bias (12)
Representation dist
56.83
17.41
2.099
0.972
56.05
17.83
2.103
56.28
18.01
2.102
56.97
17.66
2.110
0.1275
Table 6: Ablation study on the alignment performance in pre- vee-traiiin stage. The scores obtained from the ablation experiments conducted on dev En-De. "ce loss" refers to the use CE loss. "cte loss" indicates the use of. "embed" indicates the of MSE loss with the loaded MT embedding. "mixup" means the of mixup between speech and text representations. "Avg length bias" shows the average L2 distance of the predict length and the groundtruth length. "Representation dist" meatures distance between speech and text representations.
the word vectors of pre-trained MT closer, we ulti- mately chose the current combination.
From the first 3 metrics (accuracy, WER, and length prediction error), cannot distinguish the importance of mixup. However, define metric measure the distance between speech and text representations called Representation dist,
min 22 ejl L [E(LLLM)
(11)
Since the length of two representations may be the same during inference, we select the min- imum pairwise distance. In this metric, we can observe the mixup training can significantly reduce the representation distance.
A.7 Numeric Results for Figures
These are the numerical results corresponding to our method's simultancous ST, shown in Table 7

---

Wait-k
2
3
4
5
7
10
15
25
Offline
Our
BLEU
16.68
19.18
21.36
22.76
24.46
25.42
25.49
25.55
25.90
AL(mm)
787
1049
1275
1509
1956
2569
3427
4532
inf
Ours
BLEU
17.30
20.24
22.77
23.99
26.22
27.46
28.01
28.10
28.12
AL(ms)
752
1014
1261
1494
1939
2560
3410
4518
inf
(a) En-De
Wait-k
2
3
4
5
10
15
20
Offline
Our
BLEU
27.97
31.07
33.14
34.45
36.05
36.28
36.63
36.60
36.65
AL(ms)
1058
1294
1525
1744
2192
2786
3601
4208
inf
Ours*
BLEU
28.88
32.44
34.68
36.53
38.11
38.54
38.77
39.04
39.05
AL(ms)
1062
1297
1544
1779
2224
2805
3617
4218
inf
(b) En-Fr
Wait-k
2
3
7
10
15
20
Offline
Our
BLEU
19.81
21.97
23.82
25.68
28.02
29.27
29.50
29.49
29.51
AL(ms)
602
861
1100
1341
1816
2484
3417
4128
inf
Ours
BLEU
19.85
22.44
24.97
26.77
28.82
29.82
30.20
30.44
30.44
AL(mm))
625
900
1140
1381
1852
2518
3448
41599
inf
(b) En-Es
Table 7: Numeric results on MuST-C En-De, En-Fr, and En-Es Ist-COMMON set. (Figure 3)


Reviews:
[{"review_id": "82bqgDSiKW", "reviewer": "Reviewer_7R6j", "summary": "", "questions": "- L211-216: could you please clarify this point? It is not clear to me why the cropping of 320k audio samples produces the alignment between acoustic-to-tokens $a$ and the word-level transcription. \n- Eq2: does it mean that $b_1 = 1$ and $e_w = T$?\n- L239: Is this the quantity loss proposed in Dong and Xu, 2020?\n- L271:  $z = Linear_{emb}(h)$: Is this linear layer initialized from pre-trained NMT model and frozen during training?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "#### Thank you for your positive reviews and your familiarity with the relevant works in our field. We will address your concerns as follows.\n\n>Q1: The related work section is missing some relevant citations.\n\nA1: We appreciate your attention to our field and expertise. We will add these citations in the revised version. We will also emphasize that our pre-training method mainly focuses on streaming scenario with fine-grained CIF module, while other works (e.g., MAESTRO, mSLAM, etc.) focus on offline scenario. In addition, the two different objectives may have different data requirements, e.g., regular speech pre-training may only need audio-only data without transcription. \n\nCompared with suggested speech translation related works, our work mainly focuses on the simultaneous speech translation task with ST and MT data, while other works explore better speech representation with extensive unlabeled audio. Therefore, in our current version, we did not conduct a comparison with them. \nFor example,  Ye et al. 2022 in the suggested references has been already compared (as ConST) in our Table 2.\n\n>Q2: Insufficient analysis of the speech-text alignment method in the pre-training stage.\n- Q2-a: The contribution of each auxiliary loss in the pre-training stage to the final ST performance is not analyzed.\n\nA2-a: The three pre-training losses (CE loss, embed loss, and fine-grained CIF loss) are by default optimized in fine-tuning stage, and the relevant ablation study is mainly conducted on the fine-tuning stage in Table 3. \n\nFor this response, we conduct several new ablation studies of the auxiliary loss in the pre-training stage. In the following Table, if the loss is not marked as `+`, it means the corresponding loss is removed in both pre-training and fine-tuning stages.\n\n| CE loss | embed loss | fine-grained CIF | BLEU（beam5） |\n| :---: | :---: | :---: |:---: |\n| + | + | + | 29.15 |\n| + |    | + | 28.77 |\n|    |    | + | 27.84 |\n\nAccording to the above Table, we can observe that removing only the embed loss has little impact on the final results, but we also notice that it will significantly affect the convergence speed during fine-tuning. Normally, the best dev BLEU is achieved around the 10th epoch, but without the embed loss, it becomes the 13th epoch. \n\nRemoving both the CE loss and embed loss in both the pre-training and fine-tuning stages results in a significant drop in performance. Based on the previous ablation experiment results, we can conclude that the CE loss plays a crucial role in the pre-training stage but has minimal impact in the fine-tuning stage.\n\n- Q2-b: The proposed pre-training method should be compared against other pre-training methods\n\nA2-b: Thanks for the insightful suggestion. In general, the pre-training model is difficult to compare directly, and it requires some downstream tasks for evaluation. Therefore, we compared with two pre-training based SimulST models (MoSST and MU-ST) in Figure 2. \n\nIn addition, our pre-training is designed for streaming scenario with the proposed fine-grained CIF module, while other works focus on offline pre-training. It is also unfair for direct comparison. Since our model is comparable with offline ST, we compared with STPT in Table 2, where STPT employs an offline (token masking with bidirectional attention) pre-training framework. Note that STPT used the comparable amount of ASR data but extremely large-scale unlabeled audio (60K hours) in pre-training.\n\n>Q3: The fine-tuning stage lacks sufficient ablation experiments. For example, is $L_{pt}$ really needed during the fine-tuning stage?\n\nA3: Whether applying pre-training stage has been discussed in Table 3, and we also include some ablation studies on individual pre-training loss in Table 3. We would like to conduct more experiments to address the raised concern. \n\n| $L_{pt}$ (when finetune) | BLEU（beam5） | BLEU（beam1） |\n| --- | :---: | :---: |\n| + | 29.15 | 28.12 |\n|    | 24.49 | 23.32 |\n\nIn the bottom row of above Table, after the pre-training stage, all three losses in $L_{pt}$ are removed in fine-tuning. As expected, the BLEU on the dev set significantly dropped. \n\nTheoretically, the fine-grained CIF loss in $L_{pt}$ is crucial because it provides the ability of audio segmentation for the streaming input. \nWithout CIF loss, we found the length prediction becomes extremely worse, and impose a big gap between the training and inference. In addition, the convergence becomes disordered. The convergence is usually achieved after 10+ epochs. Surprisingly, when $L_{pt}$ is removed, the highest BLEU on dev set is observed in the first epoch, followed by a decline in subsequent epochs. Therefore, our results indicate that $L_{pt}$ is indeed crucial during the fine-tuning stage. Actually, including the $L_{pt}$ in fine-tuning stage becomes common in speech translation, e.g., offline model STPT and Simul model MoSST. \n\n### Questions for the authors:\n\n>Q1: L211-216: Could you please clarify this point? It is not clear to me why the cropping of 320k audio samples produces the alignment between acoustic-to-tokens $a$ and the word-level transcription.\n\nA: There may be some misleading in this sentence. We did not apply `cropping` of `320k` audio samples. We did a `downsampling` by `320x` on the audio frames, i.e., reducing the audio length by 320x.\n\nIn the previous sentence, we mentioned that we used the MFA tool to achieve word-level alignment between transcripts and raw audio frames. Due to the 320x compression through the 7-layer CNN downsampling of wav2vec2, the alignment after compression can still be achieved because the strides of CNN layers are known.\n\n>Q2: Does Eq2 mean that $b_1 = 1$ and $e_w = T$?\n\nA: Yes, that is correct, $b_1 = 1$ and $e_w = T$.\n\n>Q3: Is the quantity loss proposed in Dong and Xu, 2020 referenced in L239?\n\nA: Yes, the quantity loss mentioned in line 239 is proposed in Dong and Xu, 2020, which is the traditional CIF loss. We proposed the fine-grained CIF loss.\n\n>Q4: L217:$z = Linear_{emb}(h)$ Is this linear layer initialized from pre-trained NMT model and frozen during training?\n\nA: The linear layer maps the speech representation $h$ to the source embedding space, $R^{d} \\rightarrow R^{d}$, where $d$ is hidden dimension (equal to embedding dimension in our case). This linear layer did not exist in the NMT model, and it is trained in both stages.\n\n\n### Missing References:\n\nWe appreciate your professional suggestions and will add relevant references in the revised version. \n\nWe will also mention that the suggested references are all non-streaming ST models, which are difficult to compare with our proposed simultaneous approach. For example, it is unfair to plot the common BLEU-AL curve for non-streaming ST models. \n\nSince our model is comparable with non-streaming ST task, we would like to include the recent offline ST works. E.g., Le et al. 2023 was released in May, 2023 and we did not notice this paper when preparing our submission. CRESS has two versions by being built upon HuBERT and Wav2Vec2. Our model is built upon Wav2Vec2 and achieves a better result than CRESS Wav2Vec2, but worse than CRESS HuBERT. We will also add the SOTA of non-streaming ST model - SpeechUT in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well-written with clear motivation and easy-to-follow structure.\n- The proposed method has the potential to be applied in various speech applications other than simultaneous and offline ST as speech-text alignment is an important problem in many speech-related tasks.\n- The ablation study and analysis is interesting, offering insights into the effectiveness of the proposed components in different settings (online and offline).\n- The method performs well on both simultaneous and offline ST tasks and seems to be robust for different latency requirements.", "reasons_to_reject": "1) Related work section misses a highly relevant line of work that aims to align speech and text in speech in general (and in ST in particular) that would help to better position itself in the field. \n\n\t- Different alignment methods have been proposed for self-supervised pre-training for speech such as Bapna et al. 2021 (SLAM: A unified encoder for speech and language modeling via speech-text joint pre-training), Chen et al. 2022 (MAESTRO: matched speech text representations through modality matching), Bapna et al. 2022 (mSLAM: Massively multilingual joint pre-training for speech and text), for supervised ST pre-training such as Alinejad and Sakar. 2020 (Effectively pretraining a speech translation decoder with machine translation data) and Le et al. 2023 (Pre-training for Speech Translation: CTC Meets Optimal Transport), or in a multi-task learning framework for ST such as Liu et al. 2020 (Bridging the modality gap for speech-to-text translation), Tang et al. 2021 (Improving speech translation by understanding and learning from the auxiliary text translation task), Ye et al. 2022 (Cross-modal contrastive learning for speech translation), Ouyang et al. 2022 (WACO: word-aligned contrastive learning for speech translation), etc., just to name a few off the top of my head.\n\n2) The analysis of the speech-text alignment method in the pre-training stage is not very convincing to me. In particular,\n\t- The contribution of each auxiliary loss in the pre-training stage to the final ST performance is not analyzed.\n\t- The proposed pre-training method should be compared against other pre-training methods (at least the most common method).\n\n3) The fine-tuning stage consists of many objectives (7 individual losses) and the paper lacks an analysis of the contributions of these individual losses to the final ST performance. For example, whether $L_{pt}$ is really needed during the fine-tuning stage?", "questions_to_authors": "- L211-216: could you please clarify this point? It is not clear to me why the cropping of 320k audio samples produces the alignment between acoustic-to-tokens $a$ and the word-level transcription. \n- Eq2: does it mean that $b_1 = 1$ and $e_w = T$?\n- L239: Is this the quantity loss proposed in Dong and Xu, 2020?\n- L271:  $z = Linear_{emb}(h)$: Is this linear layer initialized from pre-trained NMT model and frozen during training?", "paper_topic_and_main_contributions": "This paper proposes pre-training and fine-tuning methods for simultaneous speech-to-text translation (ST). The pre-training step which focuses on aligning speech and text sequences in terms of both length and representation space is based on the Continuous Integrate-and-Fire (CIF) module. Additional pre-training losses including L2-distance and cross-entropy (CE) loss are also used. The fine-tuning phase is performed in a multi-task learning setup with multiple objectives, including the ones in the pre-training stage. A novel wait-$k$ strategy is proposed for the fine-tuning stage to improve robustness of the model in various latency settings.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "In addition to the references cited in \"Reasons to reject\", I think the paper should include the following additional references:\n- L133: Le et al. 2023 (Pre-training for Speech Translation: CTC Meets Optimal Transport) performs a comprehensive study on the effects of supervised pre-training to ST training. It specifically aims to better align speech and text at the pre-training stage, which is highly related to this work. It also demonstrates the benefits of using CTC loss in pre-training (which is relevant to Table 4 of this work).\n- L137: important papers in multi-task learning such as Bahar et al. 2019 (A Comparative Study on End-To-End Speech to Text Translation) should be mentioned as well.\n- L139: First work which adopts knowledge distillation for ST such as Liu et al., 2019 (End-to-End Speech Translation with Knowledge Distillation) should be appreciated.\n- As a minor suggestion, a wait-$k$ strategy with fixed window is also proposed for offline ST in Le et al. 2020 (Dual-decoder Transformer for Joint Automatic Speech Recognition and Multilingual Speech Translation).\n- Table 2 should include reference to the current SoTA results in ST of SpeechUT (Zhang et al. 2022 \"SpeechUT: Bridging speech and text with hidden unit for encoder-decoder based speech-text pre-training\") and CRESS (Fang et Feng. 2023 \"Understanding 668 and bridging the modality gap for speech translation\").", "Typos_Grammar_Style_and_Presentation_Improvements": "- It is not very clear from Figure 2 what the role of NMT modules are. \n- Interesting visualization such as Figure 8 should be included in the main content.\n- Figure 6: Should add caption referring to Table 3 so that readers know what models a-g mean."}}, {"review_id": "3uAemxw6zA", "reviewer": "Reviewer_j7oz", "summary": "", "questions": "\nSee above.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "#### Thank you for your helpful reviews and comments. We will address your concerns individually.\n\n> Q1: The training processing of the proposed method is more complicated. It has multiple training stages and losses.\n\nA1: Thanks for proposing this concern. Similar to most speech translation works (e.g., STPT, SpeechUT, MoSST), we have only two standard training stages: pre-training and fine-tuning. Some work (e.g., MAESTRO) even requires three stages.\n \nOur approach is a simultaneous ST model for streaming input. \n\n- In pre-training stage, our main objective is speech-text alignment for streaming input. We have `3` losses, where $L_{ce}$ and $L_{emb}$ aim at speech-text alignment, and fine-grained $L_{cif}$ aim at audio segmentation for streaming input. \n- In fine-tuning stage, our main objective is simultaneous translation with `2` additional losses -- offline and streaming translation losses, where they are applied to two modalities: speech and text, i.e., $L_{off}^{st}, L_{off}^{mt}, L_{wk}^{st}, L_{wk}^{mt}$\n\nFor this response, we compare with a non-streaming ST model - STPT, which has `4` losses and `1` additional model. \n- Their masked prediction loss $L_{ssl}$ and the speech phoneme classification loss $L_{s2p}$ are used for speech-text alignment. The two losses are equivalent to our $L_{ce}$ and $L_{emb}$\n- Their $L_{s2p}$ loss also requires to train an additional `HMM-GMM model` for label annotations.\n- Their text-to-text loss $L_{t2t}$ and speech-to-text loss $L_{s2t}$ are exactly our $L_{off}^{mt}, L_{off}^{st}$.\n\nIn contrast, our fine-grained $L_{cif}$ and $L_{wk}^{st}, L_{wk}^{mt}$ are newly introduced, but they are both designed for simultaneous scenario.\n\n[1] STPT:  Yun Tang, Hongyu Gong, Ning Dong, Changhan Wang, Wei-Ning Hsu, Jiatao Gu, Alexei Baevski, Xian Li, Abdelrahman Mohamed, Michael Auli, and Juan Pino. 2022. Unified speech-text pre-training for speech translation and recognition. In Proceedings 874\nof the 60th Annual Meeting of the Association for Computational Linguistics (ACL Volume 1: Long Papers), pages 1488–1499, Dublin, Ireland. Association for Computational Linguistics.\n\n[2] SpeechUT:  Ziqiang Zhang, Long Zhou, Junyi Ao, Shujie Liu, Lirong Dai, Jinyu Li, and Furu Wei. 2022. SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1663–1676, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.\n\n[3] MoSST:   Qian Dong, Yaoming Zhu, Mingxuan Wang, and Lei Li. 2022. Learning when to translate for streaming speech. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL Volume 1: Long Papers), pages 680–694, Dublin, Ireland. Association for Computational Linguistics.\n\n[4] MAESTRO:  Zhehuai Chen and Yu Zhang, Andrew Rosenberg, Bhuvana Ramabhadran, Pedro J. Moreno, Ankur Bapna, and Heiga Zen. MAESTRO: Matched Speech Text Representations through Modality Matching. 2022 Interspeech.\n\n\n> Q2: Why is your model significantly better than other models in Figure3? This question needs to be explained clearly.\n\nA2: Thanks for raising this issue. We've already mentioned two main reasons in our paper.\n1. In the abstract and the paragraph in L80 of introduction, we mentioned that our approach finds an appropriate method for bridging the gap between streaming MT and streaming ST. The fine-grained CIF module successfully transforms the audio feature from audio length into transcription length. Then we can leverage the text simultaneous translation tricks to improve simultaneous speech translation. \n2. In L419 and L431 of experimental section, we mentioned that En-De is explored by many related SimulST works while En-Fr and En-Es are seldom researched. Therefore, our improvement on En-De over other baselines is not as significant as the other two language pairs.\n\nA possible reason that is not discussed in the paper is that our method combines the advantages of previous offline translation models (e.g., forced alignment trick) and simultaneous speech translation models (e.g., CIF audio segmentation trick). Our pre-training with MFA annotation and fine-tuning with random/robust wait-k-tokens strategy also make the transition from offline ST (infinity latency) to SimulST (small and finite latency) much smoother. Therefore, we maintain good ST performance for low latency region.\n\n### Reproducibility: 2 The contribution depends on data that are simply not available outside the author's institution or consortium; not enough details are provided.\nWe have provided all the hyper-parameter settings, training steps, and model architecture sizes in the paper. Additionally, we have uploaded the code package, which includes script files for each step of the process. The data we used is open-sourced as well and underwent the same data preprocessing steps as STEMM.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "\n\n1. This paper introduces a cross-modal pre-training method for acoustic model, with the fine-grained CIF module. And a robust wait-k tokens strategy is proposed to optimize the simultaneous speech translation tasks.\n\n2. The proposed method achieves a better trade-off between translation quality and latency than baselines.", "reasons_to_reject": "\n\n1. The training processing of the proposed method is more complicated. It has multiple training stages and losses.\n\n2. Why is your model significantly better than other models in Figure3? This question needs to be explained clearly.", "questions_to_authors": "\nSee above.", "paper_topic_and_main_contributions": "\nThis paper introduces a two-stage training method for simultaneous speech translation. It first conducts a token-level semantic alignment pre-training between audio and transcription. Then it utilizes a robust and random wait-k tokens policy to enable simultaneous decoding.\nThe authors verify the proposed method on MuST-C English to German, French, and Spanish speech translation pairs. Results demonstrate the effectiveness of the method.\n", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wIhhEUOmVs", "reviewer": "Reviewer_6LVe", "summary": "", "questions": "A. What version of wav2vec 2.0 are you using? I assume it is the non-ASR fine-tuned, but please indicate it. Also if it is the base or large.\n\nB. Did you carry out any ablation studies regarding the impact of mixup (Fang et al 2022) during the pre-training?\n\nC. In L410-L411 it is stated that no additional ASR data are used. But doesn't the use of the external alignment tool indirectly implies the use of ASR data?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "#### Thank you for your valuable feedback and suggestions. We’ll go over your concerns below.\n\n\n> Q1: What version of wav2vec 2.0 are you using? Is the non-ASR fine-tuned base model?\n\nA1: Yes, thank you for your domain expertise and familiarity with experimental details. We indeed use the non-ASR fine-tuned base model of wav2vec 2.0.\n\n\n> Q2: Did you carry out any ablation studies regarding the impact of mixup (Fang et al 2022) during the pre-training?\n\nA2: We did not perform ablation studies specifically on the impact of mixup. We appreciate your professional suggestion and have included additional experiments to address this.\n\n| Mixup | Accuracy | WER | Avg length bias (L2) |\n| --- | :---: | :---: | :---: |\n| + | 56.83 | 17.41 | 2.099 |\n|    | 56.97 | 17.66 | 2.110 |\n\n\nWe are somewhat surprised as well. It appears that mixup has minimal impact on our pre-training.\nWe will further clarify the scope of our research.\n\n> Q3: In lines 410-411, it is stated that no additional ASR data are used. However, doesn't the use of the external alignment tool indirectly imply the use of ASR data?\n\nA3: Thank you for raising this insightful question. You are correct that the use of the external alignment tool, MFA, implies the use of Librispeech ASR data. We utilized MFA as a readily available alignment tool instead of training our own alignment tool.\n\nIn theory, training our own MFA alignment model with Must-C ASR data might yield better results. Here are a couple of reasons for considering this:\n\n(1) There is a domain gap between Librispeech and Must-C, and our previous experience suggests that incorporating more Must-C ASR data yields better results for English ASR tasks. For example, in a previous work using an Encoder-Decoder architecture for En-De Must-C ST data, the word error rate (WER) was 18.2. After incorporating Librispeech, the WER decreased to 17.8. However, using all eight language pairs of Must-C ASR data reduced the WER to 16.8, even though some ASR data for other language pairs overlapped with the En-De ASR data. This trend was also observed in the IWSLT competition, where training with a larger dataset like Must-C V2 yielded better results than Must-C V1.\n\n(2) MFA might not provide perfect alignment. On one hand, MFA can not handle noisy or speech signals that cannot extract CMVN features, resulting in the exclusion of some training data (after filtering, train data size is reduced from 225K to 217K). Following the approach of SpeechUT, training an encoder-decoder model for speech-to-text unit alignment using a small portion of ASR data could be beneficial. In theory, we could leverage this model to train our own speech-text alignment tool, which would prevent us from losing a portion of the training data.\n\nTherefore, training an in-domain MFA model using Must-C data might lead to better results. We acknowledge the value of your professional suggestion and will continue exploring these possibilities in the future.\n\nThank you for your insightful input. This is indeed an area worth further investigation.\n\nAnd we'll revise the next edition in writing according to your suggestions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is very well written and makes several contributions to the field of Simultaneous Speech Translation. The proposed pretraining strategy using audio-text alignment is effective in birding the modality gap, and making it easier to do transfer learning from SimulMT. It also seems to be able to accurately handle the sub-world level segmentation and the to predict the length of the text representation from the acoustic information of wav2vec2.0. Furthermore the proposed robust & random-k wait policy is integral to the success of the model as indicated by Figure 6, and manages to have high translation quality across different latencies. Overall, the paper would offer significant value to the NLP community both in terms of the novel techniques proposed and the comprehensive analysis and understanding of the challenges in simultaneous speech translation.", "reasons_to_reject": "The use of external alignment tools and pretrained models like wav2vec2.0 could potential make the method less effective in other scenarios with low-resource language pairs and non-English source. But I think that given the benefits and the known difficulty of the task, it is not a compelling reason of concern.", "questions_to_authors": "A. What version of wav2vec 2.0 are you using? I assume it is the non-ASR fine-tuned, but please indicate it. Also if it is the base or large.\n\nB. Did you carry out any ablation studies regarding the impact of mixup (Fang et al 2022) during the pre-training?\n\nC. In L410-L411 it is stated that no additional ASR data are used. But doesn't the use of the external alignment tool indirectly implies the use of ASR data?", "paper_topic_and_main_contributions": "This paper addresses the challenges in Simultaneous Speech Translation (SimulST), a task designed to produce real-time translations of audio input. The authors identify the modality gap - a disparity due to differing word boundaries between audio and text - as a primary issue in the performance of offline speech translation and the application of simultaneous text translation (SimulMT) policies to SimulST.\n\nTo mitigate this issue, they propose the use of the Montreal Forced Aligner (MFA) and the utilization of audio transcription pairs for pre-training the acoustic encoder. This method facilitates a token-level cross-modal alignment, enabling the wait-k policy from SimulMT to adapt more effectively to SimulST.\n\nFurthermore, the paper introduces a robust and random wait-k-tokens strategy to optimize the SimulST task. This approach enables a single model to address diverse latency requirements and reduces error accumulation of boundary alignment during inference, thereby enhancing the robustness and efficiency of the SimulST process.\n\nResults in three standard language directions of MuST-C v1.0 indicate that the proposed method surpasses other SimulST methods, in both constrained and unconstrained scenarios (regarding external MT data), and across various latency conditions. Notably the same model is also very competitive in offline ST task. The authors carry out several ablation studies that shed light to the usefulness of each component in their model. Furthermore they showcase the ability of their method to accurately predict the length of text sequence from the acoustic information, as well as  the alignment between text and acoustic tokens.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Figure 2: consider using a variable (n) instead of an actual number (3), for the number of Transformer Layers."}, "extra_text": {"Missing_References": "I think that since SpeechUT is the current SOTA in offline ST, it should be included in Table 2 (second part / with external MT).\n\nZiqiang Zhang, Long Zhou, Junyi Ao, Shujie Liu, Lirong Dai, Jinyu Li, and Furu Wei. 2022. SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1663–1676, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics."}}]

Author rebuttal:

Meta-review: The paper proposes a new approach to simultaneous speech translation, where real-time translations are produced by a system as audio is incrementally consumed. Reviewers were impressed with the proposed method, both in terms of soundness and novelty, especially after the authors addressed some of the concerns during the rebuttal period. The reviewers also commented positively on the paper's writing and the thoroughness of the experiments. The result is a good piece of work that could have wide interest.
Decision: Accept-Main