Unified Segment-to-Segment Framework for Simultaneous Sequence Generation
Authors: Shaolei Zhang
Yang Feng
Venue: NeurIPS 2023 poster (2023)
Keywords: Machine Translation
Speech Translation
Speech Recognition
Simultaneous Generation
Simultaneous Translation

Abstract: Simultaneous sequence generation is a pivotal task for real-time scenarios, such as streaming speech recognition, simultaneous machine translation and simultaneous speech translation, where the target sequence is generated while receiving the source sequence. The crux of achieving high-quality generation with low latency lies in identifying the optimal moments for generating, accomplished by learning a mapping between the source and target sequences. However, existing methods often rely on task-specific heuristics for different sequence types, limiting the model’s capacity to adaptively learn the source-target mapping and hindering the exploration of multi-task learning for various simultaneous tasks. In this paper, we propose a unified segment-to-segment framework (Seg2Seg) for simultaneous sequence generation, which learns the mapping in an adaptive and unified manner. During the process of simultaneous generation, the model alternates between waiting for a source segment and generating a target segment, making the segment serve as the natural bridge between the source and target. To accomplish this, Seg2Seg introduces a latent segment as the pivot between source to target and explores all potential source-target mappings via the proposed expectation training, thereby learning the optimal moments for generating. Experiments on multiple simultaneous generation tasks demonstrate that Seg2Seg achieves state-of-the-art performance and exhibits better generality across various tasks.

Unified Segment-to-Segment Framework for Simultaneous Sequence Generation
Shaolei Zhang Yang Fengl.2+
Key Laboratory of Intelligent Information Processing,
Institute of Computing Technology. Chinese Academy of Sciences (ICT/CAS) University of Chinese Academy of Sciences
changshaolei20zeicc ac. en Fengyangeict.c .cc cn
Abstract
Simultaneous sequence generation is pivotal task for real-time scenarios, such streaming speech recognition, simultaneous machine translation and simultaneous speech translation, where the target sequence is generated while receiving the source sequence. The crux of achieving high-quality generation with low latency lies in identifying the optimal moments for generating, accomplished by learning a mapping between the source and target sequences. However, existing methods often rely task-specific heuristics for different sequence types, limiting the model's capacity to adaptively learn the source-tate strrett mapping and hindering the exploration of multi-task learning for various simultaneous tasks. In this paper, we propose a unified segnnent-to-egmannn framework (Seg2Seg) for simultaneous sequence generation, which learns the mapping in adaptive and unified manner. During the process of simultaneous generation, the model alternates between waiting for source segment and generating target segment, making the segment serve as the natural bridge between the source and target. To accomplish this, Seg2Seg introduces a latent segment as the pivot between source target and explores all potential source-target mappings via the proposed expectation training, thereby learning the optimal moments for generating. Experiments on multiple simultaneous generation tasks demonstrate that Seg2Seg achieves state-of-the-we performance and exhibits better generality across various tasks"
Introduction
Recently, there has been growing interest simultancous sequence generation tasks 2
7 8 9
due to the rise of real-time scenarios, such as international conferences, live broadcasts and online subtitles. Unlike conventional sequence generation [10, simultaneous sequence generation receives a streaming source sequence and generates the target sequence simultaneously. in order to provide low-latency feedback [Л]. To achieve high-quality generation under such low-latency conditions, simultaneous models must learn to establish mapping between the target sequence and the source sequence 122 and thereby identify the optimal moments for generating
Directly mapping source and target sequences is non-trivial due to inherent differences between the two sequences, such as modalities or languages, resulting in significant representational and structural gaps. For instance, in streaming automatic speech recognition (Streaming ASR) [8] 9 14, 15
[6]
speech needs to be mapped text, while simultaneous machine translation (SimulMT)
17
18 13
requires the mapping from a source language to a target language (i.e., cross-lingual
* Corrreponndig author: Yang Feng
"Code is available at: htcts:///gttuuo.cm/////ttt///e/p
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Target:
Target:
Decoder
Decoder
ssggment Teed geose
Encoder
Encoder
Source:
Source:
(a) Sequeecceto-Seqqeeee framework.
(b) Segment-to-Segmen framework.
Illustration of the conventional sequence-to- to-seuuencee framework for offline generation and the proposed segment-to-segmen framework for simultaneous generation.
Figure I:
aligmment [99 Simultaneous speech translation (SimulST) [ 2 20 21 encounters challenges
that encompass both cross-modal and cross-lingual aspects. Therefore, developing an approach to bridge source and target is critical to simultaneous sequence generation.
Existing methods for simultaneous generation often rely on task-specific heuristics to bridge the source and target sequences. For example, streaming ASR methods assume strong correlation between the target token and local speech, employing a fixed-width window to directly predict the corresponding word [22 14]. SimulMT methods consider that the source and target sequences have similar lengths, employing fixed wait-k policies 4 23 [8 or attention mechanisms [5] 24 25
establish a token-to-tole to-tkeen mapping. Such assumptions of similar length limit their ability to handle sequences with significant length differences [12). SimulST methods divide the speech into multiple
segments to overcome length differences 26 3 and then apply the fixed wait-k policy (20 27].
These task-specifit heuristics not only hinder the adaptive learning of the source-target mapping but also impede the integration of various simultaneous tasks into a unified framework, restricting the
potential of utilizing multi-task learning [28] 29 30 in simultaneous generation tasks.
Under these grounds, we aim to bridge the source and target sequences in adaptive and unified manner without any task-specific assumptions. In simultaneous generation process, the model necessarily waits for source segment and outputs a target segment alternately, with each segment comprising one or more tokens. As such, the source sequence and target sequence should correspond terms of the segment and idcally agree on the segment representation [II, enabling the segment to serve as natural bridge between source and target. In this paper, we propose a unified segment- ro-segment framework (Seg2Seg) for simultaneous sequence generation, which introduces latent segments as pivots between source and target. As illustrated in Figure given streaming source sequence, Seg2Seg determines whether the received source tokens can be aggregated into latent segment. Once aggregated, the latent segment starts emitting the target tokens until the latent segment can no longer emit any further target tokens. Seg2Seg repeats the above steps until finishing generating. To learn when to aggregate and emit, Seg2Seg employs expectation training to explore all possible source-target mappings and find the optimal moments for generating. Experiments on multiple simultancous generation tasks, including streaming ASR, SimulMT and SimulST, demonstrate that Seg2Seg achieves state-of-the-an performance and exhibits better generality across various simultaneous tasks.
2 Related Work
Streaming ASR Recent streaming ASR methods primarily rely on two approaches: transducer and local attention. Transducer involves a speech encoder and label predictor, which are aligned via a joiner to determine whether to generate [32 33 34). Local attention approach utilizes monotonic
attention to determine whether to generate based on the speech within a local window [6] 35 22].
Moreover, various methods have been proposed to reduce the latency based on these two approaches by optimizing the alignment process
36
38 39 40
SimulMT Recent SimulMT methods are mainly based on pre-defined rules or alignment mecha- nisms. For pre-defined rules, Ma et 4 proposed wait-k policy, which waits for k source tokens before alternately waiting/generating one token. Some methods were proposed to improve the flexi-
bility of fixed rules through training 23
42 simultaneous framework 45 the ensemble
of wait-k [46, [8] or post-evaluation
47 For alignment mechanisms, previous works employ

---

monotonic attention S 24 311 Gaussian attention [25, binary search [48], non antaaoorrrrrssvvee
structure 49 or hidden Markov models [33 learn the alignments between the source and target token-to-token, and make waiting or generating decisions accordingly.
SimulST Recent SimulST methods focus on the segmentation of speech 50
Ma et al.
[26] proposed fixed pre-decision to divide speech into equal-length segments, and applied SimulMT methods such as wait-k 4 and MMA [24]. Some other methods first use CTC results [3 20],
27 to detect the number of words in speech, and then apply the wait-k policy. Further, Zhang and Feng [53] proposed ITST, which judges whether the received information is sufficient for translation. Zhang et al. [1] proposed MU-ST, which constructs segmentation labels based on meaning units and uses them to train a segmentation model. Zhang and Feng [54] proposed differentiable segmentation (DiSeg) to directly learn segmentation from the underlying translation model via an unsupervised manner.
ASR results [52] or integrate-and-fiie
Previous methods for simultaneous generation often involve task-specific heuristics, which hinder adaptive learning and limit their applicability other tasks. The proposed Seg2Seg utilizes the latent segments as pivots to achieve fully adaptive learning of source-target mapping. Furthermore, Seg2Seg serves unified framework for various simultaneous generation tasks, making multi-task learning in simultaneous generation feasible.
3 Method
In this paper, we propose a segment-to-segne -eeemmet framework (Seg2Seg) to map the source sequence to the target sequence, using latent segments as the pivots. With the latent segments, Seg2Seg can adaptively learn to map source to target, enabling it to find the optimal moments to generate the target tokens during the simultaneous generation process. Details are introduced in the following sections.
3.1 Mapping with Latent Segments
Seg2Seg leverages the Transformer (encoder-decoder) [55] as the backbone, and further converts the sequence- to vo-sequence framework the segment-to-see teeeemmentt frame- work by introducing latent segments. Formally, we denote the source sequence x 1 2,} with length J, and the target sequence as y = (vi,... ... yr } with length T. In Seg2Seg, the source tokens are first aggregated into several latent segments (source tokens=>laten segment), and then the latent segment emits the target tokens (latent segment=>target tokens), as shown in Figure 2
aggregate
X
seg
011=1
621 1
0g=0
3
byg=1
a==1
5
0
latent segment
bys=l
Source tokens Latent segment For aggregation, Seg2Seg produces a Bernoulli variable a, for each source token ž, to determine whether the currently received source tokens can be aggregated into segment. An ag- gregation probability a; predicted as the parameter for the variable aj. calculated as:
Figure 2: Diagram of source-target map- ping with latent segments. The arrows in color and gray represent the mapping in inference and training, respectively.
a2 = sigmoid (FFN (Rep (zz))),
aj Bernoulli (az),
(I)
where FFN (.) is feed-forward network, Rep (1z)) is the representation of xy,, and oj is aggregation probability at Ty. As shown in Figure 2 if a, = O. Seg2Seg waits for the next input, otherwise, aggregates the tokens received after the previous segment into new segment. Once latent segment is aggregated, we calculate its representation by summing the representations of all the source tokens it contains. Specifically, the representation of the tth latent segment is denoted as seg,, calculated as:
seg Wasc--8eg
Rep (2z),
IY ExogE
where Ware-+eeg is the learnable projection from source to latent segment space.
Latent segment = Target tokens Given latent segment representation, Seg2Seg judges whether seg, can emit y by producing Bernoulli variable bi with the emission probability 8i,
3

---

E sogge
(segi
f 2y
P(u E E
p(zy (segr. s0ge
Ma
(a) Hard mapping in inference.
(b) Expected mapping training.
Figure 3: Illustration of source-target mapping in inference and training. (a) The color indicates which latent segment the token belongs to, and the final matrix indicates whether the model receives Iy when generating Y/ (i.e., the cross-attention, where the white space is masked out because those source tokens are not received yet.). (b) The shade indicates the probability that the token belongs to different latent segments, and the final matrix indicates the probability that ys can pay attention zy.
parameter. Specifically, by is calculated dot- dot-product form:
Wi Rep (yi
segk
8i sigmoid
bi Bernoulli (3xx),
Vd
where Wigt->seg the learnable projection from target to latent segment space, and is the input dimension. During emission, if bix 1, Seg2Seg generates y; based the current received source tokens, otherwise stops emitting and waits for the next input. Take Figure aa an example, y/ will not be emitted from seg, b31 0. After aggregating sog2. V/3 emitted from seg2 b32
Overall, Seg2Seg alternates between waiting for enough source tokens to aggregate latent segment (i.e., wait until aj 1). and outputting the target tokens until the current latent segment can no longer emit any further tokens (i.e., output until bsk = 0). Take the mapping Figure for instance, Seg2Seg waits for 2 source tokens and generates 2 target tokens, then waits for 3 and generates tokens, then waits for 1 and generates token. Figure 3(a) gives the corresponding mapping in matrix form, where the final matrix indicates whether the model receives z, when generating y/-
3.2 Training
During training, Seg2Seg tends to learn the aggregation and emission in an adaptive manner. However, significant challenge arises from the use of Bernoulli variables aj and bi for aggregation and emission, which prevents the back propagattnt [76 to the aggregation probability ay and emission probability Ba- To address this issue, we propose expectation training that employs aj and Aix instead of Bernoulli variables to calculate the expected mapping, which can be jointly trained with the underlying Transformer model. As illustrated in Figure 3 in expectation training, the source tokens and target tokens are no longer forced to be associated with single latent segment, but rather can belong multiple latent segments by probability.
For the aggregation process from source tokens to latent segment, we introduce p p2r, E segg) to represent the probability that zy belongs to the latent segment segg. Since the aggregation process is monotonic with the streaming source sequence, i.c., which segment æ, belongs to is only related to Ij-1. P. (x, E seg,) can be calculated via dynamic programming:
p E sega) = p (zj-1 segt-1) x aj-1 + (xz-1 seg+) (1 a3-1)
We consider all possible latent segments in the expectation training, so ranges from 1 J (i.e.. aggregate at most segments with one token in each segment), even if the source tokens may belong the later latent segment with small probability, as shown in Figure 3(b) With p (I. E sogg), calculate the expected representation of latent segment by weighting all source tokens:
seg, Warc-segg Σ (x, E seg,) x Rep (2,).
(5)
For the emission process from latent segment to target tokens, we introduce (y: E seg,) to represent the probability that yı can be emitted from latent segment seg,. Since the emission process is mono-

---

tonic with the simultaneous generation, (is E segk) can be calculated via dynamic programming:
p (yr E segg) Bigk Σ p p(v-1 E segi)
II m=f
Br.m)
(6)
We give detailed introduction the dynamic programming algorithm in Appendix A
Learning Mapping To adaptively learn a and B. we jointly train p (zz E seg,) and p E seg,) with Transformer via the cross-entropy loss Lee. During inference, each target token in Seg2Seg longer focuses on all source tokens, but can only pay attention to the source token within the same latent segment the previous segments (i.e., the current received tokens), shown in Figure (aa)
So in training, we calculate the probability that yi can pay attention to Ij, denoted as Mizi
Maj =
Ep(rs p (y
segu) xp (xy E {seg1. segr))= =(ad))=Ep(  segx) x Σ p (ay segi). (7)
Then, we multiply the mapping Mi, with the original cross- aesss-attention 5 58 and normalize it to
get the final attention distribution, which is used to calculate the expected target representation. By jointly training mapping and generation via the cross-entropy loss Lee, Seg2Seg will assign higher Mij between those related source and target tokens, thereby learning reasonable mapping.
Learning Latency Besides learning mapping for high-quality generation, also introduce latency loss Ltatency to encourage low latency. We utilize two commonly used latency metrics, consecutive wait (CW) [17 and average lagging (AL) [4], to calculate the expected latency of Seg2Seg, where CW measures the number of latent segments (i.e., streaming degree (25]), and AL measures the lagging of target token (i.e., lagging degree). Therefore, Llatency calculated as:
Llatency = Ccw (ox) + CAL (M),
x ly
[
)
where
Cew (a)
Σ
aj
Σ MaxPool ai
A
|>
1=1
(8)
2
y
CAL (M)
My.
For the number of latent segments Cew (ax), following Zhang and Feng 154 we constrain Seg2Seg
via the expected segment number E aj and the uniformity of aggregation, where MaxPool (-) For the expected lagging CAL (M), we the max polling operation with kernel size of X
constrain the expected lagging D Mij of target token yt. À is a hyperparameter that controls the overall latency of Seg2Seg. A larger encourages Seg2Seg to aggregate more latent segments, thereby achieving low latency. When A + O, the number of latent segments decreases and latency becomes higher, finally degenerating into a sequence- sequence-to-sequene to-seqeence framework when 2=0.
Overall, the total training objective Seg2Seg is the trade-off between Lce for generation quality and Llatency for generation latency, calculated as:
L Lee + Liateney
3.3 Inference
In inference, we set a, = 1 when a, 2 0.5 and bu: when 8q, 2 0.5 without sampling 24 [3] Algorithm iillustratss the specific inference process of Seg2Seg. Given streaming source sequence, Seg2Seg continuously repeats the process of aggregating source tokens into a latent segment (lines 2-6) when a, 1 and then emitting target tokens from the latent segment (lines 8-12) while bi until the generation process is completed.
Owing to generating the target sequence in units of segment, it is natural for Seg2Seg to use beam search inside each target segment. Therefore, the following experiments, we set the size of the beam search for each segment to 5.
5

---

Algorithm Inference of Segment-to-Segment Framework
Input: Source sequence X.
Output: Target sequence y. Initialization: Received source sequence x = []: Target sequence y= [[[bas)]; Index j-0, i= 1,
I: while 0i-1 # (eos) do
3: while a, = and + x do
D WAIT
4:
Wait for the next source token 2j+1;
5:
- x
05++1
6:
j j+1+1;
7:
end
8:
Calculate representation latent segment segx according to Eq.
10:
11:
while (bea == 1 & == x) and 8x-1 # (eos) do
D Generate
12:
Generate the target token based on x; gr: 5 + --1+1
14:
15:
end
17: c-
18: end
19: return y;
4 Experiments
4.1 Datasets
We conduct experiments on the most common benchmarks of three representative simultaneous generation tasks, including streaming ASR, SimulMT and SimulST.
Streaming ASR We apply LibriSpeect benchmark (59] which consists of 960 hours English audio. We use dev-clean (5.4 hours) and dev-other (5.3 hours) as validation sets, and test-cl lan (5.4 hours) and test-other (5.1 hours) as test sets, where test-othed set contains more noisy audio. For speech, use the raw 16-bit 16kHz mono-channel audio wave. For text, SentencePiece [60] to generate a unigram vocabulary of size 10000.
SimulMT We apply WMT1 German->Englin (De->En) benchmark, including 4.5M sentence pairs for training. We use newstest2013 as validation set (3000 pairs), and newstest2n as test set (2169 pairs). 32K BPE 61 is applied and vocabulary is shared across languages.
SimulST We apply MUST- English- (En->De) (408 hours, 234K pairs) and English Spanish (En->Es) (504 hours, 270K pairs) benchmarks 62] We use dev as validation set (1423 pairs for En-*ee, 1316 pairs for En En- and tst -COMONN as test set (2641 pairs for En-+De, 2502 pairs for En->Es), respectively. The pre-processing is the same as streaming ASR tasks.
4.2 Systems Settings
We conducted experiments on several strong baselines for all three tasks, described as follows.
Offline [55] model waits for the complete source sequence before generating the target sequence. Offline model is decoded with beam 5.
Streaming Automatic Speech Recognition (Streaming ASR)
T-T 34 uses Transformer Transducer to determine waiting/generating via alignments from the joiner between the speech encoder and text predictor. Some methods, including ConstAlign [36), FastEmit and SelfAlign (38 proposed to further reduce the latency of the Transducer.
37
wmt 5/

---

MoChA [22] applies monotonic chunkwise attention to generate the target token based on the speech
within local window. Various training methods, such as DeCoT 39 MinLT [3] and CTC 440
are proposed to further constrain the latency of MoChA.
# Simultaneous Machine Translation (SimulMT)
Wait-k [4] first waits for k source tokens, and then alternately generates and waits for one token. trains wait-k model via randomly sampling different & between batches. MoE Wait-k [8] applies mixture-of-expert (MoE) to learn multiple wait-k policies during training. Adaptive Wait-k 46 trains a set of wait-k models (e.g., from wait-1 to wait- 13), and heuristically composites these models based on their outputs during inference.
Multipath Wait-k 23
MMA 24 applies monotonic multi-heat heat attention and predicts a variable to indicate waiting or generating, which are trained through monotonic attention [35].
introduces Gaussian multi-head attention and uses a Gaussian prior to learn the alignments via attention. With alignments, GMA decides when to start translating based on the aligned positions.
GMA 25
GSiMT 63
generates waiting/generating decisions, and considers all possible decisions in training.
HMT
proposes Hidden Markov Transformer, which uses a HMM to correspond translating moments with the target tokens, thereby learning the optimal translating moments for generating.
6
Simultaneous Speech Translation (SimulST)
Wait-k, MMA 26 for SimulMT can be applied to SimulST by making fixed pre-decision to split the speech into 280ms durations, where each duration corresponds one word.
generates n tokens every n² 280ms to address the issue of length differences between speech and text. We 2 following their best result.
Wait-k-Stride-n [20]
SimulSpeech
3 divides the speech based on a CTC word detector, and then applies wait-k policy. uses the shortest hypothesis in ASR results as word number, and then applies wait-k policy,
SH 25
TeanTrans 20 detects the word number in the streaming speech via counting the blank in CTC results, and then applies the wait-k-stride-n policy.
MMA-CMDR 64 incorporates cross- modal decision regularization MMA, which leverages the transcription of speech to improve the decision of MMA.
MoSST 27 uses the integrate-and- and-fiiin method (65 to segment the speech based the cumulative acoustic information, and then applies the wait-k policy.
ITST 53 quantifies the transported information from source to target, and subsequently determines whether to generate output based on the accumulated received information.
MU-ST 21 trains external segmentation model based on the constructed data to detect the meaning unit, and uses it to decide whether to generate.
DiSeg (54] jointly learns the speech segmentation with the underlying translation model via the proposed differentiable segmentation in an unsupervised manner.
All implementations are adapted from Fairseq Library [66]- In Seg2Seg, we use the standard Transformer-Base encoder and 6 decoder layers) (55) for SimulMT. For streaming ASR and
SimulST, we replace the word embedding layer in Transformer-Bast
with a pre-trained Wav2Vec2. 67 to extract the acoustic embedding, and the rest remains the same SimulMT.
Evaluation
We use S SimulEval
688 to evaluate the quality and latency of simultaneous gener- word error rate (WER)
ation. For streaming ASR, following Inaguma and Kawahara [40], for the quality and the mean alignment delay for the latency, which considers the average difference between generating moments and ground-truth alignments. For SimulMT, following Ma al. [4] and Zhang and Feng 13, we use BLEU [69] for the generation quality and average lagging (AL.) 4
for latency, which measures the average offsets that outputs lag behind inputs (using token the unit of AL). For SimulST, following Ma et al. (26), we use sacreBLEU [70) for the generation quality and average lagging (AL) for latency (using millisecond ms as the unit of AL). Refer Appendix Cl the detailed calculation of the latency metrics.
d. atl.aaattuucilleeeeee//arrreraea http://wwwwveeeammm.aee.em

---

IMI
leglog
Averag Laggle//At.
Abstug Lagalng (ll,
Averuga Rigiigg
Figure SimulMT results of quality v.s. latency (AL, tokens) on WMT15 De->En.
(a) En-
(b) En-Es
Figure 5: SimulST results of quality v.s. latency (AL, ms) on MuST-C En-*De and En-+Es.
4.3 Main Results
Results Streaming ASR Table the results on streaming ASR, where Seg2Seg ex- hibits better trade-off between generation qual- ity and latency. Compared with T-T using the transducer align speech and text token-to-token 34]. or MoChA generating text based on local speech (22]. Seg2Seg adaptively learns source- target mapping through latent segments, thereby performing better.
Table 1: Streaming ASR results.
WER())
Systems
Latency(L)
clean
other
Offline
3.51
8.49
T-T
3.40
9,50
610
+ConstAlign
4.00
11.10
328
+FastEmit
4.00
10.40
195
+SelfAlign
4.00
10.70
145
MoChA
4.80
14.20
320
Results on SimulMT
Consistent with the pre-
+CTC
4.00
11.20
240
vious methods 4
24 13), adjust the value of +DeCoT
3.90
11.60
240
(refer to Eq. 8 to show the performance of Seg2Seg under varying latency. Figure shows that Seg2Seg outperforms previous SimulMT methods at all latency. Compared to methods based on pre-defined rules, such as wait-k and MoE wait-k, Seg2Seg more flexible in making generating/waiting decisions, achieving significant advantages. Other methods, such as MMA, GMA and HMT, align the target and source token-to- token. However, since the alignment between the two languages may not one-to-one 4 some local reordering and multi-word structures can affect the performance [12]. By mapping source to target at the segment level, Seg2Seg is more line with the simultaneous generation process and mitigates these issues, ultimately achieving ttate-of-the-art performance.
+MinLT
4.50
11.70
320
Seg2Seg
3.55
8.73
324
Results SimulST For the most challenging SimulST in Figure [5] Seg2Seg achieves state-of-the art performance, especially at low latency. Most of the previous SimulST methods either segment the speech into fixed lengths 26 64 detect the number of words in the speech 52 20 (7] and then apply a wait-k policy, where both non-derivable word detection and wait-k policy hinder the adaptive learning of the model. Owing the proposed expectation training, Seg2Seg completely differentiable and able to jointly learn the mapping from the source to the latent segment and from the latent segment to the target, thereby finding the optimal moments for generating.
4.4 Superiority of Unified Framework on Multi-task Learning
In the sequence-tor woatenncettt-eqquuee se-queeece framework, multi-task learning composed of ASR, MT and ST is shown to improve the performance on difficult tasks (c.g. speech translation) by sharing knowledge among different tasks [1 72 73.. However, in previous simultaneous generation methods, different tasks often involve different architectures and heuristics, leaving no room for multi-task learning. Owing not involving any task-related heuristics, the proposed unified provides a possibility to apply multi-task learning in simultancous generation. In Seg2Seg, multi- task learning can include streaming ASR, SimulMT and SimulST, and these three tasks share all parameters, except that SimulMT has a text embedding and streaming ASR/SimulST have shared speech embedding.
framework

---

Figure demonstrates the improvements brought by multi- task learning on the most challenging SimulST task. By employing multi-task learning in unified framework, Seg2Seg can achieve further improvements. Specifically, jointly training with streaming ASR yields more discernibee improvements, which is mainly because the monotonic properties between speech and text inherent in streaming ASR assist SimulST in learning the source-target mapping [52 27 64 54]. Therefore, the unified Seg2Seg facilitates the sharing of knowledge among various simultaneous tasks through multi-task learning and is helpful for the difficult tasks, such as SimulST.
-Wilrr por add fooemnn
-WB fir JCT (47
Agiiig Shoddt Zearing Skadiit
1009
3530
2500
300
350
Figure 6: SimulST results on MuST-C En-+De with multi-task learning.
Analysis
We conducted extensive analyses to investigate the specifie improvements of Seg2Seg. Unless otherwise specified, all the results are reported on SimulST with MuST-C En->De test set, which is more difficult simultaneous generation task. Refer to Appendix for extended analyses.
5.1 Improvements of Adaptive Learning
Seg2Seg learns the mappings from source to segment and from segment to target in an adaptive manner, with- out any task-specific assumptions. To verify the effect of adaptive learning, respectively replace the source-to- segment and segment-to-targer map- pings with heuristic rules, such fixed-length segment (i.e., fixed-seg) policy
Supfay iobpit
Lakl west Bucher onch
Arcrag (AA..
(26) and hott-k/waat-k-stridee (i.e., fixed-emit) [4] 20), and show the SimulST En results in Figure
(a) who adaptive aggregation.
(b) w/o adaptive emission.
Figure 7: Improvements brought by adaptive learning.
The results show that adaptive learning significantly outperforms heuristic rules. Compared with dividing the source into fixed lengths of 200/280/360mss, Seg2Seg can adaptively determine whether the received source token can be aggregated into a latent segment, bringing about BLEU improve- ment. Compared with the rule-based wait-k policy, Seg2Seg judges whether to emit the target token based on the latent segment, thus finding more reasonable generating moments
5.2 Quality Aggregation and Emission
Seg2Seg learns aggregation and emission adaptively, we further explore the quality of aggregation and emission, respectively. We apply streaming ASR and SimulMT tasks for evaluation. The detailed calculation of the metrics for aggregation and emission quality are shown in Appendix B.1
Aggregation Quality To verify whether Seg2Seg can aggregate the source tokens into latent segment the appropriate moments in streaming ASR, following Zhang and Feng [54], we conduct experiments on the Buckeye datase (78). which is speech segmentation benchmark with the annotated word boundaries.
Table 2: Segmentation accuracy of Seg2Seg.
Systems
P(t)
R(t)
R-value(t)
K-Means [74]
30.7
18.0
39.7
BES GMM 775
31.7
13.8
37.9
VQ-CPC 76
18.2
54.1
-86.5
VQ-VAE 76
16.4
56.8
-126.5
Table 2 shows the segmentation quality of Seg2Seg with some segmentation baselines, and the metrics include precision (P), recall (R) and
DSegKNN
30.9
32.0
40.7
Fixcd(280ms)
28.1
16.3
38.4
R-value (comprehensive score) [7].. Seg2Seg achieves better segmentation precision and higher
Seg2Seg
41.1
18.1
41.2
htttp:::/aacceeeecoppuusooo

---

comprehensive score R-value, showing that Seg2Seg can perform aggregation and segment the speech at reasonable moments (i.e., token boundaries instead breaking the continuous speech of word (27).
Emission Quality To verify whether the model emits at sonable moments, we follow Zhang and Feng [31] to evaluate the emission quality in SimulMT based on alignments. We apply RWTH De-En alignment dataset, and calculated the proportion of the model emitting the target token after receiving its aligned source token, used as the emission accuracy. Figure & shows that Seg2Seg can receive more aligned source tokens before emitting under the same latency, meaning that Seg2Seg finds more favorable moments for generating and achieves high- quality generation.
Emiisiion Accurccy.
1,00
0.55
Utfize
Septiigg
0.90
Walk
Averago Lagging (AL, token)
Figure 8: Emission accuracy of Seg2Seg in SimulMT.
5.3 Effect of Latent Segments
To investigate whether the latent segments can act as bridge between the source and target, we calculate the representations of the source tokens E2 T)). target tokens Eggeseg y:). and latent segment seg, within each segment during SimulST on En->De We then apply the T-SNE dimensionality reduction algorithm to project these representations into a 2D space. By doing so, we obtain bivariate kernel density estimation of the representation distribution of source segment, target segment and latent segments, depicted in Figure 9 The visualization clearly demonstrates that the latent segment locates between the source and target sequences in the representation space, effectively serving as a bridge connecting the source and target.
Table 3: Representational similar-
ity with the latent segment.
Similarity
source + target
0.53
source + segment
20.01
segment <> target
14.66 %
Furthermore, calculate the cosine similarity between the rep- resentations of the source, target and latent segments, as shown in Table It is evident that the similarity between the source and target representations is low, posing a challenge for the model to directly map the source to the target. Conversely, the similarity between the latent segment and the source, as well as the latent segment and the target, is significantly higher. Hence, by introducing the latent segment as a pivot, the model can more easily learn the mapping from the source the latent segment, and subsequently from the latent segment to the target, thereby finding the optimal moments for generating and achieving better performance.
1
Figure Bivariate kernel density estimation visualization on the rep- resentations of source, target and latent segment.
6 Conclusion
In this paper, we propose unified segment-to-segmeen framework for simultaneous sequence genera- tion, which bridges the source and target sequences using latent segments pivots. Unified Seg2Seg enables the handling of multiple simultaneous generation tasks and facilitates multi-task learning. Experiments and analyses show the superiority of Seg2Seg on performance and generalization.
Limitations
The proposed Seg2Seg employs the encoder-decoder architecture as its backbone, and exhibits better generality aeross multiple simultaneous generation tasks. In addition its primary application generation tasks, the encoder (aggregation process) or decoder (emission process) of Seg2Seg can also be separately used for some real-time tasks based on encoder-only or decoder-only architecture, such as streaming tagging and online parsing. We leave this for further exploration in future work.
10

---

Acknowledgements
We thank all the anonymous reviewers for their insightful and valuable comments.
References
[1] Christian Fügen, Alex Waibel, and Muntsin Kolss. Simultaneous translation of lectures and speeches. Machine Translation, 21(4))209-25) 252, 2007. ISSN 09226567. 15730573. URL
springer...... https:://inn http:/inn.s.riierr....mm com/ art.aaattiiiiooee 0 1007 0000//000000000-997777
[2] Yusuke Oda, Graham Neubig, Sakriani Sakti, Tomoki Toda, and Satoshi Nakamura. Optimizing segmentation strategies for simultaneous speech translation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 551-556, Baltimore, Maryland, June 2014. Association for Computational Linguistics. doi: 10.3115/v1/P14-000 URL bEtp:::::22. ac http//aaaanncoooo..oooooooooo.aa .rg. 2090
[3] Yi Ren, Jinglin Liu, Xu Tan, Chen Zhang, Tao Qin, Zhou Zhao, and Tie-Yan Liu. Simul- Speech: End-to-end simultaneous speech to text translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3787-3796, Online, July 2020. Association for Computational Linguistics. doi: 10. 1/053///2020.ac-mmaain https: //ac accl ant hontholoy...ooo org/ 20200.cc2 main. 350
URL
[4] Mingbo Ma, Liang Huang, Hao Xiong, Renjie Zheng, Kaibo Liu, Baigong Zheng, Chuanqiang Zhang, Zhongjun He, Hairong Liu, Xing Li, Hua Wu, and Haifeng Wang. STACL: Simultaneous translation with implicit anticipation and controllable latency using prefix-to-prefix framework. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3025-3036, Florence, Italy, July 2019, Association for Computational Linguistics. doi: 1/118653/1/11111222 URL https: www. a wwwwww..wwwgg......ee....... ac..eeb.orgg/aathooorr////e
1289
[5] Naveen Arivazhagan, Colin Cherry, Wolfgang Macherey, Chung-Cheng Chiu, Semih Yavuz, Ruoming Pang, Wei Li, and Colin Raffel. Monotonic infinite lookback attention for simultaneous machine translation. In Anna Korhonen, David Traum, and Lluís Mårquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, pages 1313- 1323, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/ P19-1126. URL https: aatp//alaannntttt http://aaaaaatooo..oooooo///////
1126
[6] William Chan and Ian Lane. On Online Attention-Based Speech Recognition and Joint Man- darin Character- Pinyin Training. In Proc. Interspeech 2016, pages 4404-3408, 2016. doi: 014457//nnersppech2211-6663 interspeech_ chanl 6c lerllll___iintteeee htw__intesssseehchhhhet html
URL htL.httpss:/ wwwp///wwww.mm i sca sarreechoog//rretee
[7] Junfeng Hou, Shiliang Zhang, and Li-Rong Dai. Gaussian prediction based attention for online end-to-end speech recognition. In Proc. Interspeech 2017, pages 3692-3696, 2017. doi: 10.
doi. 21437/Intesppeech.e 2017-751. URL httttp///oo..aaa..ooo///.222///. 2017-751
[8] Bo Li, Anmol Gulati, Jiahui Yu, Tara Sainath, Chung-Chema Chiu, Arun Narayanan, Shuo- Yiin Chang, Ruoming Pang, Yanzhang He, James Qin, Wei Han, Qiao Liang, Yu Zhang. Trevor Strohman, and Yonghui Wu. A better and faster end-to-end model for streaming asr. In ICASSP 2021 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5634-5638, 2021. doi: 10.1109/CAAS9P397788.00011 222199998822221 URL https 7/ //ieeexplore. leee org/ 1 /1////443349/9// 0111133///41333///10
pdf [9] Bo Li,, Shuo-yiin Chang, Tara N. Sainath, Ruoming Pang, Yanzhang He, Trevor Strohman, and Yonghui Wu. Towards fast and accurate streaming end-to-end asr. In ICASSP 2020 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6069-6073, 2020. doi: N0.1109//AASS44077660000 200.///...220004575....
[10] Ilya Sutskever, Oriol Vinyals, and Quoe V Le. Sequence to sequence learning with neural networks. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 27, Curran Associates, Inc., 2014. www. neur URL al2a555af2772200539eeeee3333
/ttp:/ppprrlleee//ppp//0///////0 pdf

---

[11] Jiatao Gu, Graham Neubig, Kyunghyun Cho, and Victor O.K. Li. Learning to translate in real- time with neural machine translation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Pupers, pages 1053-1062, Valencia, Spain, April 2017. Association for Computational Linguistics. URL. https:: wwwp///www... ac web. orgg antho wot.thooooy///11110/
[12] Shaolci Zhang, Shoutao Guo, and Yang Feng. Wait-info policy: Balancing source and target at information level for simultaneous machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2249-2263, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. 10.18653/v1/2022. ceeting--emnl...e URL 166
[13] Shaolei Zhang and Yang Feng. Hidden markov transformer for simultaneous machine translation.
In The Eleventh International Conference on Learning Representations, 2023. URL https
7/ /ooeereview net / wtw.://wwww..eeettooreeee------- 9YCFF va
[14] Hirofumi Inaguma, Masato Mimura, and Tatsuya Kawahara. Enhancing Monotonic Multihead Attention for Streaming ASR. In Proc. Interspeech 2020, pages 2137-2141, 2020. doi: 10. 21 21437/mterspeecch.2 2020- 1780. URL //attp://áx... doi. ottp./.............////e.ee..e..
2020-1780
[15] Niko Moritz, Takaaki Hori, and Jonathan Le. Streaming automatic speech recognition with the transformer model. In ICASSP 2020 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6074-6078, 2020, doi: 10.11091CASSP40776 2020.9054476. URL htttp:::/ieeeppppeeeeeee
ieee. org/ wwtp://eerrr//cocccoeeeeeeeeeeee http///ccuuettttceeeee /9044476
[16] Jiahui Yu, Wei Han, Anmol Gulati, Chung-Cheng Chiu, Bo Li, Tara N Sainath, Yonghui Wu, and Ruoming Pang. Dual-mode (asr): Unify and improve streaming (asr) with full- context modeling. In International Conference on Learning Representations, 2021. URL
https:://penreet
net/ forum heddd==dddqfceeeeeeeeeeed
[17] Kyunghyun Cho and Masha Esipova. Can neural machine translation do simultaneous transla- tion? CoRR,
2016.
02012
[18] Shaolei Zhang and Yang Feng. Universal simultaneous machine translation with mixture-of- experts wait-k policy. In Proceedings of the 202/ Conference on Empirical Methods Natural Language Processing, pages 7306 7317, Online and Punta Cana, Dominican Republic, Novem- ber 2021, Association for Computational Linguistics. doi: 10. 18653/v1/2021 http:////22211.1mmmmaaam emnnlpmmain581 URL htl.https:/// ac antho ogy. - 20rg/20211 emn ma n 581
[19] Shaolei Zhang, Qingkai Fang, Zhuocheng Zhang. Zhengrui Ma, Yan Zhou, Langlin Huang, Mengyu Bu, Shangtong Gui, Yunji Chen, Xilin Chen, and Yang Feng. Bayling: Bridging cross- lingual alignment and instruction following through interactive translation for large language models, 2023. URL https: /tttpp:///rxiv.... org/ abs//333 10968
[20] Xingshan Zeng, Liangyou Li, and Qun Liu. RealTranS: End-to-end simultaneous speech translation with convolutional weighted-sshrnking transformer. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2461-2474, Online, August 2021.
Association for Computational Linguistics. doi: 10. 122118655/11022.fnnnnnngg.... findings--cc..218. URL
https://aat
aataannthhhoo /ttpss://aclantholooggo org 2021 www.0....inn..... findings acl. .28
[21] Ruiqing Zhang, Zhongjun He, Hua Wu, and Haifeng Wang. Learning adaptive segmentation policy for end-to-end simultaneous translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 7862-7874 Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 18653/v 1/2022. acl- long.542. URL https: / ac anthology wwt.ogy..rg//222// acl http:///aaaaattllloooo//////////
542
[22] Chung-Cheen Chiu and Colin R Monotonic chunkwise attention. In International Con- ference on Learning Representations, 2018. URL /tttps://operreviit net/ fet/foorm?
id Cd-Hko85ploo
12

---

[23] Maha Elbayad, Laurent Besacier, and Jakob Verbeek. Efficient Wait-k Models for Simultancous Machine Translation, 2020. URL http://cxxx....//////... http:///s....c.oooo../..///////.
2020-1241
[24] Xutai Ma, Juan Miguel James Cross, Liezl Puzon, and Jiatao Gu. Monotonic multihead attention. In International Conference on Learning Representations, 2020. URL https
//apenreview. net/ forum?id Hyg
[25] Shaolei Zhang and Yang Feng. Gaussian multi-head attention for simultaneous machine translation. In Findings of the Association for Computational Linguistics: ACL 2022, pages 3019-3030, Dublin, Ireland, May 2022. Association for Computational Linguistics. URL https: / http///cclattoooo...oroorrr 202/ 2022. http:///222...nnnnn... findings-aad acl.
[26] Xutai Ma, Juan Pino, and Philipp Koehn. SimulMT SimulST: Adapting simultaneous text translation to end-to-end simultaneous speech translation. In Proceedings of the Ist Conference of the Asia-Pacifi Chapter of the Association for Computational Linguisties and the 10th International Joint Conference
Natural Language Processing, pages 582-587, Suzhou, China, December 2020. Association for Computational Linguistics. URL https
7/aclants ogy. org/ 2020. aac 1- ma n. 58
[27] Qian Dong, Yaoming Zhu, Mingxuan Wang, and Lei Li. Learning when to translate for streaming speech. In Proceedings the 60th Anmual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pages 680-694, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 10.16655l/22 URL
[28] Sebastian Ruder. An Overview of Multi-Task Learning in Deep Neural Networks. arXiv e-prints, art. arXiv:1706.05090 June 2017. doi: 10.48550/arXiv. 170..05990..
[29] Yu Zhang and Qiang Yang. An overview of multi-task learning. National Science Review, 5(1): doi: 10.1093/ns/mwxx005. URL https:://oii.orgg///
30 43-43,300 09 2017. ISSN 2095-5138.
(G93/nsr/nwx/
[30] Antonios Anastasopoulos and David Chiang. Tied multitask learning for neural speech transla- tion. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume I (Long Papers), pages 82-91, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10. 00.18653/v1/N188 1008. htRlttttss:/ ac antho wow.:lll.....ggg.......... 8-1008
[31] Shaolei Zhang and Yang Feng. Modeling dual read/write paths for simultaneous machine translation. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 2461-2477, Dublin, Ireland, May 2022. Association for Computational Linguisties. URL
wwwpoloooooorg/222200 acl long
176
[32] Alex Graves. Sequence transduction with recurrent neural networks. 1111v:1211.3711, 2012.
arXiv preprint
[33] Navdeep Jaitly, David Sussillo, Quoe Le, Oriol Vinyals, Ilya Sutskever, and Samy Bengio. A neural transducer, 2016.
04868
[34] Ching-Feng Yeh, Jay Mahadeokar, Kaustubh Kalgaonkar, Yongqiang Wang, Duc Le, Mahaveer Jain, Kjell Schubert, Christian Fuegen, and Michael L Seltzer. Transformer--ransddueer End- to-end speech recognition with self-attention. arXiv preprint arXiv: 2019, URL https://arxix> org abs 19w...191.... 12977
[35] Colin Raffel, Minh- Minh-Thang Thang Luong, Peter J. Liu, Ron J. Weiss, and Douglas Eck. Online and linear-time attention by enforcing monotonic alignments. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 2837-2846 PMLR, 06-11 Aug 2017. URL https: hctps://proceecingg m.r....mm.......mm.....mmm...... www.spesssspe..e / w htmlll.mmll
13

---

[36] Tara N. Sainath, Ruoming Pang. David Rybach, Basi García, and Trevor Strohman, Emit- ting Word Timings with End-to-End Models. In Proc. Interspeech 2020, pages 3615-3613 2020. doi: 10.21437//mterspeecchhoo 2000 2020- 1059. URL Stl.http:// /dx do
214377
Interspeech. 20t.e
[37] Jiahui Yu, Chung- Cheng Chiu, Bo Li, Shuo-yiin Chang, Tara N Sainath, Yanzhang He, Arun Narayanan, Wei Han, Anmol Gulati. Yonghui Wu, et al. Fastemit: Low-latency streaming asr with sequence-level emission regularization. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6004-6008, IEEE, ore. leee. www../////aaaaatt///oooooooooaaa
2021. URL hwt.:httts:/eeeeee
[38] Jaeyoung Kim, Han Lu, Anshuman Tripathi, Qian Zhang, and Hasim Sak. Reducing Streaming ASR Model Delay with Self Alignment. In Proc. Interspeech 2021, pages 3440-3444, 2021. doi:
10.21457/nnersppeech..2 hwtp:/sppeeech..233222 2021-322. URL https: /www. scaa pdfs/interspeeee
202120022220 kikkm int.rsseeeccchhcccccm p
[39] Hirofumi Inaguma, Yashesh Gaur, Liang Lu, Jinyu Li, and Yifan Gong. Minimum latency
training strategies for streaming peuueecce-t-sequeeee asr. In ICASSP 2020 2020 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 6064- /ttps://///eee
6068, 2020. doi: 10. 10A00/AAAAS444776000000966666 90540998
ieee. arree.orgabbstattt/bbeeee httptaaacce/cccmmmme//00005533
[40] Hirofumi Inaguma and Tatsuya Kawahara. Alignment knowledge distillation for online streaming attention-based speech recognition. IEEE/ACM Transactions on Audio, Speech, and Lunguage Processing, 31:1371-1385, 2023. doi: 10. 1111109T
URL
https:://eeexpp.oe www.:eeeeeeeee.eeeeeeeeeee..ee http://abstraet//oocueeettt////t 96445776/
[41] Shaolei Zhang, Yang Feng, and Liangyou Li. Future-guided incremental transformer for si- multaneous translation. Proceedings of the AAAI Conference on Artificial Intelligence, 35
(12):14428- May 2021. URL httpt::////// aaai www.aa..r//a.......a..aee...eeee www.rrrrgnn....oooeeoeeeeeeeeeee http:iiidxxxxp.e wwtpo///n...ppp///...aa.... php/AAAI
article/view/17)
[42] Shaolei Zhang and Yang Feng. ICT's system for AutoSimTrans 2021:
Robust char-level simultaneous translation. Proceedings of the Second Workshop on Automatic Simultane- ous Translation, pages 1-11, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021 autosimtranss-l.... URL
autosimt rans 1.1
[43] Shoutao Guo, Shaolei Zhang, and Yang Feng. Simultaneous machine translation with tailored reference. Findings of the Association for Computational Linguistics: EMNLP 2023, Associ- ation for Computational Linguistics, December 2023.
2310 2310.1360 13588
[44] Shaolei Zhang and Yang Feng. Reducing position bias in simultaneous machine translation with length-aware framework. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 67888, Dublin, Ireland, May 2022. Association for Computational Linguistiss. 2022 2022.ac1- long. 467
httphttsp//aaaaatolloog..ooo org/
[45] Shoutao Guo, Shaolei Zhang, and Yang Feng. Glancing future for simultaneous machine translation, 2023. URI
[46] Baigong Zheng, Kaibo Liu, Renjie Zheng, Mingbo Ma, Hairong Liu, and Liang Huang. Si- multaneous translation policies: From fixed adaptive. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2847-2853, Online, July 2020. Association for Computational Linguistics. doi: 120.100///10200.lam.. www.:200....mmm.225005 URL. https: www. ac web. org ho www.hhooooyyyy/oo
ogyoogy///o//// /2000 ac ma in 254
[47] Shoutao Guo, Shaolei Zhang, and Yang Feng. Turning fixed to adaptive: Integrating post- evaluation into simultaneous machine translation. Findings of the Association for Computa- tional Linguistics: EMNLP 2022, pages ber 2022. Association for Computational Linguistics. doi: 10. 111118661//222.222 htw.:///22.nniinnneemmlpp.p www.ai2....iinngneeeeeee eww.annnnseemmpppmmmmmmm 167
Abu Dhabi, United Arab Emirates, Decem-
167. URL
14

---

[48] Shoutao Guo, Shaolei Zhang. and Yang Feng. Learning optimal policy for simultaneous machine translation via binary search. In Proceedings of the 6lst Anmual Meeting of the Association for
Computational Linguisties (Volume 1: Long Pupers), pages 2318 2233, Toronto, Canada, July 2023. Association for Computational Linguisties. doi: 10. 1/2555
URL.
130
[49] Zhengrui Ma, Shaolei Zhang, Shoutao Guo, Chenze Shao, Min Zhang, and Yang Feng. Non- autoregressive streaming transformer for simultaneous translation, 2023. URL https://
bxxiv.org/abs.t 138831
[50] Mahsa Yarmohammadi, Vivek Kumar Rangarajan Sridhar, Srinivas Bangalore, and Baskaran Sankaran. Incremental segmentation and decoding strategies for simultaneous translation. In Proceedings of the Sixth International Joint Conference on Natural Language Processing, pages 1032-1036, Nagoya, Japan, October 2013. Asian Federation of Natural Language Processing. URL httts:://aclattholl htt.a/aaaaaooooooooooooo.... 1141
[51] Vivek Kumar Rangarajan Sridhar, John Chen, Srinivas Bangalore, Andrej Ljolje, and Rathi- navelu Chengalvarayan. Segmentation strategies for streaming speech translation. In Proceed- ings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 230 238, Atlanta, Georgia, June 2013. Asso- ciation for Computational Linguistics. URL UBL_pttpes//aal.attaoooooooo//..
[52] Junkun Chen, Mingbo Ma, Renjie Zheng, and Liang Huang. Direct simultaneous speech- to-text translation assisted by synchronized streaming ASR. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 461 8-14442,, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2022 1/202 http:////2.....nnnnnnnn..oo findings-acl.406. URL org/
bettps://aclantho..oo wtt.aaatwooooooorr//000....aaaa. fttg:/222...nnnnngg.ee
[53] Shaolei Zhang and Yang Feng. Information traiiiionntranssoot-bbaeee policy for simultancous trans- lation. In Proceedings of the 2022 Conference Empirical Methods in Natural Lan- guage Processing, pages 992-1013, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. 10.18665//112222..mmmmmmmmm
URL
httpss:/acclattholooyy org/ wwwp://..../2o...ooo wmn.p-mmmmmmmmmmee.mm.mmm main 65
[54] Shaolei Zhang and Yang Feng. End-to-end simultancous speech translation with differentiable segmentation. In Findings of the Association for Computational Linguistics: ACL 2023, pages 7659-7680, Toronto, Canada, July 2023, Association for Computational Linguistics. doi: 10.18653/v122023 1t0.6////1//222....nnnn-aaaaa// findings-acl.85.. URL https: httt::://aclanchooocyy.or/////// Findings-acl. 485
[55] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz. Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998-6008 Curran Associates, Inc., 2017. URL http::/papers nips. wwtppcpppppppeeeee 7881 attention is all y-uu- need
pdf
[56] Ruslan Salakhutdinov and Geoffrey Hinton. Semantic hashing. International Journal of Approx- imate Reasoning, 50(7):969 978, 2009. ISSN 0888-613X. doi: https://doi.org/// orrg/10......0...ooo ijja/j.jjar. 2008. 11 11.006. URL httpp::///et //wp::://www../// sc encedi rect cww/ se
S0888613X000 30000000000000000000000000000000 813 Special Section Graphical Models and Information Retrieval.
[57] Jakob Foerster, loannis Alexandros Assael, Nando de Freitas, and Shimon Whiteson. Learn- ing to communicate with deep multi-agent reinforcement learning. D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors. Advances in Neural Information Processing Sys- tems, volume 29, Curran Associates, Inc 2016. URL hollitttpp::pprceeeiingg nww.iigssneesrssp http:///....................22..
[58] Shaolei Zhang and Yang Feng. Modeling concentrated cross-attention for neural machine translation with Gaussian mixture model. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1401-1411, 1, Punta Cana, Dominican Republic, November
2021. Association for Computational Linguistics. doi: 10. 111118851122221.10 11116/5/110222...nnnnnn--- findings--mmnlp.ll
URL http:://aalaathooooyyoor///...0. wtt.::22....iiniiioie ewwpssssemmpp..... 121
15

---

[59] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An ast corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5206-5210, 2015. doi: 10.1109/
ICASSP2201..1188944
[60] Taku Kudo and John Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66- 71, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10. 10.16653/v1/D18 8-1022. BRI.https:// htt.ttttpp///////oooooooooooogo wtw.eaattttooo...ogooooooo 2C12
[61] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pages 1715-1725, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653/w1/P11--166. URL
https: www ac wwe.borg//e wttpo///nnn....ggo.ooo1--1.1
[62]
Mattia A. Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi. MuST-C: a Multilingual Speech Translation Corpus. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 2012-2017, Minneapolis, Minnesota, June 2019, Association for Computational Linguisties. doi: 11111866//////// 1202. URL httph:ttp///m. //a http://acllaattoooo.orrr///////
1202
[63] Yishu Miao, Phil Blunsom, and Lucia Specia. A generative framework for simultaneous machine translation. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pages 6697-6706, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. URL htttsss//acllatthoooooooooo
emn 536
[64] Mohd Abbas Zaidi, Beomseok Lee, Sangha Kim, and Chanwoo Kim. Cross- Decision Regularization for Simultaneous Speech Translation. In Proc. Interspeech 2022, pages 116-120,
2022. doi: 10.2 1021447/nneespeech2222--0000
URL http:::/t/////t //ww: i isca speech. org/
a archive int httthi/eeennerrppsscc//////////2
htti:/innessseehhhtte html
[65] Linhao Dong and Bo Xu. Cif: Continuous integrate-and-ffe for end-to-end speech recognition.
In ICASSP 2020- 2020 IEEE International Conference on Acousties, Speech and Signal Pro- cessing (ICASSP), pages 6079 2020. doi: 10. 21019///AAAAA0777200000000.000 9054250. URL
https:://eeexppoos htt.://pppprr..eeeeeoeo///////// htt.:///.......ooooooo/////////0
[66]
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)) pages 48-53, Minneapolis, Minnesota, June 2019. Association for Computational Linguisties. doi: 10. 10.18653/1//N11-4000 URL httph:tts:/ / www. ac webb
org/ http//atthoooo//111o11
[67] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural In- formation Processing Systems, volume 33, pages 12449-12460. Curran Associates, wttppcppppppp.///c///ppppp
Inc., 2020. URL lattps://rroeeedin www.eieerrpppp Ttt./////a..22000000000000000000 Paper pdff
[68] Xutai Ma, Mohammad Javad Dousti, Changhan Wang. Jiatao Gu, and Juan Pino. SIMULEVAL:
An evaluation toolkit for simultaneous translation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 144-150. Online, October 2020. Association for Computational Linguistics. doi: 10. 10.18653/v1/0200 emnlp-demos. 19. htLpstt:::/ wwlanthooogooyoooooooooooooooota org/ emnip demos. 19
[69] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association
16

---

for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA, July 2002.
Association for Computational Linguistics. doi: 10.3115/1073383.1073333.. URL https
7 //www... ac web.
[70] Matt Post. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers. pages 186-191, Brussels, Belgium, October 2018.
Association for Computational Linguisties. doi: 10. 10.18653/111118-33111 URL https
7/ /wwwwaaclwwt www.ww.wwwwbccc.. org/ www.atotooooooooooooooooo h11- 6319
[71] Qingkai Fang, Rong Ye, Lei Li, Yang Feng, and Mingxuan Wang. STEMM: Self-learning with speech-text manifold mixup for speech translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 7050-7062, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 10.18853//11220 1/2022.
acl-long.486 URL https: http:///acaaathhooocooo aanthooottyy.........o org 2022. acl long. 486
[72] Qingkai Fang and Yang Feng. Understanding and bridging the modality gap for speech translation. In Proceedings of the 61st Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pages 35864-15881, Toronto, Canada, July 2023. Association for Computational Linguisties. doi: 10. 2/20:///ll22...looo https: / ac antho logy. org aconog 884
URL.
ac
[73] Yan Zhou, Qingkai Fang, and Yang Feng. CMOT: Cross- modal mixup via optimal transport for speech translation. In Proceedings of the 6lst Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7873-7887, Toronto, Canada, July URL
2023. Association for Computational Linguistics. doi: 10. 1205
https: //as: anthonaohhyyy. ogy. org/ 1.023aacl-2.
436
[74] Herman Kamper, Karen Livescu, and Sharon Goldwater. An embedded segmental k-means model for unsupervised segmentation and clustering of speech. In 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 719-726, 2017. doi:
10.1099/ASUU.2117.225000 URL attps:// www http://ww.kampeee kwwkkammpe.r...... com papers kamper Livescu+golddateer.coo wrwpleeerrrrru21111 pdf
[75] Herman Kamper, Aren Jansen, and Sharon Goldwater. segmental framework for fully- unsupervised large-vocabulary speech recognition. Computer Speech & Language, 16:154
174, 2017. ISSN 0885-2308.
008. URL https doi: ott.:///...........o.o......aaaa
www..sciencediree. wtt.///neeerrerc..cooooooeeeeeee
[76] Herman Kamper and Benjamin van Niekerk. Towards unsupervised phone and word segmentation using self-supervised vector-quantized neural networks. arXiv e-primss, art. arXiv:2012.07551, December 2020. URL htttps://ii.etti. att.ui.ddsbbs.... www.ddaa..aaaarrrd..eo edw/abs
2020arX1v20120753
[77] Tzeviya Sylvia Fuchs, Yedid Hoshen, and Joseph Keshet. Unsupervised word segmentation using k nearest neighbors. arXiv preprint arXiv:2204.1309. 2022. URL htttp://aaxxx
erg/ /eeaab///2224.2 2204. 13094
[78] Mark A. Pitt, Keith Johnson, Elizabeth Hume, Scott Kiesling, and William Raymond. The buckeye corpus of conversational speech: labeling conventions and a test of transcriber reliability, Speech Communication. 45(1):89 2005. ISSN 0167-6393. doi: https:
URL https:/ http://wwwwccieeeeerrr..coo com/
[79] Okko Räsänen, Unto Laine, and Toomas Altosaar. An improved speech segmentation qual- ity measure: the r-value. In 10th Interspeech Conference, Brighton, UK, September 6-10, 2009, 2009. URL wtw.thtts////tccmm htte://ciitseeeeee wecciiesseex... ist psu edu/ document? repid=repppereeeee Sype-pdf&dos 1eseii---11 0
[80] Kris Demuynck and Tom Laureys. A comparison of different approaches to automatic speech segmentation. Petr Sojka, Ivan Kopeček, and Karel Pala, editors, Text. Speech and Dialogue, pages 277-284, Berlin, Heidelherg, 2002. Springer Berlin Heidelberg. ISBN 15544458455544444115555 URL htt.htttps:/iiitt http://iiinn...innge..eee springer httpp//ee.../aaaaee///////// 100
17

---

A Dynamic Programming of Mapping
s0g1
8082
s0gg
sogs S0E6
80g)
S082 SKiS
80g4 9085 soge
aggregate with 03-1
1/
noo omiil with
22
aggregato with oi
V2
-0
23
0
0
p(ay 6ogg))
Vo
-
0
0
O pluc & suga))
24
0
0
0
V1
+0
0
0
E.g. plao eogga) piin segs.:
25
0
Vb
[1 Roa) (1 Paa) x fag
36
not from snta 1984 Stemnt fromsseg,
(a) Dynamic programming of p(æ, E Esegy).
(b) Dynamic programming of p(y: E Eseg,)
Figure 10: Schematic diagram of the dynamic programming of mapping in expectation training.
In Sec 3.2 we propose expectation training for Seg2Seg to learn when to aggregate and emit through p (zy E segk) and p (y) seg,). Given the monotonic property of simultancous sequence generation tasks, we can calculate p (æ, E seg,) and p (3: E seg,) using dynamic programming algorithm. In the following sections, we will provide a detailed explanation of the dynamic programming approach.
A.1 Source Tokens Latent Segment
Given the streaming source sequence, Seg2Seg predicts aggregation probability aj for zj rep- resent the probability aggregating the received source tokens into a latent segment at Tz. Given aggregation probability ag,, we calculate p (x, seg,) via dynamic programming.
The whole aggregation process is monotonic with inputs, which means that the (k + 1)th latent segment can only be aggregated once the 1tth segment has already been aggregated. Additionally, source token, otherwise latent segment
each latent segment must he aggregated by least without representation would be meaningless. As a result, whether Iy belongs to latent segment sega depends which segment that - is located in, consisting of situations:
If zj-1 E SegA-1: As illustrated by the red line in Figure TO(a) Zz belongs to latent segment seg, when Seg2Seg aggregate øÿ-1 with probability aj-1:
If xj-1 E seg,: As illustrated by the blue line in Figure [0(a) Ij belongs to latent segment segg (i.e., the same latent segment with æz-1) when SegZSeg does not aggregate at 2z-1 with probability - aj-1;
Otherwise: &j can not belong to seg, anyway, i.e., with probability
By combining these situations, p (are segx) is calculated as:
p (z, E segg)= = (xj-1 seg+-1) x aj-1 + (x)-1 seg,) (1 a)-1),
(10)
where the initialization is
if =
p(a1 segg)
(11)
0
if
because the first source token inevitably belongs to the first segment. With the above dynamic programming algorithm, we calculate p (zrg seg,), for I.... and k I,...
A.2 Latent Segment Target Tokens
After getting the latent segments, Seg2Seg predicts the emission probability 8ik, which indicates the probability emitting the target token y; from the latent segment segk: With the emission probability 3ik, we compute (yı E seg,) using dynamic programming well. Note that there is one difference between the aggregation process and the emission process when employing dynamic programming. In the aggregation process, each latent segment must be aggregated by at least one
18

---

source token. However, in the emission process, the latent segment has the option to not generate any target token, not all source tokens have corresponding target tokens.
Whether yi belongs to latent segment seg, depends on which segment that yi-1 is emitted from, consisting of situations:
If y:-1 E seg.: 3i is emitted from latent segment seg, with probability Bas;
If 9/-1 E seg, for / 1.... , not emitted from seg, to - and then emitted from seg,- Taking Figure [O[b] as an example, if 1/2 E seg3, the premise of ys E segs is that ys not emitted from seg3 and seg4- and is emitted from segs. Formally, the probability is calculated as:
y: is emitted from latent segment segı, when y4 is
k-1
Bik x II (1-B),m): m=/
(12)
where I (1 Be.m) is the probability that yı is not emitted from seg, segk-15 If l/-1 seg, for I 1, Yi can not be emitted from segx anyway, the emission process monotonic, i.e., with probability O.
By combining these situations, p (yr E segx) is calculated
p (yt E segu) = Bi,k Σ
p py E segi)
II (1 (em)) m=
(13)
where the initialization is
( E segg) = B1, II (1 ml,m)
(14)
With the above dynamic programming algorithm, we can calculate (v: seg,), for 1,..
and h 1,...
B Extended Analyses
B.1 Detailed Calculation of Aggregation and Emission Quality
In 5.2 we evaluate the aggregation and emission quality of Seg2Seg. Here, we give detailed calculations of aggregation and emission quality.
Aggregation Quality We verify whether Seg2Seg can aggregate (segment) the source speech sequence at the appropriate moments on the speech segmentation task [80] For our evaluation, we utilize the Buckeye dataset, where the ground-truth segmentation based on word units. Evaluation metrics consist of precision (P), recall (R), and the comprehensive score R-value. Note that since the ground-truth segmentation in Buckeye is in units of words, and the aggregation of Seg2Seg is in units of segments, which makes the segment number of Seg2Seg may be less than the ground-truth segment number, so precision can better reflect whether the aggregation moments of Seg2Seg is reasonable. R-value [79 is more robust comprehensive metric for speech segmentation task, calculated
R-value = Inil + |r2|
(15)
2
where ri V (a- 11)
R P 1
-(B
1) R
+
72
(16)
v2
A larger R-value indicates better segmentation quality, where R-value 100% if and only if P. = 100% and = 100%.
Emission Quality We verify whether Seg2Seg can emit the target token at appropriate moments in SimulMT. simultaneous machine translation, is crucial for the model to emit the corresponding target token after receiving its aligned source token (5), the alignments can be used as the basis for
19

---

judging whether the emitting moments are reasonable. Following Zhang and Feng G3o et al. 477, Zhang and Feng [33. we calculate the proportion of the ground -ouuh aligned souree tokens received before emitting as the emission quality. We apply RWTH De-FEn alignment dataset and denote the ground-truth aligned source position of y: as ds, while use 4 to record the emitting moments of y/, Then, the emission quality is calculated as:
yy
[1, a S t4
Score
where Laish =
(17)
0, > 4i
1=1
B.2 Visualization of Mapping
(a) Source tokens= aatent segment (b) Latent segment= Targen tokens
(c) Source tokens=) Target tokens
Figure 11: Visualization of the mapping with the latent segment during inference on En->De SimulST (Case #2258). The red rectangles in (a) and (b) indicate the correspondence between token and latent segment. (c) indicates the received source tokens when generating each target token. The English meaning of target sequence: Wir (_we) sprechen (_speak) nicht (_not) gerne gladlyy über (_about) unsere (_our) Geschichte (_story).
(a) Source tokens=ILatem segment (b) Latent segment=TTagge Target tokens
(c) MMs
Figure 12: Visualization the mapping with the latent segment during training on En En SimulST (Case #2258). The shade of red rectangles in (a) and (b) indicate the probability that the token belongs to different latent segments. (c) indicates the probability that y; can pay attention to æy. Note that Seg2Seg considers all possible latent segments during training, so there are more latent segments in (a) and (b) than inference.
We visualize the proposed source-targed mapping with the latent segment as the pivot in Figure I1 and 12 where the case from the most challenging SimulST task.
Inference Figure 1 shows the mapping during inference. Seg2Seg effectively aggregates lengthy speech sequence (approximately 101 tokens) into 6 latent segments, and then these 6 latent segments
20

---

emit 8 target tokens. As depicted in Figure [T
Seg2Seg exhibits high-quality aggregation by selectively splitting and aggregating the speech at appropriate boundaries, thereby preserving the integrity of the acoustic information 277. During emission, Seg2Seg exhibits the ability to generate multi-word structures, such as 'unsere Geschichte' (meaning 'our history' in English), within the same latent segment. Overall, Seg2Seg achieves good source-to-target mapping through adaptive aggregation and emission.
Expectation Training Figure shows the mapping during training. In expectation training, Seg2Seg considers all possible latent segments, with the number of segments ranging from 1 to J, shown in Figure [2(a] and [2bb By incorporating the constraint of the latency loss on the number of segments, Seg2Seg can effectively learn aggregate an appropriate number of latent segments. Furthermore, Figure [2(c) demonstrates the probability that each target token can pay attention to the source token (referred to as May (7)) This expectation aligns well with the inference process depicted in Figure [T(c) thereby highlighting the effectiveness of expectation training.
B.3
Case Study
Source
Reference
she not afraid that thwart her inclatations
Outputs
efraid that fhwort her inclinations
#Latent Segments
Received Source Inputs
Target Outputs
fow
is
2
she
3
not
afraid that
5
will thwart
her inclinations <ip-
Figure 13: Case study demonstrate the simultaneous generation process, we correspond target outputs and received source inputs to show the moments of generating each target token.
in LibriSpeech) of Seg2Seg on streaming ASR task. To
Source
Latas())
to
spieler
aus
Har@iee vey ist dieson Monat dralklé)
English menning
Lotto
player
from
Harry
tkis
monds
Jurn
Reference
a Harvey lotto player the month...
Outputs
Lotto player from Harvey is this month.
#Latent Segments
Received Source Inputs
Target Outputs
ein
ein Lot
Lot@l
ein
Lota
1 spieler
player
cin
Lota
to
spieler aus Har@a
from
cin Lota
spieler aus Hargt@ vey
Har(0g) vey
6 ein
Lotas)
spieler aus Har@s vey ist
is
7
Lo
to
spieler aus Harlaa vey diesen Monat dra this mon th </g>>
Figure 14: Case study (#1366 in WMT15 De- of Seg2Seg SimulMT task. To demonstrate the simultaneous generation process, we correspond target outputs and received source inputs to show the moments of generating each target token.
Figure 13 14 and 15 visualize the simultaneous generation process of Seg2Seg on the cases from streaming ASR, SimulMT and SimulST. In Streaming ASR and SimulST, for a clear illustration, we use an external offline speech-text alignment too to align the transcription with the speech sequence, and the aligned transcription is displayed above the speech waveform.
alignnentttuuoooii... tutorial.
htmi
21

---

Source
Transcription
challenge
which they why
Reference
Aber die Herausferderung isi verschen herauszutinden, welche and und waram wir sie behahen, und ob sie sinavoll sind.
Outputs
Aber die Berausfeddsuun ist es, au versudies herausurfidenn welche se snnd und waram maclien wis das? Und habes sie Bedcutung?
PLatent Segments
Received Inputs
Target Outputs
Abet
2
die Heruasforderung chailenge)
3
( (o)
vensrchen
[
6
berauszufinden, welche
Aguure owess
Cikey
sind und machen Lare
wir das? Und haben that? they?
10
Bedewnung!
Lanooiing
Figure 15: Case study (#1 1666 119 in MuST-C En DDe) of Seg2Seg on SimulST task. To demonstrate the simultaneous generation process, we correspond target outputs and received source inputs to show the moments of generating each target token.
Case of Streaming ASR streaming ASR often exhibit a predominantly monotonic relationship. and Seg2Seg can handle this monotonic mapping well. Seg2Seg can segment and aggregate source sequences at speech boundaries, and then emit the corresponding target sequences accurately.
shown in Figure 13 the target sequence and source sequence in
Case of SimulMT Figure 14 shows case on SimulMT. Seg2Seg can generate the target token after receiving the corresponding source tokens, e.g., generating *Lor@ @ to after receiving 'Lor@ @ to'. generating 'player' after receiving 'spieler' and generating 'Har@ vey' after receiving 'Har@ vey', which effectively ensures the generation quality of SimulMT. Besides, for some related target tokens, especially the subwords after the hpe operation, Seg2Seg can emit them together from the same latent segment, thereby achieving lower latency.
Case of SimulST Figure 15 shows case on SimulST, which more challenging as the source and target sequences involve different modalities and languages. Despite these evident differences, Seg2Seg demonstrates its capability to find the reasonable generating moments, such as generating "herauszufinden' after receiving 'figure out' in the speech, and generating 'Bedeutung after receiving 'meaning' in the speech. This mainly attributed to expectation training, which explores all possible mappings in training. allowing Seg2Seg to learn to aggregate and emit at reasonable moments. As seen, Seg2Seg aggregates the related speech frame into the same latent segment and will not break the acoustic integrity of the speech. For emission, Seg2Seg can accurately determine whether a latent segment can emit the target token, where almost all emitted target outputs are correspond to the source speech contained in the latent segment.
C Latency Metrics
For the latency evaluation simultaneous generation task, we use mean alignment delay for streaming ASR and average lagging for SimulMT and SimulST.
22

---

Mean Alignment Delay [40] is defined as the average word time difference between the ground-truti truth alignments (speech and transcription) and generating moments:
y
1
Dean
Σ (4),
(18)
where t, is the ground-trutt alignment of yıs and é, is the generating moment yt-
Average Lagging (AL) 4 evaluates the average number of tokens (for SimulMT) or speech duration (for SimulST) that target outputs lag behind the source inputs. We use to denote the generating moments of y/. and AL is calculated as:
AL
where argmin (f, [x])
(19)
y x
In addition to average lagging, we also use some other latency metrics for SimulMT and SimulST, described as follow.
Consecutive Wait (CW) I|| evaluates the average number of source tokens waited between two target tokens, ie., the number of segments:
X
CW-
(20)
Li=1
1 14:4-1200
where Lt,-t,133 50 counts the number of 4; 4--1 O. i.e., the number of segments. It is worth mentioning that the latency loss Liatency in training employs the denominator part of the CW metric, as the numerator is constant.
Average Proportion (AP) (17] evaluates the proportion between the number received source tokens and the total number of source tokens, calculated as:
AP
4
(21)
X
y
Differentiable Average Lagging (DAL))
5 is a differentiable version of average lagging, calculated
as:
yy
t max (t,, where 1,
i=
DAL
[x] i>1"
(22)
y
/x|/x/
1x))
D Numerical Results
Table and report the numerical results of Seg2Seg on SimulMT and SimulST, where À hyperparameter that controls the overall latency of Seg2Seg (refer to Eq g
Table 4: Numerical results of Seg2Seg on De SimulMT.
SimulMT WMT15
>n
CW
AP
AL
DAL
BLEU
0.4
1.78
0.60
2.21
4.52
27.90
0.3
2.30
0.63
3.78
5.83
29.28
0.2
2.63
0.71
5.01
7.06
30.67
0.1
5.07
0.81
7.13
13.89
31.44
0.05
7.20
0.85
11.23
15.32
32.20
23

---

Table 5: Numerical results of Seg2Seg En- SimulST.
SimulST on MuST-C En
A
CW
AP
AL
DAL
BLEU
0.4
588.99
0.68
672.58
1444.73
14.47
0.3
653.43
0.71
893.27
1533.80
16.98
0.2
831.69
0.73
1068.11
1736.00
18.44
0.1
1563.63
0.79
1602.01
2477.97
21.75
0.05
2532.03
0.92
3125.51
4121.58
23.63
Table 6: Numerical results of Seg2Seg En +Es SimulST.
SimulST on MuST-C En-
A
CW
AP
AL
DAL
BLEU
0.4
682.84
0.71
772.23
1645.60
20.72
0.3
874.32
0.74
1129.03
1891.32
23.96
0.2
1676.52
0.80
1536.29
2724.39
26.53
0.1
1963.08
0.82
2329.22
3022.22
28.14
0.05
2326.10
0.89
2838.15
4074.75
28.69
24


Reviews:
[{"review_id": "UCWT6p9dKZ", "reviewer": "Reviewer_D731", "summary": "The paper addresses the task of simultaneous sequence generation in real-time scenarios, where the target sequence is generated while receiving the source sequence. The authors propose a unified segment-to-segment framework called Seg2Seg to tackle this challenge. Seg2Seg aims to learn the optimal moments for generating by alternating between waiting for a source segment and generating a target segment, using a latent segment as the bridge between the source and target. The proposed framework employs expectation training to explore all potential source-target mappings and achieve adaptive and unified learning. The authors applied their proposed method to three real-time tasks: Streaming Automatic Speech Recognition (ASR), Simultaneous Machine Translation (SimulMT), and Simultaneous Speech Translation (SimulST). The experimental results demonstrate that their approach achieves a favorable balance between performance and latency in all three tasks.\n", "questions": "1. Although the proposed approach shares similarities with the MMA framework, the paper does not explicitly mention the similarities and differences between the two methods. Could the authors provide more insight into how their approach differs from and aligns with the MMA framework?\n2. It appears that the training cost of this framework is higher compared to fixed strategies like Wait-k. Could the authors provide a comparison of training time and training cost between their proposed approach and fixed strategies?\n", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The authors aim to introduce a unified framework for training and inference in real-time tasks, and they have successfully achieved this unified approach.", "weaknesses": "1. The proposed approach lacks novelty as it bears similarities to the MMA framework.\n2. The efficiency of the proposed method is relatively low as it requires backpropagation through multiple training objectives, resulting in increased training time.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful and valuable comments, and we will refine the paper following your suggestions. Following, we will respond to your questions in detail.\n\n&nbsp;\n\nQ1: The difference and superiority of Seg2Seg compared to MMA?\n\nA1: Compared with MMA, there are several fundamental distinctions that make Seg2Seg a more innovative and superior solution:\n\n1. **Strong generalization across multiple tasks**: MMA was originally introduced in SimulMT, which is good at aligning source and target sequences of similar length in units of tokens. Since its dynamic programming involves token-by-token multiplication operations, it is difficult for MMA to work when the source sequences are very long and is in a different modality than the target sequence. As a result, many additional heuristics are required to be added to the source sequence when MMA is applied to SimulST, e.g., fixed pre-decision of 280ms. On the contrary, Seg2Seg introduces natural segments instead of tokens as a bridge between the source and the target, so it is not affected by the length and modality of the source sequence and thus can be applied to various simultaneous generation tasks such as streaming ASR, SimulMT and SimulST with a unified framework without any heuristic rules. Moreover, this unified framework facilitates multi-task learning, which is more helpful for difficult tasks like SimulST.\n\n2. **Different decision-making methods**: MMA uses monotonic attention to enable each attention head to independently calculate token alignments, while Seg2Seg takes a different approach by directly determining whether to aggregate latent segment and emit the target token at the embedding level. Many previous works [1,2,3] have shown that the decision-making method of MMA will be affected by some abnormal heads, because MMA can output only when all heads (e.g., 6 layers * 8 heads = 48 heads totally) decide to WRITE, and must wait even only one head select READ. In contrast, Seg2Seg only needs to make a decision once before the encoder and decoder, so as to obtain a more stable decision.\n\n3. **Less decision-making consumption**: Completely different decision-making methods make Seg2Seg have less decision-making consumption during inference, compared to MMA. For each READ/WRITE decision, MMA needs to forward the entire model, because the READ/WRITE decisions are made collectively by all the heads of the decoder. Denoting the source and target lengths as $J$ and $I$ (corresponding to $J$ READ actions and $I$ WRITE actions), MMA needs forward $(J+I) \\times (Enc Embdding+Encoder+Dec Embdding+Decoder) $ in total. However, Seg2Seg has a completely different way of making decisions, deciding whether to aggregate latent segments at the encoder embedding and subsequently deciding whether to emit at the decoder embedding. Therefore, Seg2Seg only needs forward $J \\times Enc Embdding + K \\times Encoder + I \\times Dec Embdding + I * Decoder $ where $K$ is the segment number, thereby reducing decision-making consumption.\n\n4. **Different latency control**: Seg2Seg and MMA also differ in their approaches to control latency. MMA constrains the average latency through DAL metric. In contrast, Seg2Seg controls the latency more intuitively by learning the segment number, directly influencing the number of READ/WRITE alternations during inference, which better aligns with the user's perception of latency in a simultaneous system [4].\n\nOverall, **Seg2Seg is different from MMA in its unique READ/WRITE decision-making and latency-controlling**. More importantly, Seg2Seg can adapt to a variety of simultaneous tasks without any modification, which sets Seg2Seg apart from MMA. Following your suggestions, we will provide a detailed exposition of the differences between Seg2Seg and MMA in the next version.\n\nReference:\n\n[1] ICLR 2020: Monotonic Multihead Attention.\n\n[2] ACL 2022: Modeling dual read/write paths for simultaneous machine translation.\n\n[3] Interspeech 2022: Cross-Modal Decision Regularization for Simultaneous Speech Translation. \n\n[4] ACL 2022: Gaussian Multi-head Attention for Simultaneous Machine Translation.\n\n&nbsp;\n\nQ2: Efficiency of the proposed method?\n\nA2: Thank you for your concern about training costs. Indeed, as you pointed out, Seg2Seg, like most adaptive methods, requires quality and latency training objectives to trade off between generation quality and latency, so the training cost of the adaptive method is generally slightly higher than that of the fixed wait-k policy. However, the performance gain brought by the adaptive method compared with the fixed policy is also significant.\n\nTo evaluate the training efficiency of Seg2Seg, we show the UPS (updates per second) of Seg2Seg and strong baselines on the SimulMT task in the table below.\n\n| Methods | UPS  |\n| ------- | ---- |\n| Wait-k  | 4.52 |\n| Seg2Seg | 2.61 |\n| HMT     | 1.36 |\n| MMA     | 0.47 |\n| GSiMT   | 0.20 |\n\nCompared with the fixed wait-k policy, Seg2Seg does train slightly slower. However, considering the significant performance improvement brought by Seg2Seg compared with the fixed wait-k, we believe that the slightly higher training cost is acceptable. More importantly, **Seg2Seg is the most efficient in training among previous strong adaptive methods**. The additional training cost of Seg2Seg arises from the calculation of emission probability, while MMA involves looping calculation in each attention head, HMT needs to upsample the target sequence several times, and GSiMT needs to forward the model multiple times in a loop. Therefore, compared with the adaptive methods, Seg2Seg demonstrates a lower training cost while achieving superior performance.\n\nThanks for your suggestion, we will incorporate a detailed analysis of the training cost in the next version.\n\n&nbsp;\n\nIf our responses answer your questions well and reassure your concerns, we would appreciate if you could reassess our work and increase the rating."}, {"review_id": "EATdaZzBlM", "reviewer": "Reviewer_ZDUt", "summary": "This paper proposes a method for simultaneous sequence to sequence modeling.\nThe proposed approach introduced a latent segment, which represents the number of consecutive reads followed by consecutive writes.\nThe method is applied to three simultaneous sequence to sequence modeling task, streaming ASR, simultaneous MT and simultaneous ST.\nResults demonstrate better latency-quality tradeoffs compared to a number of baselines. Finally, the framework is applied in a multitask setting (ASR + MT + ST) where it is shown that combining more tasks presents better latency-quality tradeoffs.", "questions": "* In eq 1, Emb (xj ) is used. This design choice is not justified in the paper. For example, why not use encoder output states, etc.\n* the simultaneous MT experiments only concern one language direction. The results would be more convincing with at least two directions.\n* the streaming ASR results only have one datapoint. It may be interesting to have a range of latency/quality points (even though it doesn't seem to be common practice in the streaming ASR literature).\n\nSome presentation suggestions:\n* 48: \"without any heuristics\": some of the previous work is also based on models rather than heuristics. Suggest rephrasing to \"without any task-specific assumptions\"\n* 92: \"Details are introduced following.\": remove or \"Details are introduced in the following sections.\"\n* 155: \"By jointly trained\": \"By jointly training\"", "limitations": "N/A", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "* the method nicely generalizes previous ideas on simultaneous modeling and is broadly applicable to both ASR and translation\n* the main results are strong compared to the state of the art.\n* the ablation analyses convincingly show that the improvements come from the proposed approach", "weaknesses": "See more detailed comments/questions below on how the paper can be improved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful and valuable comments, and we will refine the paper following your suggestions. Following, we will respond to your questions in detail.\n\n&nbsp;\n\nQ1: The design choice of Emb(xj) instead of using encoder output states?\n\nA1: Thank you for raising a question regarding the calculation of the aggregation probability in Eq. (1). The decision to use embeddings for predicting the aggregation probability is based on two considerations:\n\n(1) **Efficiency**: In a simultaneous system, we want to make decisions with the least computational cost. If the encoder output is used to predict the aggregation probability, then when each new source token arrives, we need to forward the whole encoder to make a decision, resulting in higher computational overhead. Conversely, by employing embeddings for predicting aggregation probability, we only need to forward the embedding for each new token when making decisions. The complete encoder is forwarded only after deciding to aggregate a latent segment.\n\n(2) **Consistency**: After aggregating into latent segments, we need to calculate the emission probability. This step needs to be completed before the decoder, because the mapping (aggregation and emission probability) needs to be trained jointly with cross-attention. Therefore, the emission probability needs to be predicted via target embedding, i.e., within the embedding space. To maintain consistency, we should predict the aggregation probability and latent segment representation in the embedding space as well, ensuring that Eq.(3) is computed within the embedding space.\n\nThanks for your reminder, we will provide further justifications for this design choice in the next version.\n\n&nbsp;\n\nQ2: Simultaneous MT experiments on other language direction?\n\nA2: Due to space constraints, we only show the SimulMT results on the most widely used and difficult benchmark WMT15 De-En. We also conducted SimulMT experiments on IWSLT15 English-Vietnamese. The table below shows the performance of Seg2Seg on IWLST15 En-Vi, and the BLEU-AL curves of Seg2Seg and strong baselines are reported in the PDF uploaded in the rebuttal. The results show that Seg2Seg also outperforms the previous state-of-the-art SimulMT method HMT [1].\n\n| AL   | BLEU  |\n| ---- | ----- |\n| 2.76 | 28.10 |\n| 4.01 | 28.75 |\n| 6.54 | 28.82 |\n| 8.65 | 28.88 |\n| 10.2 | 28.90 |\n\nThanks for your suggestion, we will add the SimulMT results on IWLST15 En-Vi in the next version.\n\nReference:\n\n[1] ICLR 2023: Hidden Markov Transformer for Simultaneous Machine Translation.\n\n&nbsp;\n\nQ3: About datapoint in streaming ASR results?\n\nA3: Indeed, as you said, the common practice of the streaming ASR literature is to report one datapoint with a minimum latency. This is mainly because the source and target sequences in ASR are monotonic, and higher generation quality can be achieved at lower latency. In Table 1, we report the performance of Seg2Seg at extremely low latency. Considering that Seg2Seg is already close to offline performance at this latency and for fair comparison with previous works, we only report one datapoint. According to your suggestion, we will add more datapoints in the next version.\n\n&nbsp;\n\nQ4: About presentation suggestions?\n\nA4: Thanks for your careful inspection and valuable suggestions. We will refine the paper based on your suggestions.\n\n&nbsp;\n\nThanks again for your careful and valuable comments, and we will refine the paper following your suggestions."}, {"review_id": "KGabBfAuuS", "reviewer": "Reviewer_CEsh", "summary": "This paper presents an approach (Seg2Seg) for simultaneous sequence generation, which attempts to balance high generation quality with low latency. Rather than processing the entire source sequence at once (incurring high latency), the model splits it into segments. Latent segments are used to aggregate source tokens and generate the corresponding target sub-sequence. While latent segments are discrete for inference, Seg2Seg uses expectation training to learn optimal segment boundaries, combining the cross-entropy loss with a latency loss. Experiments are conducted on streaming automatic speech recognition, simultaneous machine translation and simultaneous speech translation.", "questions": "How would the approach work between languages with a high amount of reordering (e.g. English (subject-verb-object) and Japanese (subject-object-verb))?\n\nInstead of expectation training, did you consider using the gumbel-max trick? If so, what did you observe?\n\nWhy not include [1] (already cited) as a machine translation baseline?\n\n[Minor] In figure 4, why is the Y axis label \"BLEU score\" for SimulMT and \"SacreBLEU\" for SimulST?\n\n[1] Arivazhagan et al. Monotonic Infinite Lookback Attention for Simultaneous Machine Translation. ACL. 2019", "limitations": "Yes, limitations are addressed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The model introduced in this paper learns when to stop aggregating information based on latent segments instead of using pre-defined heuristics.\n\nThe approach is applicable to multiple simultaneous generation tasks (written and spoken modalities).\n\nOver the 3 considered tasks, the proposed approach generally has favorable BLEU/lagging trade-offs compared to many baselines.", "weaknesses": "The notion of adaptive wait policies has been explored in previous work (cited by the authors). This work incrementally improves on known concepts.\n\nSome figures are quite small and might be difficult to read if the paper is printed out.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful and valuable comments, and we will refine the paper following your suggestions. Following, we will respond to your questions in detail.\n\n&nbsp;\n\nQ1: About the notion of adaptive wait policies?\n\nA1: Indeed, I agree with you that the concept of adaptive policies existed before. However, it is important to highlight the key distinction of Seg2Seg. Our focus is that the previous adaptive policies are adaptive on READ/WRITE decisions, but the whole framework (training or some attention design) always involves some task-specific heuristics, such as locality in ASR and similar sequence lengths in MT. These heuristics often limit the direct applicability of previous adaptive policies across multiple tasks.\n\nTherefore, The motivation and innovation of Seg2Seg lie in the development of a universal and adaptive unified framework that can handle various simultaneous tasks, which avoids artificially setting some task-specific heuristics for different tasks and further makes multi-task learning possible.\n\nThanks for your suggestion, we will emphasize the difference between Seg2Seg and previous adaptive policies in the next version.\n\n&nbsp;\n\nQ2: How would the approach work between languages with a high amount of reordering?\n\nA2: Actually, word reordering is one of the major challenges for both SimulMT and SimulST. To address this issue, Seg2Seg uses the segment as bridge instead of taking the token as the unit like the previous methods, so Seg2Seg will wait until the received words can be aggregated into segments and then translate the target sequence segment-by-segment. This allows Seg2Seg to handle certain levels of local reordering. For instance, in source languages with a word order like 'SVO,' Seg2Seg aggregates the 'VO' into one segment and emits the corresponding 'OV' segment in the target 'SOV' language. For extremely long-distance reordering under low latency requirements, Seg2Seg will generate a target translation with a similar word order to the source sequence, thus balancing translation quality and latency. \n\nIn the main experiment, we specially selected the most challenging translation languages for simultaneous generation, German and English, where German is the SOV language and English is the SVO language. Both SimulMT and SimulST results between German and English demonstrate the superior performance of Seg2Seg on languages with such differences in word order.\n\n&nbsp;\n\nQ3: Did you consider using the gumbel-max trick instead of expectation training?\n\nA3: Indeed, the Gumbel-Max trick can facilitate backpropagation in Seg2Seg, enabling the learning of aggregation and emission. While we haven't explored the use of the Gumbel-Max trick in our current work, we sincerely appreciate your valuable proposal. We will certainly consider incorporating the Gumbel-Max trick in our future research endeavors to potentially enhance the performance of our approach. Thank you for bringing this to our attention.\n\n&nbsp;\n\nQ4: Why not include MILk[1] as a machine translation baseline?\n\nA4: When Monotonic Infinite Lookback Attention (MILk) [1] was proposed, it applied monotonic attention on the LSTM architecture. The subsequent Monotonic Multi-head Attention (MMA) [2] is the Transformer implementation of MILk (using the same method), and it also achieves better performance than MILk. Considering that MMA and MILk use the same method, and propose that Seg2Seg is also based on Transformer, we did not add MILk in the main experiment.\n\nThanks for your suggestion, we will emphasize the relationship between MMA and MILk in the system setting section in the next version.\n\nReference:\n\n[1] ACL 2019: Monotonic Infinite Lookback Attention for Simultaneous Machine Translation.\n\n[2] ICLR 2020: Monotonic Multihead Attention.\n\n&nbsp;\n\nQ5: Why is the Y axis label \"BLEU score\" for SimulMT and \"SacreBLEU\" for SimulST?\n\nA5: Apologies for the confusion caused, and the inconsistency in the label name is a writing typo. Thanks for your observation, we will fix it in the next version.\n\n&nbsp;\n\nQ6: About some figures size?\n\nA6: Thank you for your careful inspection and valuable suggestions. We will enlarge the figures based on your suggestions.\n\n&nbsp;\n\nIf our responses answer your questions well and reassure your concerns, we would appreciate if you could reassess our work and increase the rating."}, {"review_id": "zwA9U8yCk6", "reviewer": "Reviewer_ooDH", "summary": "This paper proposes a unified Seg2Seg framework for simultaneous sequence generation tasks. The framework introduces a latent segment as the pivot between the source and target, exploring all potential mappings via the proposed expectation training. With this approach, the authors achieve high-quality generation with low latency across multiple simultaneous generation tasks, including streaming ASR, SimulMT and SimulST.", "questions": "1. In experiments, what values of \\lambda are being used?\n2. Have you experimented with different coefficients for the mapping and latency training loss?\n3. The performance difference between the proposed method and HMT is minimal. Have you directly compared the performance of HMT on streaming ASR and SimulST tasks when using wav2vec 2.0 to convert audio into a sequence of discrete hidden states?", "limitations": "n/a", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- This paper introduces latent segments as the pivots to explicitly model the mapping between source and target in simultaneous generation tasks.\n- The proposed Seg2Seg framework exhibits better performance across various simultaneous generation tasks compared to existing methods.\n\n", "weaknesses": "- The current version of the proposed framework lacks some details, making it unclear how the prediction of target tokens is learned.\n- As the proposed framework leverages wav2vec 2.0 to convert audio into a sequence of discrete hidden states, streaming ASR and SimulST tasks actually could be viewed as special SimulMT tasks. It is necessary to conduct experiments, such as GSiMT and HMT, to verify the performance of previous SOTA SimulMT methods on these tasks.\n- There are some confusing experimental results. Figure 4 shows that the performance of the offline model is better than that reported in the HMT paper, despite using the same training dataset and model structure. It is unclear whether the improvement of the proposed method contributes to the better offline model or the Seg2Seg framework compared to HMT.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful and valuable comments, and we will refine the paper following your suggestions. Following, we will respond to your questions in detail.\n\n&nbsp;\n\nQ1: It is unclear how the prediction of target tokens is learned?\n\nA1: Seg2Seg leverages the Transformer as the backbone and additionally learns aggregation and emission decisions. For predicting target tokens, Seg2Seg follows the same approach as the Transformer and learns this through traditional cross-entropy loss $L_{ce}$ (in Eq.(10)). Note that the mapping and translation are jointly learned via cross-entropy loss $L_{ce}$.\n\nThank you for your reminder, we will give a more clear explanation in the next version.\n\n&nbsp;\n\nQ2: Streaming ASR and SimulST tasks actually could be viewed as special SimulMT tasks. Have you directly compared the performance of HMT on streaming ASR and SimulST tasks when using wav2vec 2.0 to convert audio into a sequence of discrete hidden states?\n\nA2: Indeed, we have considered the possibility of evaluating the performance of HMT and GSiMT on streaming ASR and SimulST tasks as you pointed out. **However, it is not feasible to directly apply SimulMT methods, such as HMT and GSiMT, to streaming ASR and SimulST via introducing a wav2vec, mainly because SimulST and SimulMT have significant differences in sequence length, and SimulMT methods often involve heuristic design about sequence length.** For streaming ASR and SimulST, although wav2vec2.0 can convert audio into a sequence of discrete hidden states, the sequence length of speech hidden states is still significantly longer the text sequence length in SimulMT. For example, the average source sequence length of WMT15 De-En SimulMT is 20, while the average length of speech hidden states of MuST-C En-De is as high as 300. This difference makes SimulMT methods such as HMT and GSiMT unable to process speech inputs. The reason will be described in detail below.\n\n- For HMT, it involves upsampling the target sequence K times, corresponding to K states, with each state representing a translating moment in the source sequence. Therefore, K is dependent on the length of the source sequence. In SimulMT, HMT requires a maximum upsampling of only 20 times. However, for streaming ASR and SimulST, HMT will require a maximum upsampling of the target sequence by 300 times, with each state corresponding to one translating moment among speech features. The 300 times upsampling is completely unrealistic and unacceptable. Therefore, the HMT framework can only be applied to shorter source sequences in SimulMT and is difficult to directly apply to SimulST.\n- As for GSiMT, its original article already pointed out its high training complexity, requiring cyclic forward the entire model I*J times, where I and J are the lengths of the source and target sequences, respectively. When the source sequence length increases from 20 in SimulMT to 300 in SimulST, the overall training time and resource consumption increase significantly, making GSiMT unviable for SimulST.\n\nOverall, the construction of previous SimulMT methods often involves task-specific heuristics on sequence and thus results in the limitations of applying SimulMT methods to streaming ASR and SimulST tasks via introducing a wav2vec2.0. This is also the primary motivation behind the development of Seg2Seg, that is, to develop a unified simultaneous generation framework that can be applied universally across multiple tasks/sequences. Thank you for your reminder, we will add an explanation of the limitations of applying the SimulMT method to SimulST in the next version.\n\n&nbsp;\n\nQ3: In Figure 4, the performance of the offline model is better than that reported in the HMT paper?\n\nA3: As far as we know, the offline model in the HMT paper uses uni-directional attention (mentioned in the original HMT). The offline model we report uses the standard Transformer, that is, bi-directional attention (in order to be consistent with the SimulST setting), so it is normal to be better than the offline model in HMT paper. Actually, our proposed method and HMT use exactly the same settings, and the HMT performance we implemented is almost the same as the original reported results.\n\nIn addition, we additionally compared Seg2Seg and strong baselines on IWSLT15 English-Vietnamese SimulMT, and the results are reported in the PDF uploaded by rebuttal. The results show that Seg2Seg also surpasses previous strong baselines, especially at low latency, demonstrating the advantage of Seg2Seg over SimulMT.\n\nFurthermore, the results in Figure 8 (always be used to evaluate the policy quality) also show that Seg2Seg achieves a more accurate policy than HMT. \n\nTherefore, I hope this experiment can dispel your doubts about whether most of the improvement of Seg2Seg comes from the improvement of translation quality. Thank you for pointing out this issue, we will emphasize the setting of the offline model in the next version to avoid misunderstandings.\n\n&nbsp;\n\nQ4: In experiments, what values of $\\lambda$ are being used?\n\nA4: $\\lambda$ is introduced in Eq.(9), which is used to constrain the delay of Seg2Seg. Appendix D reports the values of $\\lambda$ and the corresponding numerical results in SimulMT and SimulST experiments. For streaming ASR, following previous work, the results in Table 1 are the generation quality at the lowest latency ($\\lambda=0.4$). Based on your suggestion, we will add the $\\lambda$ setting to the experimental setup.\n\n&nbsp;\n\nQ5: Have you experimented with different coefficients for the mapping and latency training loss?\n\nA5: We did not add any weight coefficients before $L_{ce}$ and $L_{latency}$, and Seg2Seg has already achieved superior performance under this setting. Searching on the coefficients may lead to further improvements, which we leave for future work.\n\n&nbsp;\n\nIf our responses answer your questions well and reassure your concerns, we would appreciate if you could reassess our work and increase the rating."}]

Author rebuttal:
Thanks to all the reviewers for their careful and valuable comments.

&nbsp;

Following the reviewers’ suggestion, we additionally report the results on a commonly used SimulMT benchmark IWSLT15 English→Vietnamese (En→Vi), to more comprehensively evaluate the performance of Seg2Seg on SimulMT. As shown in the Figure 1 of the uploaded PDF, Seg2Seg also outperforms strong baselines in IWSLT En→Vi, especially at low latency.

Overall, Seg2Seg achieved better performance than previous strong baselines on the SimulMT results of IWSLT15 En-Vi and WMT15 De-En. At the same time, Seg2Seg is also a unified framework that can be directly used among various simultaneous generation tasks, such as streaming ASR and SimulST, which is another advantage compared to the previous SimulMT methods. Therefore, **superior performance and better versatility on various simultaneous generation tasks are the innovation and value of Seg2Seg**.

&nbsp;

Thanks again to all the reviewers for their valuable comments, and we will refine the paper following your suggestions.

Meta-review: The paper proposes an end-to-end model for simultaneous sequence to sequence generation (e.g. simultaneous speech translation). Compared to previous approaches, it does not rely on task-specific heuristics and enables multi-task learning. The experiments show strong results compared to previous approaches across multiple tasks with better quality vs. latency trade-offs compared to the previous state of the art. While the impact may be somewhat limited due to the specific nature of the task for which the model is proposed, the paper makes a solid contribution to this problem.
Decision: Accept (poster)