SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Multipath Wait-k [23] trains a wait-k model via randomly sampling different  $k$  between batches.
MoE Wait-k [18] applies mixture-of-experts (MoE) to learn multiple wait-k policies during training.
Adaptive Wait-k [46] trains a set of wait-k models (e.g., from wait-1 to wait-13), and heuristically composites these models based on their outputs during inference.
MMA [24] applies monotonic multi-head attention and predicts a variable to indicate waiting or generating, which are trained through monotonic attention [35].
GMA [25] introduces Gaussian multi-head attention and uses a Gaussian prior to learn the alignments via attention. With alignments, GMA decides when to start translating based on the aligned positions.
GSiMT [63] generates waiting/generating decisions, and considers all possible decisions in training.
HMT [63] proposes Hidden Markov Transformer, which uses a HMM to correspond translating moments with the target tokens, thereby learning the optimal translating moments for generating.
Simultaneous Speech Translation (SimulST)
Wait-k, MMA [26] for SimulMT can be applied to SimulST by making a fixed pre-decision to split the speech into 280ms durations, where each duration corresponds to one word.
Wait-k-Stride-n [20] generates  $n$  tokens every  $n \times 280ms$  to address the issue of length differences between speech and text. We set  $n = 2$  following their best result.
SimulSpeech [3] divides the speech based on a CTC word detector, and then applies wait-k policy.
SH [52] uses the shortest hypothesis in ASR results as word number, and then applies wait-k policy.
RealTrans [20] detects the word number in the streaming speech via counting the blank in CTC results, and then applies the wait-k-stride-n policy.
MMA-CMDR [64] incorporates cross-modal decision regularization to MMA, which leverages the transcription of speech to improve the decision of MMA.
MoSST [27] uses the integrate-and-firing method [65] to segment the speech based on the cumulative acoustic information, and then applies the wait-k policy.
ITST [53] quantifies the transported information from source to target, and subsequently determines whether to generate output based on the accumulated received information.
MU-ST [21] trains an external segmentation model based on the constructed data to detect the meaning unit, and uses it to decide whether to generate.
DiSeg [54] jointly learns the speech segmentation with the underlying translation model via the proposed differentiable segmentation in an unsupervised manner.
All implementations are adapted from Fairseq Library [66]. In Seg2Seg, we use the standard Transformer-Base (6 encoder and 6 decoder layers) [55] for SimulMT. For streaming ASR and SimulST, we replace the word embedding layer in Transformer-Base with a pre-trained Wav2Vec2.0 $^{6}$  [67] to extract the acoustic embedding, and the rest remains the same as SimulMT.
Evaluation We use SimulEval [68] to evaluate the quality and latency of simultaneous generation. For streaming ASR, following Inaguma and Kawahara [40], we use word error rate (WER) for the quality and the mean alignment delay for the latency, which considers the average difference between generating moments and ground-truth alignments. For SimulMT, following Ma et al. [4] and Zhang and Feng [13], we use BLEU [69] for the generation quality and average lagging (AL) [4] for latency, which measures the average offsets that outputs lag behind inputs (using token as the unit of AL). For SimulST, following Ma et al. [26], we use sacreBLEU [70] for the generation quality and average lagging (AL) for latency (using millisecond ms as the unit of AL). Refer to Appendix C for the detailed calculation of the latency metrics.
$^{6}$ dl.fbaipublicfiles.com/fairseq/wav2vec/wav2vec_small.pt
<sup>7</sup>https://github.com/facebookresearch/SimulEval
Figure 4: SimulMT results of quality v.s. latency (AL, tokens) on WMT15 De→En.
(a)  $\mathrm{En}\rightarrow \mathrm{De}$
(b)  $\mathrm{En}\rightarrow \mathrm{Es}$
Figure 5: SimulST results of quality v.s. latency (AL, ms) on MuST-C  $\mathrm{En}\rightarrow \mathrm{De}$  and  $\mathrm{En}\rightarrow \mathrm{Es}$ .
Results on Streaming ASR Table 1 reports the results on streaming ASR, where Seg2Seg exhibits a better trade-off between generation quality and latency. Compared with T-T using the transducer to align speech and text token-to-token [34], or MoChA generating text based on local speech [22], Seg2Seg adaptively learns source-target mapping through latent segments, thereby performing better.
Results on SimulMT Consistent with the previous methods [4, 24, 13], we adjust the value of  $\lambda$  (refer to Eq.(8)) to show the performance of Seg2Seg under varying latency. Figure 4 shows that Seg2Seg outperforms previous SimulMT methods at all latency. Compared to methods based on pre-defined rules, such as wait-k and
Table 1: Streaming ASR results.
Systems	WER(↓)	Latency(↓)
clean	other
Offline	3.51	8.49	-
T-T	3.40	9.50	610
+ConstAlign	4.00	11.10	328
+FastEmit	4.00	10.40	195
+SelfAlign	4.00	10.70	145
MoChA	4.80	14.20	320
+CTC	4.00	11.20	240
+DeCoT	3.90	11.60	240
+MinLT	4.50	11.70	320
Seg2Seg	3.55	8.73	324
MoE wait-k, Seg2Seg is more flexible in making generating/waiting decisions, achieving significant advantages. Other methods, such as MMA, GMA and HMT, align the target and source token-to-token. However, since the alignment between the two languages may not be one-to-one [4], some local reordering and multi-word structures can affect the performance [12]. By mapping source to target at the segment level, Seg2Seg is more in line with the simultaneous generation process and mitigates these issues, ultimately achieving state-of-the-art performance.
Results on SimulST For the most challenging SimulST in Figure 5, Seg2Seg achieves state-of-the-art performance, especially at low latency. Most of the previous SimulST methods either segment the speech into fixed lengths [26, 64] or detect the number of words in the speech [3, 52, 20, 27] and then apply a wait-k policy, where both non-derivable word detection and wait-k policy hinder the adaptive learning of the model. Owing to the proposed expectation training, Seg2Seg is completely differentiable and able to jointly learn the mapping from the source to the latent segment and from the latent segment to the target, thereby finding the optimal moments for generating.
4.4 Superiority of Unified Framework on Multi-task Learning
In the sequence-to-sequence framework, multi-task learning composed of ASR, MT and ST is shown to improve the performance on difficult tasks (e.g. speech translation) by sharing knowledge among different tasks [71, 72, 73]. However, in previous simultaneous generation methods, different tasks often involve different architectures and heuristics, leaving no room for multi-task learning. Owing to not involving any task-related heuristics, the proposed unified segment-to-segment framework provides a possibility to apply multi-task learning in simultaneous generation. In Seg2Seg, multi-task learning can include streaming ASR, SimulMT and SimulST, and these three tasks share all parameters, except that SimulMT has a text embedding and streaming ASR/SimulST have a shared speech embedding.
Figure 6 demonstrates the improvements brought by multi-task learning on the most challenging SimulST task. By employing multi-task learning in a unified framework, Seg2Seg can achieve further improvements. Specifically, jointly training with streaming ASR yields more discernible improvements, which is mainly because the monotonic properties between speech and text inherent in streaming ASR assist SimulST in learning the source-target mapping [52, 27, 64, 54]. Therefore, the unified Seg2Seg facilitates the sharing of knowledge among various simultaneous tasks through multi-task learning and is helpful for the difficult tasks, such as SimulST.
Figure 6: SimulST results on MuST-C  $\mathrm{En}\rightarrow \mathrm{De}$  with multi-task learning.
We conducted extensive analyses to investigate the specific improvements of Seg2Seg. Unless otherwise specified, all the results are reported on SimulST with MuST-C En→De test set, which is more difficult simultaneous generation task. Refer to Appendix B for more extended analyses.
5.1 Improvements of Adaptive Learning
Seg2Seg learns the mappings from source to segment and from segment to target in an adaptive manner, without any task-specific assumptions. To verify the effect of adaptive learning, we respectively replace the source-to-segment and segment-to-target mappings with heuristic rules, such as fixed-length segment (i.e., fixed-seg) [26] and wait-k/wait-k-stride-n policy (i.e., fixed-emit) [4, 20], and show the SimulST En→De results in Figure 7.
(a) w/o adaptive aggregation.
(b) w/o adaptive emission.
Figure 7: Improvements brought by adaptive learning.
The results show that adaptive learning significantly outperforms heuristic rules. Compared with dividing the source into fixed lengths of  $200 / 280 / 360ms$ , Seg2Seg can adaptively determine whether the received source token can be aggregated into a latent segment, bringing about 1 BLEU improvement. Compared with the rule-based wait-k policy, Seg2Seg judges whether to emit the target token based on the latent segment, thus finding more reasonable generating moments [5].
5.2 Quality of Aggregation and Emission
Seg2Seg learns aggregation and emission adaptively, so we further explore the quality of aggregation and emission, respectively. We apply streaming ASR and SimulMT tasks for evaluation. The detailed calculation of the metrics for aggregation and emission quality are shown in Appendix B.1.
Aggregation Quality To verify whether Seg2Seg can aggregate the source tokens into a latent segment at the appropriate moments in streaming ASR, following Zhang and Feng [54], we conduct experiments on the Buckeye dataset[78], which is a speech segmentation benchmark with the annotated word boundaries. Table 2 shows the segmentation quality of Seg2Seg with some segmentation baselines, and the metrics include precision (P), recall (R) and R-value (comprehensive score) [79]. Seg2Seg achieves better segmentation precision and higher
Table 2: Segmentation accuracy of Seg2Seg.
Systems	P(↑)	R(↑)	R-value(↑)
ES K-Means [74]	30.7	18.0	39.7
BES GMM [75]	31.7	13.8	37.9
VQ-CPC [76]	18.2	54.1	-86.5
VQ-VAE [76]	16.4	56.8	-126.5
DSegKNN [77]	30.9	32.0	40.7
Fixed(280ms)	28.1	16.3	38.4
Seg2Seg	41.1	18.1	41.2
<sup>8</sup>https://buckeyecorpus.osu.edu
comprehensive score R-value, showing that Seg2Seg can perform aggregation and segment the speech at reasonable moments (i.e., token boundaries instead of breaking the continuous speech of a word [27]).
Emission Quality To verify whether the model emits at reasonable moments, we follow Zhang and Feng [31] to evaluate the emission quality in SimulMT based on alignments. We apply  $\mathrm{RWTH}^9$  De  $\rightarrow$  En alignment dataset, and calculated the proportion of the model emitting the target token after receiving its aligned source token, used as the emission accuracy. Figure 8 shows that Seg2Seg can receive more aligned source tokens before emitting under the same latency, meaning that Seg2Seg finds more favorable moments for generating and achieves high-quality generation.
Figure 8: Emission accuracy of Seg2Seg in SimulMT.
5.3 Effect of Latent Segments
To investigate whether the latent segments can act as a bridge between the source and target, we calculate the representations of the source tokens  $(\sum_{x_j \in seg_k} x_j)$ , target tokens  $(\sum_{y_i \in seg_k} y_i)$ , and latent segment  $seg_k$  within each segment during SimulST on  $\mathrm{En} \rightarrow \mathrm{De}$ . We then apply the T-SNE dimensionality reduction algorithm to project these representations into a 2D space. By doing so, we obtain a bivariate kernel density estimation of the representation distribution of source segment, target segment and latent segments, depicted in Figure 9. The visualization clearly demonstrates that the latent segment locates between the source and target sequences in the representation space, effectively serving as a bridge connecting the source and target.
Furthermore, we calculate the cosine similarity between the representations of the source, target and latent segments, as shown in Table 3. It is evident that the similarity between the source and target representations is low, posing a challenge for the model to directly map the source to the target. Conversely, the similarity between the latent segment and the source, as well as the latent segment and the target, is significantly higher. Hence, by introducing the latent segment as a pivot, the model can more easily learn the mapping from the source to the latent segment, and subsequently from the latent segment to the target, thereby finding the optimal moments for generating and achieving better performance.
Table 3: Representational similarity with the latent segment.
Similarity
source ⇌ target	0.53 %
source ⇌ segment	20.01 %
segment ⇌ target	14.66 %
Figure 9: Bivariate kernel density estimation visualization on the representations of source, target and latent segment.
In this paper, we propose a unified segment-to-segment framework for simultaneous sequence generation, which bridges the source and target sequences using latent segments as pivots. Unified Seg2Seg enables the handling of multiple simultaneous generation tasks and facilitates multi-task learning. Experiments and analyses show the superiority of Seg2Seg on performance and generalization.
The proposed Seg2Seg employs the encoder-decoder architecture as its backbone, and exhibits better generality across multiple simultaneous generation tasks. In addition to its primary application on generation tasks, the encoder (aggregation process) or decoder (emission process) of Seg2Seg can also be separately used for some real-time tasks based on encoder-only or decoder-only architecture, such as streaming tagging and online parsing. We leave this for further exploration in future work.
$^{9}$ https://www-i6.informatik.rwth-aachen.de/goldAlignment/
A Dynamic Programming of Mapping
Figure 10: Schematic diagram of the dynamic programming of mapping in expectation training.
In Sec.3.2, we propose expectation training for  $\operatorname{Seg2Seg}$  to learn when to aggregate and emit through  $p(x_{j} \in \mathrm{seg}_{k})$  and  $p(y_{i} \in \mathrm{seg}_{k})$ . Given the monotonic property of simultaneous sequence generation tasks, we can calculate  $p(x_{j} \in \mathrm{seg}_{k})$  and  $p(y_{i} \in \mathrm{seg}_{k})$  using a dynamic programming algorithm. In the following sections, we will provide a detailed explanation of the dynamic programming approach.
A.1 Source Tokens to Latent Segment
Given the streaming source sequence, Seg2Seg predicts aggregation probability  $\alpha_{j}$  for  $x_{j}$  to represent the probability of aggregating the received source tokens into a latent segment at  $x_{j}$ . Given aggregation probability  $\alpha_{j}$ , we calculate  $p\left(x_{i} \in \mathrm{seg}_{k}\right)$  via dynamic programming.
The whole aggregation process is monotonic with inputs, which means that the  $(k + 1)^{th}$  latent segment can only be aggregated once the  $k^{th}$  segment has already been aggregated. Additionally, each latent segment must be aggregated by at least one source token, otherwise a latent segment without representation would be meaningless. As a result, whether  $x_{j}$  belongs to latent segment  $\mathrm{seg}_k$  depends on which segment that  $x_{j - 1}$  is located in, consisting of 3 situations:
- If  $x_{j - 1} \in \mathrm{seg}_{k - 1}$ : As illustrated by the red line in Figure 10(a),  $x_{j}$  belongs to latent segment  $\mathrm{seg}_k$  when  $\mathrm{Seg2Seg}$  aggregate at  $x_{j - 1}$  with probability  $\alpha_{j - 1}$ ;
- If  $x_{j - 1} \in \mathrm{seg}_k$ : As illustrated by the blue line in Figure 10(a),  $x_j$  belongs to latent segment  $\mathrm{seg}_k$  (i.e., the same latent segment with  $x_{j - 1}$ ) when  $\mathrm{Seg2Seg}$  does not aggregate at  $x_{j - 1}$  with probability  $1 - \alpha_{j - 1}$ ;
- Otherwise:  $x_{j}$  can not belong to  $\mathrm{seg}_k$  anyway, i.e., with probability 0.
By combining these situations,  $p\left(x_{i}\in \mathrm{seg}_{k}\right)$  is calculated as:
$$
p \left(x _ {j} \in \sec_ {k}\right) = p \left(x _ {j - 1} \in \sec_ {k - 1}\right) \times \alpha_ {j - 1} + p \left(x _ {j - 1} \in \sec_ {k}\right) \times \left(1 - \alpha_ {j - 1}\right), \tag {10}
$$
where the initialization is
$$
p \left(x _ {1} \in \sec_ {k}\right) = \left\{ \begin{array}{l l} 1 & \text {i f} k = 1 \\ 0 & \text {i f} k \neq 1 \end{array} , \right. \tag {11}
$$
because the first source token inevitably belongs to the first segment. With the above dynamic programming algorithm, we can calculate  $p(x_{j} \in \mathrm{seg}_{k})$ , for  $j = 1, \dots, J$  and  $k = 1, \dots, J$ .
A.2 Latent Segment to Target Tokens
After getting the latent segments, Seg2Seg predicts the emission probability  $\beta_{ik}$ , which indicates the probability of emitting the target token  $y_i$  from the latent segment  $\mathrm{seg}k$ . With the emission probability  $\beta_{ik}$ , we compute  $p(y_i \in \mathrm{seg}_k)$  using dynamic programming as well. Note that there is one difference between the aggregation process and the emission process when employing dynamic programming. In the aggregation process, each latent segment must be aggregated by at least one
source token. However, in the emission process, the latent segment has the option to not generate any target token, as not all source tokens have corresponding target tokens.
Whether  $y_{i}$  belongs to latent segment  $\mathrm{seg}_k$  depends on which segment that  $y_{i-1}$  is emitted from, consisting of 3 situations:
- If  $y_{i - 1} \in \mathrm{seg}_k$ :  $y_i$  is emitted from latent segment  $\mathrm{seg}_k$  with probability  $\beta_{ik}$ ;
- If  $y_{i - 1} \in \mathrm{seg}_l$  for  $l = 1, \dots, k - 1$ :  $y_i$  is emitted from latent segment  $\mathrm{seg}_k$  when  $y_i$  is not emitted from  $\mathrm{seg}_l$  to  $\mathrm{seg}_{k - 1}$ , and then emitted from  $\mathrm{seg}_k$ . Taking Figure 10(b) as an example, if  $y_2 \in \mathrm{seg}_3$ , the premise of  $y_3 \in \mathrm{seg}_5$  is that  $y_3$  is not emitted from  $\mathrm{seg}_3$  and  $\mathrm{seg}_4$ , and is emitted from  $\mathrm{seg}_5$ . Formally, the probability is calculated as:
$$
\beta_ {i k} \times \prod_ {m = l} ^ {k - 1} \left(1 - \beta_ {i, m}\right), \tag {12}
$$
where  $\prod_{m=l}^{k-1}\left(1 - \beta_{i,m}\right)$  is the probability that  $y_i$  is not emitted from  $\operatorname{seg}_l$  to  $\operatorname{seg}_{k-1}$ ;
- If  $y_{i - 1} \in \mathrm{seg}_l$  for  $l = k + 1, \dots: y_i$  can not be emitted from  $\mathrm{seg}_k$  anyway, as the emission process is monotonic, i.e., with probability 0.
By combining these situations,  $p(y_{i} \in \mathrm{seg}_{k})$  is calculated as:
$$
p \left(y _ {i} \in \sec_ {k}\right) = \beta_ {i, k} \sum_ {l = 1} ^ {k} \left(p \left(y _ {i - 1} \in \sec_ {l}\right) \prod_ {m = l} ^ {k - 1} \left(1 - \beta_ {i, m}\right)\right), \tag {13}
$$
$$
p \left(y _ {1} \in \sec_ {k}\right) = \beta_ {1, k} \prod_ {m = 1} ^ {k - 1} \left(1 - \beta_ {1, m}\right). \tag {14}
$$
With the above dynamic programming algorithm, we can calculate  $p$  ( $y_{i} \in \mathrm{seg}_{k}$ ), for  $i = 1, \dots, I$  and  $k = 1, \dots, J$ .
B Extended Analyses
B.1 Detailed Calculation of Aggregation and Emission Quality
In Sec.5.2, we evaluate the aggregation and emission quality of Seg2Seg. Here, we give detailed calculations of aggregation and emission quality.
Aggregation Quality We verify whether Seg2Seg can aggregate (segment) the source speech sequence at the appropriate moments on the speech segmentation task [80]. For our evaluation, we utilize the Buckeye dataset, where the ground-truth segmentation is based on word units. Evaluation metrics consist of precision (P), recall (R), and the comprehensive score R-value. Note that since the ground-truth segmentation in Buckeye is in units of words, and the aggregation of Seg2Seg is in units of segments, which makes the segment number of Seg2Seg may be less than the ground-truth segment number, so precision can better reflect whether the aggregation moments of Seg2Seg is reasonable. R-value [79] is a more robust comprehensive metric for speech segmentation task, calculated as:
$$
R - \text {v a l u e} = 1 - \frac {\left| r _ {1} \right| + \left| r _ {2} \right|}{2}, \tag {15}
$$
$$
\text {w h e r e} r _ {1} = \sqrt {\left(1 - R\right) ^ {2} + \left(\frac {R}{P} - 1\right) ^ {2}}, \quad r _ {2} = \frac {- \left(\frac {R}{P} - 1\right) + R - 1}{\sqrt {2}}. \tag {16}
$$
A larger R-value indicates better segmentation quality, where R-value = 100% if and only if  $P = 100\%$  and  $R = 100\%$ .
Emission Quality We verify whether Seg2Seg can emit the target token at appropriate moments in SimulMT. In simultaneous machine translation, it is crucial for the model to emit the corresponding target token after receiving its aligned source token [5], so the alignments can be used as the basis for
judging whether the emitting moments are reasonable. Following Zhang and Feng [31], Guo et al. [47], Zhang and Feng [13], we calculate the proportion of the ground-truth aligned source tokens received before emitting as the emission quality. We apply  $\mathrm{RWTH}^{10}\mathrm{De}\rightarrow \mathrm{En}$  alignment dataset and denote the ground-truth aligned source position of  $y_{i}$  as  $a_{i}$ , while use  $t_i$  to record the emitting moments of  $y_{i}$ . Then, the emission quality is calculated as:
$$
\operatorname {S c o r e} = \frac {1}{| \mathbf {y} |} \sum_ {i = 1} ^ {| \mathbf {y} |} \mathbb {1} _ {a _ {i} \leq t _ {i}}, \quad \text {w h e r e} \quad \mathbb {1} _ {a _ {i} \leq t _ {i}} = \left\{ \begin{array}{l l} 1, & a _ {i} \leq t _ {i} \\ 0, & a _ {i} > t _ {i} \end{array} . \right. \tag {17}
$$
B.2 Visualization of Mapping
(a) Source tokens  $\Rightarrow$  Latent segment
(b) Latent segment  $\Rightarrow$  Target tokens
(c) Source tokens  $\Rightarrow$  Target tokens
(a) Source tokens  $\Rightarrow$  Latent segment
Figure 12: Visualization of the mapping with the latent segment during training on  $\mathrm{En} \rightarrow \mathrm{DeSimulST}$  (Case #2258). The shade of red rectangles in (a) and (b) indicate the probability that the token belongs to different latent segments. (c) indicates the probability that  $y_{i}$  can pay attention to  $x_{j}$ . Note that Seg2Seg considers all possible latent segments during training, so there are more latent segments in (a) and (b) than inference.
Figure 11: Visualization of the mapping with the latent segment during inference on  $\mathrm{En} \rightarrow \mathrm{DeSimulST}$  (Case #2258). The red rectangles in (a) and (b) indicate the correspondence between token and latent segment. (c) indicates the received source tokens when generating each target token. The English meaning of target sequence: Wir (_we) sprechen (_speak) nicht (_not) keinse (_gladly) über (_about) unsere (_our) Geschichte (_story).
(c)  $\mathcal{M}_{ij}$
We visualize the proposed source-target mapping with the latent segment as the pivot in Figure 11 and 12, where the case is from the most challenging SimulST task.
Inference Figure 11 shows the mapping during inference. Seg2Seg effectively aggregates a lengthy speech sequence (approximately 101 tokens) into 6 latent segments, and then these 6 latent segments
$^{10}$ https://www-i6.informatik.rwth-aachen.de/goldAlignment/
emit 8 target tokens. As depicted in Figure 11(a), Seg2Seg exhibits high-quality aggregation by selectively splitting and aggregating the speech at appropriate boundaries, thereby preserving the integrity of the acoustic information [27]. During emission, Seg2Seg exhibits the ability to generate multi-word structures, such as 'unsere Geschichte' (meaning 'our history' in English), within the same latent segment. Overall, Seg2Seg achieves good source-to-target mapping through adaptive aggregation and emission.