SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Expectation Training Figure 12 shows the mapping during training. In expectation training, Seg2Seg considers all possible latent segments, with the number of segments ranging from 1 to  $J$ , shown in Figure 12(a) and 12(b). By incorporating the constraint of the latency loss on the number of segments, Seg2Seg can effectively learn to aggregate an appropriate number of latent segments. Furthermore, Figure 12(c) demonstrates the probability that each target token can pay attention to the source token (referred to as  $\mathcal{M}_{ij}$  in Eq.(7)). This expectation aligns well with the inference process depicted in Figure 11(c), thereby highlighting the effectiveness of expectation training.
B.3 Case Study
Figure 13: Case study (#4992-23283-0015 in LibriSpeech) of Seg2Seg on streaming ASR task. To demonstrate the simultaneous generation process, we correspond target outputs and received source inputs to show the moments of generating each target token.
Figure 14: Case study (#1366 in WMT15 De→En) of Seg2Seg on SimulMT task. To demonstrate the simultaneous generation process, we correspond target outputs and received source inputs to show the moments of generating each target token.
Source	ein Lot@@ to@@ Spieler aus Har@@ vey ist diesen Monat dra@@ n
English meaning	_a _Lotto _player _from _Harvey _is _this _month _turn
Reference	a Harvey lotto player is in the month .
Outputs	a Lotto player from Harvey is this month .
#Latent Segments	Received Source Inputs	Target Outputs
1	ein	a
2	ein Lot@@ to@@	Lot@@ to
3	ein Lot@@ to@@ Spieler	player
4	ein Lot@@ to@@ Spieler aus Har@@	from
5	ein Lot@@ to@@ Spieler aus Har@@ vey	Har@@ vey
6	ein Lot@@ to@@ Spieler aus Har@@ vey ist	is
7	ein Lot@@ to@@ Spieler aus Har@@ vey ist diesen Monat dra@@ n	this mon@@ th .
Figure 13, 14 and 15 visualize the simultaneous generation process of Seg2Seg on the cases from streaming ASR, SimulMT and SimulST. In Streaming ASR and SimulST, for a clear illustration, we use an external offline speech-text alignment tool to align the transcription with the speech sequence, and the aligned transcription is displayed above the speech waveform.
11https://pytorch.org/audio/main/tutorials/forced Alignment_tutorial.html
Figure 15: Case study (#1166_119 in MuST-C En→De) of Seg2Seg on SimulST task. To demonstrate the simultaneous generation process, we correspond target outputs and received source inputs to show the moments of generating each target token.
Case of Streaming ASR As shown in Figure 13, the target sequence and source sequence in streaming ASR often exhibit a predominantly monotonic relationship, and Seg2Seg can handle this monotonic mapping well. Seg2Seg can segment and aggregate source sequences at speech boundaries, and then emit the corresponding target sequences accurately.
Case of SimulMT Figure 14 shows a case on SimulMT. Seg2Seg can generate the target token after receiving the corresponding source tokens, e.g., generating 'Lot@@ to' after receiving 'Lot@@ to', generating 'player' after receiving 'spieler' and generating 'Har@@ vey' after receiving 'Har@@ vey', which effectively ensures the generation quality of SimulMT. Besides, for some related target tokens, especially the subwords after the bpe operation, Seg2Seg can emit them together from the same latent segment, thereby achieving lower latency.
Case of SimulST Figure 15 shows a case on SimulST, which is more challenging as the source and target sequences involve different modalities and languages. Despite these evident differences, Seg2Seg demonstrates its capability to find the reasonable generating moments, such as generating 'herauszufinden' after receiving 'figure out' in the speech, and generating 'Bedeutung' after receiving 'meaning' in the speech. This is mainly attributed to expectation training, which explores all possible mappings in training, allowing Seg2Seg to learn to aggregate and emit at reasonable moments. As seen, Seg2Seg aggregates the related speech frame into the same latent segment and will not break the acoustic integrity of the speech. For emission, Seg2Seg can accurately determine whether a latent segment can emit the target token, where almost all emitted target outputs are correspond to the source speech contained in the latent segment.
C Latency Metrics
For the latency evaluation of simultaneous generation task, we use mean alignment delay for streaming ASR and average lagging for SimulMT and SimulST.
Mean Alignment Delay [40] is defined as the average word time difference between the ground-truth alignments (speech and transcription) and generating moments:
$$
D _ {\text {m e a n}} = \frac {1}{| \mathbf {y} |} \sum_ {i = 1} ^ {| \mathbf {y} |} \left(\hat {t} _ {i} - t _ {i}\right), \tag {18}
$$
where  $t_i$  is the ground-truth alignment of  $y_i$ , and  $\hat{t}_i$  is the generating moment of  $y_i$ .
Average Lagging (AL) [4] evaluates the average number of tokens (for SimulMT) or speech duration (for SimulST) that target outputs lag behind the source inputs. We use  $t_i$  to denote the generating moments of  $y_i$ , and AL is calculated as:
$$
\mathrm {A L} = \frac {1}{\tau} \sum_ {i = 1} ^ {\tau} t _ {i} - \frac {i - 1}{| \mathbf {y} | / | \mathbf {x} |}, \quad \text {w h e r e} \tau = \underset {i} {\operatorname {a r g m i n}} (t _ {i} = | \mathbf {x} |). \tag {19}
$$
In addition to average lagging, we also use some other latency metrics for SimulMT and SimulST, described as follow.
Consecutive Wait (CW) [11] evaluates the average number of source tokens waited between two target tokens, i.e., the number of segments:
$$
\mathrm {C W} = \frac {\left| \mathbf {x} \right|}{\sum_ {i = 1} ^ {| \mathbf {y} |} \mathbb {1} _ {t _ {i} - t _ {i - 1} > 0}}, \tag {20}
$$
where  $\mathbb{1}_{t_i - t_{i - 1} > 0}$  counts the number of  $t_i - t_{i - 1} > 0$ , i.e., the number of segments. It is worth mentioning that the latency loss  $\mathcal{L}_{latency}$  in training employs the denominator part of the CW metric, as the numerator is a constant.
Average Proportion (AP) [17] evaluates the proportion between the number of received source tokens and the total number of source tokens, calculated as:
$$
\mathrm {A P} = \frac {1}{| \mathbf {x} | | \mathbf {y} |} \sum_ {i = 1} ^ {| \mathbf {y} |} t _ {i}. \tag {21}
$$
Differentiable Average Lagging (DAL) [5] is a differentiable version of average lagging, calculated as:
$$
\mathrm {D A L} = \frac {1}{| \mathbf {y} |} \sum_ {i = 1} ^ {| \mathbf {y} |} t _ {i} ^ {\prime} - \frac {i - 1}{| \mathbf {x} | / | \mathbf {y} |}, \quad \text {w h e r e} t _ {i} ^ {\prime} = \left\{ \begin{array}{c c} t _ {i} & i = 1 \\ \max  \left(t _ {i}, t _ {i - 1} ^ {\prime} + \frac {| \mathbf {x} |}{| \mathbf {y} |}\right) & i > 1. \end{array} \right. \tag {22}
$$
Table 4, 5 and 6 report the numerical results of Seg2Seg on SimulMT and SimulST, where  $\lambda$  a hyperparameter that controls the overall latency of Seg2Seg (refer to Eq.(8)).
Table 4: Numerical results of Seg2Seg on De→En SimulMT.
SimulMT on WMT15 De→En
λ	CW	AP	AL	DAL	BLEU
0.4	1.78	0.60	2.21	4.52	27.90
0.3	2.30	0.63	3.78	5.83	29.28
0.2	2.63	0.71	5.01	7.06	30.67
0.1	5.07	0.81	7.13	13.89	31.44
0.05	7.20	0.85	11.23	15.32	32.20
Table 5: Numerical results of Seg2Seg on En→De SimulST.
SimulST on MuST-C En→De
λ	CW	AP	AL	DAL	BLEU
0.4	588.99	0.68	672.58	1444.73	14.47
0.3	653.43	0.71	893.27	1533.80	16.98
0.2	831.69	0.73	1068.11	1736.00	18.44
0.1	1563.63	0.79	1602.01	2477.97	21.75
0.05	2532.03	0.92	3125.51	4121.58	23.63
Table 6: Numerical results of Seg2Seg on En→Es SimulST.
SimulST on MuST-C En→Es
λ	CW	AP	AL	DAL	BLEU
0.4	682.84	0.71	772.23	1645.60	20.72
0.3	874.32	0.74	1129.03	1891.32	23.96
0.2	1676.52	0.80	1536.29	2724.39	26.53
0.1	1963.08	0.82	2329.22	3022.22	28.14
0.05	2326.10	0.89	2838.15	4074.75	28.69