SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Shaolei Zhang $^{1,2}$ , Yang Feng $^{1,2*}$
$^{1}$ Key Laboratory of Intelligent Information Processing,
zhangshaolei20z@ict.ac.cn, fengyang@ict.ac.cn
Simultaneous sequence generation is a pivotal task for real-time scenarios, such as streaming speech recognition, simultaneous machine translation and simultaneous speech translation, where the target sequence is generated while receiving the source sequence. The crux of achieving high-quality generation with low latency lies in identifying the optimal moments for generating, accomplished by learning a mapping between the source and target sequences. However, existing methods often rely on task-specific heuristics for different sequence types, limiting the model's capacity to adaptively learn the source-target mapping and hindering the exploration of multi-task learning for various simultaneous tasks. In this paper, we propose a unified segment-to-segment framework (Seg2Seg) for simultaneous sequence generation, which learns the mapping in an adaptive and unified manner. During the process of simultaneous generation, the model alternates between waiting for a source segment and generating a target segment, making the segment serve as the natural bridge between the source and target. To accomplish this, Seg2Seg introduces a latent segment as the pivot between source to target and explores all potential source-target mappings via the proposed expectation training, thereby learning the optimal moments for generating. Experiments on multiple simultaneous generation tasks demonstrate that Seg2Seg achieves state-of-the-art performance and exhibits better generality across various tasks $^2$ .
Recently, there has been a growing interest in simultaneous sequence generation tasks [1, 2, 3, 4, 5, 6, 7, 8, 9] due to the rise of real-time scenarios, such as international conferences, live broadcasts and online subtitles. Unlike conventional sequence generation [10], simultaneous sequence generation receives a streaming source sequence and generates the target sequence simultaneously, in order to provide low-latency feedback [11]. To achieve high-quality generation under such low-latency conditions, simultaneous models must learn to establish a mapping between the target sequence and the source sequence [12] and thereby identify the optimal moments for generating [13].
Directly mapping source and target sequences is non-trivial due to inherent differences between the two sequences, such as modalities or languages, resulting in significant representational and structural gaps. For instance, in streaming automatic speech recognition (Streaming ASR) [8, 9, 14, 15, 16], speech needs to be mapped to text, while simultaneous machine translation (SimulMT) [17, 11, 4, 5, 18, 13] requires the mapping from a source language to a target language (i.e., cross-lingual
$^{2}$ Code is available at: https://github.com/ictnlp/Seg2Seg.
(a) Sequence-to-Sequence framework.
(b) Segment-to-Segment framework.
Figure 1: Illustration of the conventional sequence-to-sequence framework for offline generation and the proposed segment-to-segment framework for simultaneous generation.
alignment [19]). Simultaneous speech translation (SimulST) [1, 2, 3, 20, 21] encounters challenges that encompass both cross-modal and cross-lingual aspects. Therefore, developing an approach to bridge source and target is critical to simultaneous sequence generation.
Existing methods for simultaneous generation often rely on task-specific heuristics to bridge the source and target sequences. For example, streaming ASR methods assume a strong correlation between the target token and local speech, employing a fixed-width window to directly predict the corresponding word [22, 9, 14]. SimulMT methods consider that the source and target sequences have similar lengths, employing fixed wait-k policies [4, 23, 18] or attention mechanisms [5, 24, 25] to establish a token-to-token mapping. Such assumptions of similar length limit their ability to handle sequences with significant length differences [12]. SimulST methods divide the speech into multiple segments to overcome length differences [26, 3], and then apply the fixed wait-k policy [20, 27]. These task-specific heuristics not only hinder the adaptive learning of the source-target mapping but also impede the integration of various simultaneous tasks into a unified framework, restricting the potential of utilizing multi-task learning [28, 29, 30] in simultaneous generation tasks.
Under these grounds, we aim to bridge the source and target sequences in an adaptive and unified manner without any task-specific assumptions. In simultaneous generation process, the model necessarily waits for a source segment and outputs a target segment alternately, with each segment comprising one or more tokens. As such, the source sequence and target sequence should correspond in terms of the segment and ideally agree on the segment representation [31], enabling the segment to serve as a natural bridge between source and target. In this paper, we propose a unified segment-to-segment framework (Seg2Seg) for simultaneous sequence generation, which introduces latent segments as pivots between source and target. As illustrated in Figure 1, given a streaming source sequence, Seg2Seg determines whether the received source tokens can be aggregated into a latent segment. Once aggregated, the latent segment starts emitting the target tokens until the latent segment can no longer emit any further target tokens. Seg2Seg repeats the above steps until finishing generating. To learn when to aggregate and emit, Seg2Seg employs expectation training to explore all possible source-target mappings and find the optimal moments for generating. Experiments on multiple simultaneous generation tasks, including streaming ASR, SimulMT and SimulST, demonstrate that Seg2Seg achieves state-of-the-art performance and exhibits better generality across various simultaneous tasks.
Streaming ASR Recent streaming ASR methods primarily rely on two approaches: transducer and local attention. Transducer involves a speech encoder and a label predictor, which are aligned via a joiner to determine whether to generate [32, 33, 34]. Local attention approach utilizes monotonic attention to determine whether to generate based on the speech within a local window [6, 7, 35, 22]. Moreover, various methods have been proposed to reduce the latency based on these two approaches by optimizing the alignment process [36, 37, 38, 39, 40].
SimulMT Recent SimulMT methods are mainly based on pre-defined rules or alignment mechanisms. For pre-defined rules, Ma et al. [4] proposed wait-k policy, which waits for  $k$  source tokens before alternately waiting/generating one token. Some methods were proposed to improve the flexibility of fixed rules through training [23, 41, 42, 43], simultaneous framework [44, 45], the ensemble of wait-k [46, 18] or post-evaluation [47]. For alignment mechanisms, previous works employ
monotonic attention [5, 24, 31], Gaussian attention [25], binary search [48], non-autoregressive structure [49] or hidden Markov models [13] to learn the alignments between the source and target token-to-token, and make waiting or generating decisions accordingly.
SimulST Recent SimulST methods focus on the segmentation of speech [1, 50, 51, 2]. Ma et al. [26] proposed fixed pre-decision to divide speech into equal-length segments, and applied SimulMT methods such as wait-k [4] and MMA [24]. Some other methods first use CTC results [3, 20], ASR results [52] or integrate-and-firing [27] to detect the number of words in speech, and then apply the wait-k policy. Further, Zhang and Feng [53] proposed ITST, which judges whether the received information is sufficient for translation. Zhang et al. [21] proposed MU-ST, which constructs segmentation labels based on meaning units and uses them to train a segmentation model. Zhang and Feng [54] proposed differentiable segmentation (DiSeg) to directly learn segmentation from the underlying translation model via an unsupervised manner.
Previous methods for simultaneous generation often involve task-specific heuristics, which hinder adaptive learning and limit their applicability to other tasks. The proposed Seg2Seg utilizes the latent segments as pivots to achieve fully adaptive learning of source-target mapping. Furthermore, Seg2Seg serves as a unified framework for various simultaneous generation tasks, making multi-task learning in simultaneous generation feasible.
In this paper, we propose a segment-to-segment framework (Seg2Seg) to map the source sequence to the target sequence, using latent segments as the pivots. With the latent segments, Seg2Seg can adaptively learn to map source to target, enabling it to find the optimal moments to generate the target tokens during the simultaneous generation process. Details are introduced in the following sections.
3.1 Mapping with Latent Segments
Seg2Seg leverages the Transformer (encoder-decoder) [55] as the backbone, and further converts the sequence-to-sequence framework to the segment-to-segment framework by introducing latent segments. Formally, we denote the source sequence as  $\mathbf{x} = \{x_{1},\dots ,x_{J}\}$  with length  $J$  and the target sequence as  $\mathbf{y} = \{y_1,\dots ,y_I\}$  with length  $I$ . In Seg2Seg, the source tokens are first aggregated into several latent segments (source tokens  $\Rightarrow$  latent segment), and then the latent segment emits the target tokens (latent segment  $\Rightarrow$  target tokens), as shown in Figure 2.
Source tokens  $\Rightarrow$  Latent segment For aggregation, Seg2Seg produces a Bernoulli variable  $a_{j}$  for each source token  $x_{j}$  to determine whether the currently received source tokens can be aggregated into a segment. An aggregation probability  $\alpha_{j}$  is predicted as the parameter for the variable  $a_{j}$ , calculated as:
Figure 2: Diagram of source-target mapping with latent segments. The arrows in color and gray represent the mapping in inference and training, respectively.
$$
\alpha_ {j} = \operatorname {s i g m o i d} \left(\operatorname {F F N} \left(\operatorname {R e p} \left(x _ {j}\right)\right)\right), \quad a _ {j} \sim \operatorname {B e r n o u l l i} \left(\alpha_ {j}\right), \tag {1}
$$
where  $\mathrm{FFN}(\cdot)$  is a feed-forward network,  $\operatorname{Rep}(x_j)$  is the representation of  $x_{j}$ , and  $\alpha_{j}$  is aggregation probability at  $x_{j}$ . As shown in Figure 2, if  $a_{j} = 0$ , Seg2Seg waits for the next input, otherwise, it aggregates the tokens received after the previous segment into a new segment. Once a latent segment is aggregated, we calculate its representation by summing the representations of all the source tokens it contains. Specifically, the representation of the  $k^{th}$  latent segment is denoted as  $\mathrm{seg}_k$ , calculated as:
$$
\operatorname {s e g} _ {k} = \mathbf {W} ^ {\text {s r c} \rightarrow \text {s e g}} \sum_ {x _ {j} \in \operatorname {s e g} _ {k}} \operatorname {R e p} \left(x _ {j}\right), \tag {2}
$$
where  $\mathbf{W}^{\mathrm{src}\rightarrow \mathrm{seg}}$  is the learnable projection from source to latent segment space.
Latent segment  $\Rightarrow$  Target tokens Given latent segment representation, Seg2Seg judges whether  $\mathrm{seg}_k$  can emit  $y_{i}$  by producing a Bernoulli variable  $b_{ik}$  with the emission probability  $\beta_{ik}$  as a
(a) Hard mapping in inference.
(b) Expected mapping in training.
Figure 3: Illustration of source-target mapping in inference and training. (a) The color indicates which latent segment the token belongs to, and the final matrix indicates whether the model receives  $x_{j}$  when generating  $y_{i}$  (i.e., the cross-attention, where the white space is masked out because those source tokens are not received yet.). (b) The shade indicates the probability that the token belongs to different latent segments, and the final matrix indicates the probability that  $y_{i}$  can pay attention to  $x_{j}$ .
parameter. Specifically,  $b_{ik}$  is calculated as a dot-product form:
$$
\beta_ {i k} = \operatorname {s i g m o i d} \left(\frac {\mathbf {W} ^ {\mathrm {t g t} \rightarrow \mathrm {s e g}} \operatorname {R e p} \left(y _ {i - 1}\right) \cdot \operatorname {s e g} _ {k} ^ {\top}}{\sqrt {d}}\right), \quad b _ {i k} \sim \text {B e r n o u l l i} (\beta_ {i k}), \tag {3}
$$
where  $\mathbf{W}^{\mathrm{tgt}\rightarrow \mathrm{seg}}$  is the learnable projection from target to latent segment space, and  $d$  is the input dimension. During emission, if  $b_{ik} = 1$ , Seg2Seg generates  $y_{i}$  based on the current received source tokens, otherwise it stops emitting and waits for the next input. Take Figure 2 as an example,  $y_{3}$  will not be emitted from  $\mathrm{seg}_1$  as  $b_{31} = 0$ . After aggregating  $\mathrm{seg}_2$ ,  $y_{3}$  is emitted from  $\mathrm{seg}_2$  as  $b_{32} = 1$ .
Overall, Seg2Seg alternates between waiting for enough source tokens to aggregate a latent segment (i.e., wait until  $a_{j} = 1$ ), and outputting the target tokens until the current latent segment can no longer emit any further tokens (i.e., output until  $b_{ik} = 0$ ). Take the mapping in Figure 2 for instance, Seg2Seg waits for 2 source tokens and generates 2 target tokens, then waits for 3 and generates 2 tokens, then waits for 1 and generates 1 token. Figure 3(a) gives the corresponding mapping in matrix form, where the final matrix indicates whether the model receives  $x_{j}$  when generating  $y_{i}$ .
During training, Seg2Seg tends to learn the aggregation and emission in an adaptive manner. However, a significant challenge arises from the use of Bernoulli variables  $a_{j}$  and  $b_{ik}$  for aggregation and emission, which prevents the back-propagation [56, 57] to the aggregation probability  $\alpha_{j}$  and emission probability  $\beta_{ik}$ . To address this issue, we propose expectation training that employs  $\alpha_{j}$  and  $\beta_{ik}$  instead of Bernoulli variables to calculate the expected mapping, which can be jointly trained with the underlying Transformer model. As illustrated in Figure 3, in expectation training, the source tokens and target tokens are no longer forced to be associated with a single latent segment, but rather can belong to multiple latent segments by probability.
For the aggregation process from source tokens to latent segment, we introduce  $p(x_{j} \in \mathrm{seg}_{k})$  to represent the probability that  $x_{j}$  belongs to the latent segment  $\mathrm{seg}_{k}$ . Since the aggregation process is monotonic with the streaming source sequence, i.e., which segment  $x_{j}$  belongs to is only related to  $x_{j-1}$ ,  $p(x_{j} \in \mathrm{seg}_{k})$  can be calculated via dynamic programming:
$$
p \left(x _ {j} \in \sec_ {k}\right) = p \left(x _ {j - 1} \in \sec_ {k - 1}\right) \times \alpha_ {j - 1} + p \left(x _ {j - 1} \in \sec_ {k}\right) \times \left(1 - \alpha_ {j - 1}\right). \tag {4}
$$
We consider all possible latent segments in the expectation training, so  $k$  ranges from 1 to  $J$  (i.e., aggregate at most  $J$  segments with one token in each segment), even if the source tokens may belong to the later latent segment with a small probability, as shown in Figure 3(b). With  $p(x_{j} \in \mathrm{seg}_{k})$ , we calculate the expected representation of latent segment by weighting all source tokens:
$$
\operatorname {s e g} _ {k} = \mathbf {W} ^ {\text {s r c} \rightarrow \text {s e g}} \sum_ {j = 1} ^ {J} p \left(x _ {j} \in \operatorname {s e g} _ {k}\right) \times \operatorname {R e p} \left(x _ {j}\right). \tag {5}
$$
For the emission process from latent segment to target tokens, we introduce  $p(y_{i} \in \mathrm{seg}_{k})$  to represent the probability that  $y_{i}$  can be emitted from latent segment  $\mathrm{seg}_{k}$ . Since the emission process is mono-
tonic with the simultaneous generation,  $p(y_{i} \in \mathrm{seg}_{k})$  can be calculated via dynamic programming:
$$
p \left(y _ {i} \in \sec_ {k}\right) = \beta_ {i, k} \sum_ {l = 1} ^ {k} \left(p \left(y _ {i - 1} \in \sec_ {l}\right) \prod_ {m = l} ^ {k - 1} \left(1 - \beta_ {i, m}\right)\right). \tag {6}
$$
We give a detailed introduction to the dynamic programming algorithm in Appendix A.
Learning Mapping To adaptively learn  $\alpha$  and  $\beta$ , we jointly train  $p(x_{j} \in \mathrm{seg}_{k})$  and  $p(y_{i} \in \mathrm{seg}_{k})$  with Transformer via the cross-entropy loss  $\mathcal{L}_{ce}$ . During inference, each target token in Seg2Seg no longer focuses on all source tokens, but can only pay attention to the source token within the same latent segment or the previous segments (i.e., the current received tokens), as shown in Figure 3(a). So in training, we calculate the probability that  $y_{i}$  can pay attention to  $x_{j}$ , denoted as  $\mathcal{M}_{ij}$ :
$$
\mathcal {M} _ {i j} = \sum_ {k} p \left(y _ {i} \in \sec_ {k}\right) \times p \left(x _ {j} \in \{\sec_ {1}, \dots , \sec_ {k} \}\right) = \sum_ {k} p \left(y _ {i} \in \sec_ {k}\right) \times \sum_ {l = 1} ^ {k} p \left(x _ {j} \in \sec_ {l}\right). \tag {7}
$$
Then, we multiply the mapping  $\mathcal{M}_{ij}$  with the original cross-attention [53, 58] and normalize it to get the final attention distribution, which is used to calculate the expected target representation. By jointly training mapping and generation via the cross-entropy loss  $\mathcal{L}_{ce}$ , Seg2Seg will assign higher  $\mathcal{M}_{ij}$  between those related source and target tokens, thereby learning a reasonable mapping.
Learning Latency Besides learning mapping for high-quality generation, we also introduce a latency loss  $\mathcal{L}_{\text {latency }}$  to encourage low latency. We utilize two commonly used latency metrics, consecutive wait (CW) [17] and average lagging (AL) [4], to calculate the expected latency of Seg2Seg, where CW measures the number of latent segments (i.e., streaming degree [25]), and AL measures the lagging of target token (i.e., lagging degree). Therefore,  $\mathcal{L}_{\text {latency }}$  is calculated as:
$$
\mathcal {L} _ {\text {l a t e n c y}} = \mathcal {C} _ {\mathrm {C W}} (\boldsymbol {\alpha}) + \mathcal {C} _ {\mathrm {A L}} (\boldsymbol {\mathcal {M}}),
$$
$$
\text {w h e r e} \quad \mathcal {C} _ {\mathrm {C W}} (\boldsymbol {\alpha}) = \left\| \sum_ {j = 1} ^ {| \mathbf {x} |} \alpha_ {j} - \lambda | \mathbf {y} | \right\| _ {2} + \left\| \sum \operatorname {M a x P o o l} \left(\alpha_ {i}, \left\lfloor \frac {| \mathbf {x} |}{\lambda | \mathbf {y} |} \right\rfloor\right) - \lambda | \mathbf {y} | \right\| _ {2}, \tag {8}
$$
$$
\mathcal {C} _ {\mathrm {A L}} \left(\boldsymbol {\mathcal {M}}\right) = \frac {1}{| \mathbf {y} |} \sum_ {i = 1} ^ {| \mathbf {y} |} \sum_ {j = 1} ^ {| \mathbf {x} |} \mathcal {M} _ {i j}.
$$
For the number of latent segments  $\mathcal{C}_{\mathrm{CW}}(\alpha)$ , following Zhang and Feng [54], we constrain Seg2Seg via the expected segment number  $\sum_{j=1}^{|y|} \alpha_j$  and the uniformity of aggregation, where MaxPool  $(\cdot)$  is the max polling operation with kernel size of  $\left\lfloor \frac{|x|}{\lambda |y|} \right\rfloor$ . For the expected lagging  $\mathcal{C}_{\mathrm{AL}}(\mathcal{M})$ , we constrain the expected lagging  $\sum_{j=1}^{|y|} \mathcal{M}_{ij}$  of target token  $y_i$ .  $\lambda$  is a hyperparameter that controls the overall latency of Seg2Seg. A larger  $\lambda$  encourages Seg2Seg to aggregate more latent segments, thereby achieving low latency. When  $\lambda \to 0$ , the number of latent segments decreases and latency becomes higher, finally degenerating into a sequence-to-sequence framework when  $\lambda = 0$ .
Overall, the total training objective of Seg2Seg is the trade-off between  $\mathcal{L}_{ce}$  for generation quality and  $\mathcal{L}_{latency}$  for generation latency, calculated as:
$$
\mathcal {L} = \mathcal {L} _ {c e} + \mathcal {L} _ {\text {l a t e n c y}}. \tag {9}
$$
In inference, we set  $a_{j} = 1$  when  $\alpha_{j} \geq 0.5$  and  $b_{ik} = 1$  when  $\beta_{ik} \geq 0.5$  without sampling [24, 13]. Algorithm 1 illustrates the specific inference process of Seg2Seg. Given a streaming source sequence, Seg2Seg continuously repeats the process of aggregating source tokens into a latent segment (lines 2-6) when  $a_{j} = 1$  and then emitting target tokens from the latent segment (lines 8-12) while  $b_{ik} = 1$ , until the generation process is completed.
Owing to generating the target sequence in units of segment, it is natural for Seg2Seg to use beam search inside each target segment. Therefore, in the following experiments, we set the size of the beam search for each segment to 5.
Algorithm 1 Inference of Segment-to-Segment Framework
Input: Source sequence x.  
Output: Target sequence  $\hat{\mathbf{y}}$ .  
Initialization: Received source sequence  $\hat{\mathbf{x}} = []$ ; Target sequence  $\hat{\mathbf{y}} = [\langle \mathrm{bos}\rangle]$ ; Index  $j = 0, i = 1, k = 1$ .  
1: while  $\hat{y}_{i-1} \neq \langle \mathrm{eos} \rangle$  do  
2: while  $a_{j} == 0$  and  $\hat{\mathbf{x}} \neq \mathbf{x}$  do  
4: Wait for the next source token  $x_{j+1}$ ;  
5:  $\hat{\mathbf{x}} \leftarrow \hat{\mathbf{x}} + x_{j+1}$ ;  
6:  $j \leftarrow j + 1$ ;  
7: end  
8:  
9: Calculate representation of latent segment  $\mathrm{seg}_k$  according to Eq.(2);  
10: while  $(b_{ik} == 1$  or  $\hat{\mathbf{x}} == \mathbf{x})$  and  $\hat{y}_{i-1} \neq \langle \mathrm{eos} \rangle$  do  
12: Generate the target token  $\hat{y}_i$  based on  $\hat{\mathbf{x}}$ ;  
13:  $\hat{\mathbf{y}} \leftarrow \hat{\mathbf{y}} + \hat{y}_i$ ;  
14:  $i \leftarrow i + 1$ ;  
15: end  
16:  
17:  $k \leftarrow k + 1$ ;  
18: end  
19: return  $\hat{\mathbf{y}}$ ;
We conduct experiments on the most common benchmarks of three representative simultaneous generation tasks, including streaming ASR, SimulMT and SimulST.
Streaming ASR We apply LibriSpeech<sup>3</sup> benchmark [59], which consists of 960 hours English audio. We use dev-clean (5.4 hours) and dev-other (5.3 hours) as validation sets, and test-clean (5.4 hours) and test-other (5.1 hours) as test sets, where test-other set contains more noisy audio. For speech, we use the raw 16-bit  $16\mathrm{kHz}$  mono-channel audio wave. For text, we use SentencePiece [60] to generate a unigram vocabulary of size 10000.
SimulMT We apply WMT $^{4}$  German  $\rightarrow$  English (De  $\rightarrow$  En) benchmark, including 4.5M sentence pairs for training. We use newstest2013 as validation set (3000 pairs), and newstest2015 as test set (2169 pairs). 32K BPE [61] is applied and vocabulary is shared across languages.
SimulST We apply MuST-C $^5$  English  $\rightarrow$  German (En  $\rightarrow$  De) (408 hours, 234K pairs) and English  $\rightarrow$  Spanish (En  $\rightarrow$  Es) (504 hours, 270K pairs) benchmarks [62]. We use dev as validation set (1423 pairs for En  $\rightarrow$  De, 1316 pairs for En  $\rightarrow$  Es) and tst-COMMON as test set (2641 pairs for En  $\rightarrow$  De, 2502 pairs for En  $\rightarrow$  Es), respectively. The pre-processing is the same as streaming ASR tasks.
4.2 Systems Settings
We conducted experiments on several strong baselines for all three tasks, described as follows.
Offline [55] model waits for the complete source sequence before generating the target sequence. Offline model is decoded with beam 5.
# Streaming Automatic Speech Recognition (Streaming ASR)
T-T [34] uses Transformer Transducer to determine waiting/generating via alignments from the joiner between the speech encoder and text predictor. Some methods, including ConstAlign [36], FastEmit [37] and SelfAlign [38] are proposed to further reduce the latency of the Transducer.
3https://www.openslr.org/12  
4https://www.statmt.org/wmt15/  
5https://ict.fbk.eu/must-c
MoChA [22] applies monotonic chunkwise attention to generate the target token based on the speech within a local window. Various training methods, such as DeCoT [39], MinLT [39] and CTC [40], are proposed to further constrain the latency of MoChA.
Simultaneous Machine Translation (SimulMT)
Wait-k [4] first waits for  $k$  source tokens, and then alternately generates and waits for one token.