Label-Synchronous Neural Transducer for E2E Simultaneous Speech Translation (2024)

Abstract: While the neural transducer is popular for online speech recognition, simultaneous speech translation (SST) requires both streaming and re-ordering capabilities. This paper presents the LS-Transducer-SST, a label-synchronous neural transducer for SST, which naturally possesses these two properties. The LSTransducer-SST dynamically decides when to emit translation tokens based on an Autoregressive Integrate-and-Fire (AIF) mechanism. A latency-controllable AIF is also proposed, which can control the quality-latency tradeoff either only during decoding, or it can be used in both decoding and training. The LSTransducer-SST can naturally utilise monolingual text-only data via its prediction network which helps alleviate the key issue of data sparsity for E2E SST. During decoding, a chunkbased incremental joint decoding technique is designed to refine and expand the search space. Experiments on the Fisher-CallHome Spanish (Es-En) and MuST-C En-De data show that the LS-Transducer-SST gives a better qualitylatency trade-off than existing popular methods. For example, the LS-Transducer-SST gives a 3.1/2.9 point BLEU increase (Es-En/En-De) relative to CAAT at a similar latency and a 1.4 s reduction in average lagging latency with similar BLEU scores relative to Wait-k.

heading: 1 Introduction
text: Simultaneous speech translation (SST) generates translations from input speech in a streaming fashion. Conventional cascaded SST performs streaming automatic speech recognition (ASR) followed by text-based simultaneous machine translation (Fügen et al., 2007). Recently, end-to-end (E2E) SST has become popular and has advantages, including lower latency (Ren et al., 2020; Ma et al., 2020c). However, E2E SST is challenging since it requires taking into account word re-ordering between source and target languages during the
streaming process (Chang and Lee, 2022). Neural transducers, the dominant model for low-latency ASR, find this difficult due to their monotonic nature. Furthermore, E2E training results in severe data sparsity (Bentivogli et al., 2021).
Current SST methods are normally based on the Transformer (Vaswani et al., 2017) attention-based encoder-decoder (AED) structure, which isn’t naturally able to deal with streaming, as noted by Xue et al. (2022). A popular approach to adapt the AED to SST is the Wait-k policy (Ma et al., 2020c), which is a fixed read-write policy that uses a fixed number of wait duration steps before translation. However, it can be too aggressive or conservative in different cases (Zheng et al., 2020). Alternative methods involve a flexible policy which lets the model decide how much input to read before generating the next translation token (Polák et al., 2023), such as monotonic multi-head attention (MMA) (Ma et al., 2020b) and Continuous Integrate-andFire (CIF) (Dong and Xu, 2020) based SST system. However, these flexible policies normally rely on a latency loss to adjust the quality-latency trade-off at training, unlike the fixed Wait-k policy that can control latency at decoding only.
To better address the challenges of E2E SST, this paper adapts the label-synchronous neural transducer (Deng and Woodland, 2023) developed for ASR to SST and denotes the resulting technique the LS-Transducer-SST. In the LS-Transducer-SST, an Auto-regressive Integrate-and-Fire (AIF) mechanism uses accumulated frame-level weights to dynamically determine when to emit translation tokens, based on which a label-level target-side encoder representation is extracted auto-regressively using an attention mechanism. Therefore, the LSTransducer-SST is naturally equipped with both streaming and re-ordering capabilities. In addition, the prediction network of the LS-Transducer-SST works as a standard language model (LM) as its output is directly combined with the extracted en-
8235
coder representation at the label level. As a benefit, the E2E SST data sparsity issue can be alleviated because the prediction network can effectively utilise monolingual text-only data, which is normally easy to collect, for tasks such as pre-training or text-based adaptation. While the standard AIF theoretically ensures low-latency output for the LS-Transducer-SST, to better control the qualitylatency trade-off, a latency-controllable AIF is proposed, which controls the latency by adjusting the decision threshold of the accumulated frame-level weights. Furthermore, the latency-controllable AIF allows the quality-latency trade-off to be controlled not only during training but also during decoding, enabling the LS-Transducer-SST to combine the advantages of typical fixed and flexible SST policies. This paper focuses on low/medium-latency scenarios to keep the low-latency advantage of E2E SST. During decoding, to improve translation quality, a chunk-based incremental joint decoding is further proposed to refine and expand the search space. The proposed LS-Transducer-SST was evaluated on Fisher-CallHome Spanish (Es-En) and MuSTC En-De corpora and gave an improved qualitylatency trade-off compared to existing popular SST methods. The main contributions are summarised below:
• LS-Transducer-SST, naturally equipped with streaming and reordering abilities, is proposed for SST and can help alleviate its data sparsity.
• A latency-controllable AIF is proposed to control the latency during decoding effectively.
• A chunk-based incremental joint decoding is proposed to expand the search space.
• Extensive experiments were conducted. Our code bridges the ESPnet and Fairseq toolkits and will facilitate future research1.

heading: 2 Related Work

heading: 2.1 Label-synchronous Neural Transducer
text: The standard neural transducer (Graves, 2012) is a frame-synchronous model, which operates on a frame-by-frame basis and uses blank tokens to augment the output sequence. However, blank token prediction is inconsistent with the LM task, which means the prediction network doesn’t operate as a standard LM and cannot effectively utilise text data (Chen et al., 2022). The label-synchronous neu-
1The code is available at: https://github.com/D-Keqi/ LS-Transducer-SST
ral transducer (Deng and Woodland, 2023) introduces an Auto-regressive Integrate-and-Fire (AIF) mechanism, which extracts label-level encoder representations that are directly combined with the prediction network at the label level. Therefore, the operation becomes label-synchronous and the prediction network is consistent with the LM task.
To be more specific, AIF learns frame-level weights (α1, · · · , αT ) for each encoder output frame E = (e1, · · · , eT ). To generate the i-th output unit, the weight αt is accumulated from left to right until it exceeds i, and then the time step of this located boundary is denoted as Ti+1. Hence, AIF estimates a monotonic alignment for streaming ASR. With this located boundary, the label-level representation haifi is extracted using dot-product attention with E1:Tj as the keys and values:
haifi = softmax(qi · E1:Ti⊤) · E1:Ti (1)
where the query qj can be the prediction network intermediate output. Suppose hpredi is the output of the prediction network which normally has the same structure as a Transformer LM, the joint network combines haifi and h pred i at the label level and computes the predicted logits li as follows:
li = FC(h aif i ) + FC(h pred i ) (2)
where FC denotes a fully-connected network that maps the dimension to the vocabulary size. Since the li has the same length as the target sequence, the cross-entropy loss can be used as the training objective. In addition, a connectionist temporal classification (CTC) loss is also computed by the encoder to improve model training.
In ASR decoding, streaming joint decoding is used that also considers an online CTC prefix score, which is modified to be strictly synchronised with the AIF alignment.

heading: 2.2 E2E Simultaneous Speech Translation
text: For E2E simultaneous speech translation (SST), the fixed Wait-k (Ma et al., 2020c) policy is still the most popular approach (Chang and Lee, 2022). While adopting this fixed policy, there are several studies that try to improve the “pre-decision" process, including using CTC (Ren et al., 2020; Zeng et al., 2021) and the Continuous Integrate-and-Fire (CIF) mechanism (Dong et al., 2022).
However, a flexible policy in which decisions to emit translation tokens is made on the fly, is desirable for simultaneous translation (Zheng et al.,
2020). A typical flexible policy is monotonic multihead attention (MMA) (Ma et al., 2020b), however, Chang and Lee (2022) noted its complex training techniques and proposed a flexible method CIF-IL, in which CIF is used directly to estimate when to output a translation token instead of being used as a pre-decision module. Given that CIF is a monotonic method with limited re-ordering capabilities, a Transformer decoder is built on top of the CIF output, in which infinite lookback (IL) cross-attention attends to the previous CIF output. However, CIF suffers from mismatched testing and training.
Xue et al. (2022) directly explored using a standard neural transducer for the SST task. However, the standard neural transducer suffers from limited re-ordering capabilities due to its monotonic alignment (Liu et al., 2021). To address this, the CAAT technique (Liu et al., 2021) augments it with cross-attention to remove the strong monotonic constraint, but leads to the exacerbated issue of using large memory during training. Similarly, (Tang et al., 2023) combined the Transducer and AED (TAED) by using the AED decoder as the prediction network for the Transducer. However, during training, the number of forward computations for the AED decoder needs to increase in proportion to the length of input speech. CAAT and TAED, to keep the streaming property while handling re-ordering, complicate the training due to the frame-synchronous nature, which is an important difference from our label-synchronous approach.
Recently, several papers (Liu et al., 2020; Papi et al., 2023a,b) used offline-trained AED models for SST inference. Local Agreement (LA) (Liu et al., 2020) generates two consecutive hypotheses
and takes agreeing prefixes as the stable hypothesis. Papi et al. (2023a) noted that this strategy affects latency and proposed to use encoder-decoder attention (EDATT) to decide when to emit translations.
However, here we focus on low and mediumlatency scenarios. Since the offline-trained models have a more serious mismatch when decoding with low and medium latency, here we focus on training models in a streaming manner.

heading: 3 Proposed Method: LS-Transducer-SST
text: The LS-Transducer-SST, as shown in Fig. 1, has four key components: encoder, AIF, prediction network, and joint network. The encoder and AIF extract label-level target-side representations (haif1 , · · · ,haifL ) from the source-language speech in a streaming fashion, while AIF controls the time steps at which the representations are emitted.
In addition, the prediction network is an autoregressive structure, e.g. Transformer LM structure, which generates representations (hpred1 , · · · ,hpredL ) based on previous translation tokens. Since the representations obtained from AIF and the prediction network have the same length, the joint network can directly add the logits obtained from them using linear fully-connected layers, and the prediction network performs as an explicit LM.
Hence, the output of the LS-Transducer-SST is a 2-dimensional matrix RL·V (V is the vocabulary size), which can use a cross-entropy loss computed with target translations for training and resolve the issue of expensive training for the standard neural transducer, whose output is a 3-dimensional tensor.

heading: 3.1 Latency-controllable AIF
text: In the LS-Transducer-SST, AIF computes framelevel weights (α1, · · · , αT ) for each encoder out-
put frame E = (e1, · · · , eT ) to dynamically decide how much input to read before emitting the next translation token giving a flexible policy. Following (Deng and Woodland, 2023), the last element et,d of each frame et was used as the raw scalar attention value to compute αt. To mitigate overly sharp weights, this paper proposes a smoothing process to compute the weights using a smoothing factor δ as follows:
αt = (1− δ)× sigmoid(et,d) + δ (3)
where d is the dimension size of et. To decide when to emit the i-th translation token, αt is accumulated from left to right until it exceeds the decision threshold i, as shown in the black solid line of Fig. 2. Suppose this time step is Ti+1, E1:Ti is used to extract haifi . Note if the decision threshold has not been reached until all the speech has been read, Ti = T , i.e. the entire E is used to extract the haifi . To help AIF learn this cross-lingual speech-text alignment, a target-side CTC branch is computed which can encourage the Transformer encoder to re-order the output according to the target translation sequence (Chuang et al., 2021; Deng et al., 2022). Furthermore, a quantity loss Lqua = | ∑T i=1 αi − L| is computed to ensure that
the accumulated AIF weight ∑T
i=1 αi approaches the target translation token length L.
After obtaining the output time step Ti, in contrast to (Deng and Woodland, 2023) that uses simple dot-product attention to generate representations for the ASR task, preliminary experiments showed that multi-head attention is more effective for the SST task, so haifi is computed as:
haifi = Multihead-attention(qi,E1:Ti ,E1:Ti) (4)
where E1:Tj is used as the keys and values, and the query qi is the prediction network intermediate output at the i-th step as shown in Fig. 1.
The AIF mechanism provides a natural approach to control the latency by adjusting the decision threshold. In the standard AIF, the decision threshold of the i-th translation token is i. By adding a hyper-parameter ϵ into the decision threshold, i.e. i+ϵ, the quality-latency trade-off can be controlled, which is called the latency-controllable AIF. As shown in Fig. 2, the red dotted line represents the case of ϵ = 1, in which more input speech will be read before deciding to output the i-th translation token, thus the translation quality improves at the cost of increased latency.
The latency-controllable AIF has many advantages over conventional E2E SST systems. First, it only uses one hyper-parameter ϵ to achieve finegrained latency control and can meet any latency requirements, because ϵ is not limited to integer values. Compared to the fixed Wait-k policy (Ma et al., 2020c) that normally needs to set two hyperparameter values, including the pre-decision ratio and k, the latency-controllable AIF is easier to tune. Second, in contrast to a typical flexible policy2 that uses a latency loss to control the quality-latency trade-off during training (Ma et al., 2020c; Chang and Lee, 2022), the latency-controllable AIF can control the latency at decoding time while only requiring a single trained model.

heading: 3.2 Chunk-based Incremental Joint Decoding
text: For the ASR task, the label-synchronous neural transducer (Deng and Woodland, 2023) is decoded based on beam search. However, for the SST task,
2We note some flexible policies applied to offline-trained models (Papi et al., 2023a,b) can control latency in decoding.
beam search re-ranks the top hypotheses while reading the input speech, making it hard to process the translation results and evaluate the latency. Hence, this paper proposes chunk-based incremental joint decoding, which prunes the hypotheses to the same prefix within a chunk. The chunk-based streaming Transformer is reviewed in this section before describing chunk-based incremental pruning.

heading: 3.2.1 Chunk-based Streaming Transformer
text: This paper uses the Chunk-based (Li et al., 2020) Transformer encoder to achieve streaming, which uses a chunk mask to limit the range of query-key dot products for each frame within the Transformer self-attention. As shown in Fig.3, the chunk mask allows the query to be computed only with the keys from the current and previous (history) chunks.

heading: 3.2.2 Incremental Pruning within Chunk
text: Since the emitted translation tokens inside a chunk actually all correspond to the same chunk speech duration, in order to expand the search space without adding extra latency, beam search can be used within a chunk while selecting only the highestscoring hypothesis after outputting the last token of this chunk. However, it is not always feasible to know whether a token is the last one in a chunk, i.e., to know in advance if the speech input required for the next token will exceed the range of this chunk.
In the LS-Transducer-SST, the AIF mechanism uses frame-level weights α to decide whether to output a translation token or not, so comparing the accumulation of frame-level weights up to the current chunk with the decision threshold of the next translation output token, it can be confirmed if a token is the last one in a chunk. As shown in the example in Fig. 3, the accumulated weights ∑t=7 t=1 αt up to the first chunk is 2.3, so in the standard AIF case, when outputting the second token y(2)j (j can be e.g. 1, · · · , 4 in Fig. 3), it is known that the decision threshold for the next token (i.e. 3) cannot be reached within this chunk. Hence, pruning is required when emitting the 2nd translation token, i.e. only the highest-scoring hypothesis is kept. A similar situation occurs with the 5th token in Fig. 3.

heading: 3.3 Training
text: During training, as mentioned in Sec. 3, the LSTransducer-SST uses the cross-entropy (CE) loss LCE as the training objective since the joint network output is a 2-dimensional matrix. In addition, a target-side CTC branch and the AIF quantity loss
Lqua are also computed. Hence, the final training objective of the LS-Transducer-SST is as follows:
L = βLCTC + (1− β)LCE + γLqua · L (5)
where L is the target translation token length as Lqua is a sentence-level loss. β is the target-side CTC weight and γ is the weight of the Lqua.
Since the LS-Transducer-SST prediction network works as a standard LM, it can be initialised by a pre-trained target-language LM before SST training. If an unseen domain is encountered during decoding, target-language target-domain text can be used to fine-tune the prediction network, giving flexible domain adaptation. Since monolingual text is normally easy to collect, the LS-Transducer-SST can help alleviate data sparsity issues in E2E SST.

heading: 3.4 Summary
text: In summary, this paper enhances the re-ordering capability of the label-synchronous neural transducer by introducing a target-side CTC branch, enabling AIF to decide when to emit translation tokens in SST tasks. A chunk-based incremental joint decoding is proposed to meet SST requirements, which prunes hypotheses within a chunk while expanding the search space. To flexibly control the quality-latency trade-off, a latency-controllable AIF is proposed that can be used at the decoding stage. With these contributions, the LS-TransducerSST becomes a natural SST method combining the advantages of typical fixed and flexible SST policies.

heading: 4 Experimental Setup

heading: 4.1 Dataset
text: SST models were trained on the Fisher-CallHome Spanish (FCS) (Post et al., 2013) and MuST-C v1.0 (Di Gangi et al., 2019) English-German (En-De) datasets. The dev/test sets of Europarl-ST (IranzoSánchez et al., 2020) Spanish-English (Es-En) were used as cross-domain test sets for the FCS corpus. The monolingual source-domain text-only data for FSC was the training set English translations and Fisher (Cieri et al., 2004) transcriptions. For the MuST-C En-De, the training set German translations and German text from TED2020 (Reimers and Gurevych, 2020) were used. The English text from Europarl-ST was used as the target-domain text. More data details are listed in Appendix A.
SST Experiments were implemented based on the ESPnet-ST (Inaguma et al., 2020) toolkit. To
evaluate the latency, SimulEval (Ma et al., 2020a) was used to measure the speech version of the word-level Average Lagging (AL) (Ma et al., 2018, 2020c). Detokenized BLEU (Papineni et al., 2002) results are reported to evaluate translation quality.

heading: 4.2 SST Model Descriptions
text: SST models built in this paper have a streaming wav2vec2.0 (Baevski et al., 2020) encoder3, which was fine-tuned with a chunk-based mask of 64 chunk size, resulting in 640 ms average latency. The encoder was first pre-trained for ASR before SST training (Inaguma et al., 2020). More training and decoding details can be found in Appendix B.
The proposed LS-Transducer-SST was built with a 6-layer unidirectional Transformer-encoderbased prediction network. The prediction network was initialised by a pre-trained source-domain LM and then fixed during SST training. Four different baseline models were implemented. First, the widely-used Wait-k fixed policy model was built, which followed the Transformer AED structure that contained a 6-layer Transformer decoder. The quality-latency trade-off of the Wait-k was adjusted by varying k and the fixed pre-decision step size.
3Note: Section 5.5 has a different setup to aid comparisons.
Second, the flexible policy method CIF-IL (see Sec. 2.2) which can performs well in low-latency scenarios (Chang and Lee, 2022) was built with a 6- layer Transformer decoder. The latency loss from (Chang and Lee, 2022) was used to control latency during training with a weight denoted λlat. Third, the CAAT (Liu et al., 2021) was built, the prediction network had the same structure as that of LSTransducer-SST, and the joint network consisted of 6-layer Transformer decoder with self-attention modules removed. Note that knowledge distillation used in (Liu et al., 2021) was not employed in the Main Experiments in this paper in order to compare fairly with other specifically built models. The decision step size d was set to 32.4 Finally, a standard neural transducer was built with the same prediction network as the CAAT.

heading: 4.3 LM and Text Adaptation
text: The source-domain Transformer LM was trained on the monolingual source-domain text-only data for 50 epochs and further fine-tuned on the crossdomain text for 15 epochs as a target-domain LM. If a density ratio (McDermott et al., 2019) was used for domain adaptation, the weights for the sourcedomain and target-domain LMs were 0.3. When conducting text-only domain adaptation for the LSTransducer-SST, the prediction network was finetuned on the cross-domain text for 25 epochs.

heading: 5 Experimental Results
text: The LS-Transducer-SST was compared to fixed and flexible policy models. To maintain the lowlatency advantage of E2E SST, we focus on the low and medium latency scenarios, i.e. AL < 1 s and 1 s < AL < 2 s following (Ansari et al., 2020).

heading: 5.1 Main Experiments
text: Table 1 gives the SST results on Fisher-CallHome Spanish (FSC) data, on which our models yield good results compared to recent work. For the popular Wait-k model, a large pre-decision step (e.g., 280 ms or 360 ms in Table 1) is important to achieve promising translation quality, which makes it suitable for medium or high-latency scenarios while performing poorly when low-latency. In contrast, the flexible policy CIF-IL outperformed the Wait-k model in low-latency scenarios. In addition, the standard neural transducer didn’t work well on the SST task due its monotonic constraint and
4Appendix B explains the reasons to set d to a fixed value.
limited re-ordering ability. It tends to accumulate more information before emitting translations, resulting in medium latency. However, CAAT, which augments the standard neural transducer with cross attention to remove the monotonic constraint, yielding good BLEU results in low-latency scenarios, outperforming the Wait-k and CIF-IL.
While these existing methods showed good results, the proposed LS-Transducer-SST gave a significantly better quality-latency trade-off with the latency-controllable AIF5. Compared to CAAT, for similar latency (i.e. AL ≈ 0.9s), the LS-TransducerSST obtained a 3.1 BLEU gain. Compared to the best BLEU results for Wait-k in Table 1, the LSTransducer-SST gave competitive BLEU results at 0.759 s AL latency (1.37 s AL reduction).
The results on the MuST-C En-De are shown in Table 2 and the conclusions were consistent with that of FSC corpus: 1. the Wait-k model worked well in medium or high-latency regimes while the CIF-IL surpassed it for low-latency case; 2. CAAT greatly outperformed the Wait-k and CIF-IL when AL is around 1 s; 3. the proposed LS-TransducerSST yielded an improved quality-latency trade-off compared with these existing methods. When the latency was around 1 s AL, the LS-TransducerSST outperformed CAAT with a 2.9 BLEU gain. Compared to the Wait-5 policy with 320 ms predecision step size, the LS-Transducer-SST could
5Note unless specifically stated, the latency-controllable AIF refers to being used during both training and decoding.
give a competitive translation quality with 1.188 s AL latency, resulting in a 1.44 s AL reduction.
Quality-latency trade-off curves corresponding to Tables 1 and 2 are shown in Appendix C. Results with additional metrics are given in Appendix E. A visual example in Appendix I.1 illustrates the re-ordering capability of the LS-Transducer-SST.

heading: 5.2 Cross-domain Experiments
text: The Europarl-ST dev/test sets were used to evaluate the cross-domain performance of the SST models trained on FSC data. As shown in Table 3, the LS-Transducer-SST performed robustly regarding the latency and translation quality in cross-domain scenarios. From the source domain (i.e. FCS) to the cross-domain, the AL result of LS-Transducer-SST only increased slightly (from 0.759s to 0.915s), whereas the latency of Wait-k more than doubled (from 2.129s to 4.603s), since in cross domain, the number of byte pair encoding (BPE) (Gage, 1994) units for target translations tends to be relatively longer than for the source domain as the BPE model is trained on source-domain text6. For example, on the evltest set of FSC, each BPE unit corresponds to an average of 207 ms speech, whereas on the test set of Europarl-ST, this value is 148 ms. This shows the robust advantage of the LS-TransducerSST as a flexible policy, as the fixed Wait-k policy tends to be affected by the data distribution. A case study is shown in Appendix I.2.
The LS-Transducer-SST exceeded the BLEU score of Wait-k in cross-domain scenarios, showing its robust generalisation. Moreover, since the LS-Transducer-SST prediction network operates as an explicit LM and can directly use targetlanguage target-domain text for fine-tuning, the cross-domain BLEU results of the LS-Transducer
6Note the modelling unit is BPE, whose output time step needs to be mapped to the word level to measure latency.
were further improved with an adapted prediction network. Even when Wait-k used the density ratio method (McDermott et al., 2019) to subtract a source-domain LM score and add a target-domain LM score during decoding for domain adaptation, there was still a 1.7 BLEU gap with LS-TransducerSST, which is more flexible as it does not rely on an additional external LM. In addition, shallow fusion (Chorowski et al., 2015) can also be used to further improve the LS-Transducer-SST with the target-domain LM (see Table 3).

heading: 5.3 Analysis of Latency-controllable AIF
text: Fig. 4 shows the quality-latency trade-off curves of the LS-Transducer-SST, in which test-time latencycontrollable AIF means using the standard AIF (i.e. ϵ = 0) during training and adjusting ϵ only during decoding. In general, the latency-controllable AIF can flexibly and efficiently control the qualitylatency trade-off. While the latency-controllable AIF with consistent training and decoding performed slightly better than the test-time-only one, adjusting ϵ only in the decoding stage achieves similar results, which is important in real-world deployment as only one model needs to be maintained. Hence, as a flexible policy, the proposed LSTransducer-SST possesses the advantage of controlling the latency during decoding, which is normally seen in fixed policies like Wait-k.

heading: 5.4 Ablation Studies
text: The LS-Transducer-SST can effectively utilise monolingual text data by initialising its prediction network with a pre-trained LM. Ablation studies were conducted to evaluate the effectiveness of this initialisation. As shown in Table 4, the prediction network initialisation was highly effective for the
LS-Transducer-SST and was essential to surpass existing methods like CAAT. The monolingual textonly data is normally easier to collect, which can help alleviate E2E SST data sparsity. However, pre-training the prediction network did not help for CAAT. This is because the CAAT inherits the frame-synchronous property from the standard neural transducer, in which the prediction network is inconsistent with the LM task (Chen et al., 2022).
Ablation studies on chunk-based decoding and multi-head attention for AIF are in Appendix D.

heading: 5.5 Comparisons to Recent Work
text: To enable further comparisons, the TAED results from (Tang et al., 2023), the CAAT results from (Liu et al., 2021) and (Papi et al., 2023a), and the EDATT, LA, and Wait-k results from (Papi et al., 2023a) were compared to the LS-Transducer-SST in the low/medium-latency scenarios that is our focus. To allow a fair comparison, in contrast to the Main Experiments, a streaming Conformer (Gulati et al., 2020) encoder was employed in the LSTransducer-SST, and sequence-level knowledge distillation (KD) (Kim and Rush, 2016) used7. Detailed training settings are given in Appendix G.
As shown in Fig. 5, CAAT shows good results in the low-latency region, while TAED performs well around 1.4 s. In general, CAAT and TAED inherit the neural transducer low-latency advantage, which also motivates our work. However, the translation quality does not always improve with higher latency for CAAT and TAED. As latency continues to increase, the mismatch with the offline case reduces, and strategies that use offlinetrained models for SST inference, like EDATT, begin to surpass other published results. However, LS-Transducer-SST (solid blue line in Fig. 5) still outperforms other models in both low and mediumlatency regions (up to an AL of about 1.7 s). Fig. 5 also shows that a wav2vec2.0 encoder gives fur-
7Note only Section 5.5 uses the Conformer and KD
ther performance benefits. Online training is especially useful in lower latency operation and the LS-Transducer-SST is able to effectively adjust the quality-latency trade-off. Numerical values for Fig 5, along with the additional LAAL metric, are given in Appendix H.

heading: 6 Conclusions
text: This paper proposes the LS-Transducer-SST, which naturally possesses streaming and re-ordering capabilities. By introducing the target-side CTC branch, the re-ordering capability of the label-synchronous neural transducer is enhanced and thus the AIF mechanism can be used for SST. Therefore, the LSTransducer-SST is a flexible policy method that dynamically decides when to emit translation tokens using the AIF. A latency-controllable AIF is further proposed to effectively control the latency at decoding or training, enabling the LS-Transducer-SST to combine the advantages of typical fixed and flexible SST policies. In addition, the LS-Transducer-SST provides a natural way to utilise monolingual textonly data, which helps alleviate the E2E SST data sparsity issue. During decoding, a chunk-based incremental joint decoding is further proposed to refine and expand the search space. With the focus on low and medium-latency scenarios, experiments showed that the LS-Transducer-SST gives a better quality-latency trade-off than existing methods.
Limitations
This paper focuses on low-latency and mediumlatency scenarios to maintain the low-latency advantages of E2E SST models, although some highlatency results are also shown. To explore solutions for high-latency SST, a comparison against cascaded SST systems is worthwhile. However, in real-world deployment, the cascade SST system normally has more available training data than the E2E SST system, such as machine translation and ASR data. The appropriate experimental setup to simulate this real deployment scenario and reasonably compare different SST systems is not straightforward, which is regarded as future work.
Sequence-level knowledge distillation (KD), which uses a neural machine translation model to generate pseudo target text given source text paired with source speech and augments the original data with it (Kim and Rush, 2016), was not used in the Main Experiments (it is used in Section 5.5 to allow a close comparison with results from the literature). The reasons are as follows: firstly, this technique requires the source text, i.e. the ASR transcripts, which are not necessarily always available. Secondly, we use the ESPnet-ST toolkit, in which this KD is not the default setting and in general we have followed the standard ESPnet-ST recipes. Thirdly, this method will double the data size, requiring twice the computational resources. This paper implements four different SST methods to compare with the LS-Transducer-SST, along with adjusting the quality-latency trade-off, making the experiments computationally expensive. Sequence-level KD can be regarded as translation data augmentation and would not be expected to affect the comparison between different methods. To verify this, we selected some key experiments and give the results after using KD in Appendix F.
Section 5.5 makes further comparisons with recent work from the literature. We have aimed to make it as close a comparison as possible with the literature and hence a Conformer encoder trained on the source speech was used in place of the wav2vec2.0 encoder used in the Main Experiments and also all comparisons use sentence-level KD. However, there are many details which may not be strictly comparable across different papers. For example, the teacher model involved in the sentencelevel KD will in general be different between different papers and hence yield different pseudo target text. Also, some recent papers have not yet released
their code. Furthermore, even when papers do provide code, others may not always be able to fully reproduce the original results (e.g. the two CAAT results in Fig. 5). We believe that the comparing to prior published work while attempting to control the precise experimental conditions in Section 5.5 and Appendix H is the best solution for comparing to the recent papers cited.
The LS-Transducer-SST has so far been evaluated on two European language pairs in this paper, each in a single translation direction (i.e. Es-En and En-De), and while we believe that the technique can be applied to other languages (including nonEuropean languages) and translation directions, the performance has not been verified and is left as future work.
Ethics Statement
E2E SST systems suffer from data sparsity issues because speech-translation parallel data is expensive to collect. For under-resourced languages or domains, this issue will be more severe. This can result in a poor user experience for minorities, making minority views to be under-represented or misunderstood. The LS-Transducer-SST proposed in this paper could be beneficial to this concern as it provides a natural approach to utilise monolingual text-only data, which is normally easy to collect, for pre-training and text-based domain adaptation.

heading: Acknowledgments
text: Keqi Deng is funded by the Cambridge Trust. This work has been performed using resources provided by the Cambridge Tier-2 system operated by the University of Cambridge Research Computing Service (www.hpc.cam.ac.uk) funded by EPSRC Tier2 capital grant EP/T022159/1.

heading: A Statistics
text: The training and test statistics are shown in Table 5. The data was pre-processed following standard ESPnet-ST recipes, in which speed perturbation was employed with factors 0.9 and 1.1. Following the ESPnet-ST recipes, raw source speech was used as input, and 500 and 4000 BPE (Gage, 1994) were used as the modelling units for FCS and MuST-C En-De, respectively. Model training was performed on 4 NVIDIA A100 GPUs each with 80GB GPU memory. For the Fisher-CallHome Spanish corpus, each epoch of training took about 26 minutes. For the MuST-C En-De corpus, each training epoch consumed about 80 minutes.

heading: B Hyper-parameters and Inference
text: For the wav2vec2.0 encoder provided by Fairseq (Ott et al., 2019), "xlsr_53_56k" was used for FCS data and “wav2vec_vox_new" was used for MuSTC data. SST training was for 35 and 20 epochs for FCS and MuST-C En-De, respectively. The hyper-parameters of the models we built are as follows, with other hyper-parameters following standard ESPnet-ST recipes.
LS-Transducer-SST The LS-Transducer-SST had a 6-layer unidirectional Transformer-encoderbased prediction network (1024 attention dimen-
sion, 2048 feed-forward dimension, and 8 heads), resulting in 372.09 M parameters. The 3rd sublayer output of the prediction network was used as the query of the AIF mechanism. δ in Eq. 3 was set to 0.05. β and γ in Eq. 5 were respectively set to 0.6 and 0.05. The beam size within a chunk was 10 during the chunk-based incremental joint decoding.
Wait-k The Wait-k model (404.66 M parameters) had a 6-layer Transformer decoder (1024 attention dimension, 2048 feed-forward dimension, and 8 heads). The decoding process is similar to the chunk-based incremental decoding of the LS-Transducer-SST. By counting the fixed predecision step size and the waiting steps k, it is easy to know whether a token is the last one of a chunk, i.e., whether incremental pruning is needed.
CIF-IL The CIF-IL model (393.63 M parameters) had a 6-layer Transformer decoder (1024 attention dimension, 2048 feed-forward dimension, and 8 heads). As explained in Section.2.2, the CIFIL used CIF to estimate when to emit translation tokens and the 6-layer Transformer decoder was built on top of the CIF output. The decoding process was similar to the incremental decoding of the LSTransducer-SST because AIF is an improved version of CIF, and they both use frame-level weights to decide when to emit.
CAAT (Liu et al., 2021) chose block-based streaming Transformer encoder and relied on d to adjust the quality-latency trade-off. However, tuning d is not very suitable for the chunk-based Transformer encoder used in this paper, and preliminary experiments showed that further increasing d did not improve performance, which was also reported in (Papi et al., 2022a). Therefore, this paper only sets d to a fixed value of 32. The attention dimension, feed-forward dimension, and attention heads of the prediction network and the joint network were set to 1024, 2048, and 8. The total parameters are 418.32 M. During decoding, within each chunk, beam search was used while chunk-based incremental pruning was performed at the last token, which is easy to decide as the CAAT is still a frame-synchronous model.
Standard Neural Transducer Decoding was based on beam search with chunk-based incremental pruning. Since the speech is decoded on a perframe basis, it’s easy to know whether a token is the last one of a chunk. The parameters are 369.04 M.
C Visualisation: Main Experiments
The quality-latency trade-off curves in Fig. 6 and Fig. 7 respectively correspond to the results from Table 1 and Table 2 of the Main Experiments.

heading: D Further Ablation Studies
text: The results of further ablation studies for the Main Experiments are included in this section to evaluate the effectiveness of the chunk-based incremental joint decoding and the usage of multi-head attention for AIF.
The results in Table 6 show that using beam search after reading the whole speech (i.e. tail beam search) is better than greedy decoding, and using beam search within each chunk (i.e., chunkbased incremental decoding) performs the best, which it is because it more fully utilises the beam search to widen the search space.
The results in Table 7 show that the multi-head attention works better than the simple dot-product attention for the AIF, which is consistent with the success of the Transformer (Vaswani et al., 2017).

heading: E Expanded Results: Main Experiments
text: In addition to the AL metric, in this section, the results from Table 1 and Table 2 in the main experiment are additionally measured using the LengthAdaptive Average Lagging (LAAL) (Papi et al., 2022b) metric. As shown in Table 8 and Table 9, consistent with the findings of (Papi et al., 2022b), the Wait-k model tends to have similar AL and LAAL values, indicating a tendency to generate less compared to other models. Overall, the experimental conclusions are consistent for both metrics.
Furthermore, a computation-aware (CA) metric (i.e. AL_CA) was also considered with A100 GPU. As shown in Table 10 and Table 11, the increase from AL to AL_CA values is smaller for LS-Transducer-SST than for CAAT, indicating that LS-Transducer-SST decodes faster.

heading: F Results with Knowledge Distillation
text: Sequence-level knowledge distillation (KD) (Kim and Rush, 2016) was not used in the main experiment because this is not the default setting of the ESPnet-ST toolkit we used and would double the computational cost. In this section, we select some key data and show the results of using the KD.
As shown in Table 12, the sequence-level KD was effective in improving translation quality at the cost of increased training computation. The experimental conclusions were consistent with and
without this KD, i.e., CAAT exceeded Wait-k and CIF-IL when AL was around 1 s, while the proposed LS-Transducer-SST gave both lower AL latency and higher BLEU scores than other systems.

heading: G Training Setting for Section 5.5
text: A chunk-based streaming Conformer (Gulati et al., 2020) encoder was built and used only in Section 5.5. 80-dimensional filter bank was computed as the input feature which was computed every 10 ms with a 25 ms window. The speech features were down-sampled by a factor of 4 via two causal convolution layers before being fed into a 12-layer chunk-based streaming Conformer encoder, in which the chunk size was 32. Therefore, the average latency from the chunk-based encoder was 640 ms, which was the same as the main experiment. The attention dimension, feed-
forward dimension and attention heads were set to 512, 2048, and 8. δ in Eq. 3 was set to 0.05 as in the main experiment but the resulting αt was multiplied by 2, considering that here the frame stride of the encoder output was twice that of the main experiment. ϵ was set to {−2,−1, 0, 1, 3} for the latency-controllable AIF. Sequence-level knowledge distillation (Kim and Rush, 2016) was built using the code provided by Liu et al. (2021).

heading: H Numerical Values for Table 5
text: Numerical values for Fig. 5 are shown in Table 13. In addition, to AL values shown in Fig. 5, the table also includes the Length-Adaptive Average Lagging (LAAL) (Papi et al., 2022b) metric.
I Visual Case Study
I.1 MuST-C En-De A visual example from the MuST-C En-De tstCOMMON set is given in Fig. 8 to show the re-ordering capability of the LS-Transducer-SST, where the LS-Transducer-SST successfully output “hat nicht nur" that corresponds to the English transcripts “doesn’t just have". Note the chunk size of the streaming Transformer encoder is 64, which means a range of 1.28 s.
I.2 Cross-domain Europarl-ST Es-En A visual case study is given to compare the LSTransducer-SST and the Wait-k in the cross-domain Europarl-ST Es-En test set. As shown in Fig. 9, the Wait-k model has read the whole speech input from the first BPE unit, because 18× 5× 0.02 = 1.8 s (0.02 s is the frame stride), which is greater than the 1.27 s length of the speech. This is a simple example without re-ordering, where the LS-TransducerSST always outputs each translation token after reading the corresponding source-language speech.
Another example in Fig. 10 from the Europarl-ST Es-En test set contains a re-ordering case (“problema real" to “real problem"). The LS-TransducerSST decided to output the English word “real" until reading the corresponding source speech.

heading: Future Work
text: The appropriate experimental setup to simulate this real deployment scenario and reasonably compare different SST systems is not straightforward, which is regarded as future work. Sequence-level knowledge distillation (KD), which uses a neural machine translation model to generate pseudo target text given source text paired with source speech and augments the original data with it (Kim and Rush, 2016), was not used in the Main Experiments (it is used in Section 5. 5 to allow a close comparison with results from the literature). The reasons are as follows: firstly, this technique requires the source text, i. e. the ASR transcripts, which are not necessarily always available. Secondly, we use the ESPnet-ST toolkit, in which this KD is not the default setting and in general we have followed the standard ESPnet-ST recipes. Thirdly, this method will double the data size, requiring twice the computational resources. This paper implements four different SST methods to compare with the LS-Transducer-SST, along with adjusting the quality-latency trade-off, making the experiments computationally expensive. Sequence-level KD can be regarded as translation data augmentation and would not be expected to affect the comparison between different methods. To verify this, we selected some key experiments and give the results after using KD in Appendix F. Section 5. 5 makes further comparisons with recent work from the literature. We have aimed to make it as close a comparison as possible with the literature and hence a Conformer encoder trained on the source speech was used in place of the wav2vec2. 0 encoder used in the Main Experiments and also all comparisons use sentence-level KD. However, there are many details which may not be strictly comparable across different papers. For example, the teacher model involved in the sentencelevel KD will in general be different between different papers and hence yield different pseudo target text. Also, some recent papers have not yet released
their code. Furthermore, even when papers do provide code, others may not always be able to fully reproduce the original results (e. g. the two CAAT results in Fig. 5). We believe that the comparing to prior published work while attempting to control the precise experimental conditions in Section 5. 5 and Appendix H is the best solution for comparing to the recent papers cited. The LS-Transducer-SST has so far been evaluated on two European language pairs in this paper, each in a single translation direction (i. e. Es-En and En-De), and while we believe that the technique can be applied to other languages (including nonEuropean languages) and translation directions, the performance has not been verified and is left as future work