Decoder-only Streaming Transformer for Simultaneous Translation (2024)

Abstract: Simultaneous Machine Translation (SiMT) generates translation while reading source tokens, essentially producing the target prefix based on the source prefix. To achieve good performance, it leverages the relationship between source and target prefixes to exact a policy to guide the generation of translations. Although existing SiMT methods primarily focus on the Encoder-Decoder architecture, we explore the potential of Decoder-only architecture, owing to its superior performance in various tasks and its inherent compatibility with SiMT. However, directly applying the Decoder-only architecture to SiMT poses challenges in terms of training and inference. To alleviate the above problems, we propose the first Decoder-only SiMT model, named Decoder-only Streaming Transformer (DST). Specifically, DST separately encodes the positions of the source and target prefixes, ensuring that the position of the target prefix remains unaffected by the expansion of the source prefix. Furthermore, we propose a Streaming Self-Attention (SSA) mechanism tailored for the Decoder-only architecture. It is capable of obtaining translation policy by assessing the sufficiency of input source information and integrating with the soft-attention mechanism to generate translations. Experiments demonstrate that our approach achieves state-of-theart performance on three translation tasks1.

heading: 1 Introduction
text: Simultaneous Machine Translation (SiMT) (Gu et al., 2017; Ma et al., 2019) is designed for generating translations in real-time scenarios such as online conferences and real-time subtitles. It predicts the target tokens (i.e., target prefix) based on the already read source tokens (i.e., source prefix), aiming to achieve good tradeoffs between latency and translation quality. During training, SiMT models
* Corresponding author: Yang Feng. 1Code is at https://github.com/ictnlp/DST
need to learn the correspondence between source and target prefixes, crucial for extracting policies that ensure superior performance during inference (Zhang and Feng, 2022c).
Existing research on SiMT primarily focuses on the Encoder-Decoder architecture and is categorized into fixed and adaptive policies. For fixed policy (Dalvi et al., 2018; Ma et al., 2019; Elbayad et al., 2020), the model utilizes heuristic rules to determine the source prefix used for generating translations, which ignores the correspondence between the source and target prefixes. This may lead to redundant or missing source information during translation, resulting in inferior performance (Zhang and Feng, 2022a). For adaptive policy (Ma et al., 2020b), the model dynamically decides whether to read or output tokens based on the relationship between the source and target prefixes. This dynamic adjustment of policy in response to the translation status allows for improved tradeoffs (Zhao et al., 2023). However, there is a lack of exploration in SiMT regarding the Decoder-only architecture.
With the rise of language models, the Decoderonly architecture has exhibited superior performance across diverse tasks (Touvron et al., 2023; Team et al., 2024). As illustrated in Figure 1,
8851
the Decoder-only architecture, compared to the Encoder-Decoder architecture with an equivalent number of parameters, can support more layers, thereby offering parameter efficiency and better scalability (Liu et al., 2023). Importantly, SiMT relies on unidirectional encoding (Elbayad et al., 2020), and the Decoder-only architecture seamlessly accommodates this requirement. Therefore, we explore the capability of Decoder-only architecture in SiMT.
However, directly applying the Decoder-only architecture to the SiMT task poses challenges in both training and inference. During inference, with the arrival of each source token, there is a position increase of the generated target prefix, necessitating the model to re-encode the target prefix. This exacerbates the inference cost, particularly at low latency (Wang et al., 2024). During training, the model learns to predict the corresponding target prefix based on a given source prefix. Consequently, it is necessary to construct corresponding target prefixes for all possible source prefixes in a sentence pair. This limitation hinders the model from learning the translation policy and leads to an increase in training costs compared to the EncoderDecoder architecture.
To overcome the above limitations, we propose the first SiMT model based on Decoder-only architecture, named the Decoder-only Streaming Transformer (DST). To alleviate the issue of reencoding, DST encodes the positional information of the source prefix and the target prefix separately. This ensures that the expansion of the source prefix does not impact the position of the generated target prefix, thereby reducing the inference costs. To assess the contribution of partial source information to generating target tokens, DST uses the proposed Streaming Self-Attention (SSA) in replace of the conventional masked self-attention in the Decoder layer to decrease training costs and derive a translation policy.
During training, SSA can consider all possible source prefixes for the target prefixes in a sentence pair. Specifically, SSA predicts attention allocation for different source prefixes and combines it with the soft-attention mechanism to obtain expected attention for all source tokens and tokens in the target prefix. This expected attention is then utilized to derive the context vector. By leveraging SSA, the model learns the importance of all source prefixes in translating the target prefix, thereby reducing training costs. During inference, SSA accumulates
the allocated attention from all prefixes of the input source tokens, enabling an assessment of the sufficiency of input source information for generating translation. The model utilizes this assessment to determine whether to read or generate tokens, thereby acquiring the translation policy. Experiments demonstrate that DST achieves state-of-theart performance on three tasks.

heading: 2 Background
text: Simultaneous Machine Translation The SiMT model (Ma et al., 2019) dynamically reads the source sentence x = (x1, ..., xJ) with length J and generates translation y = (y1, ..., yI) with length I based on a policy. To articulate the policy, we introduce gi, representing the number of source tokens involved in translating yi. Thus, the translation policy from x to y can be denoted as g = (g1, ..., gI). During training, the SiMT model undergoes optimization by minimizing the cross-entropy loss:
Lsimt = − I∑
i=1
log p(y⋆i | x≤gi ,y<i), (1)
where y⋆i represents the ground-truth token.
Masked Self-Attention Masked self-attention allows each position in the decoder to attend to all positions in the decoder up to and including that position (Vaswani et al., 2017), ensuring the auto-regressive generation. Given the target hidden states s = (s1, ..., sI), the context vector is computed as follows:
ei,k = siW Q(skW K)⊺/ √ d, (2)
αi,k =    exp(ei,k)/ i∑ l=1
exp(ei,l) if k ≤ i 0 otherwise ,
(3)
ci = i∑
k=1
αi,k(skW V ), (4)
where WQ, WK and W V are projection parameters, and d denotes the dimension of inputs.

heading: 3 Method
text: In this section, we introduce the Decoder-only Streaming Transformer (DST). DST adopts the Decoder-only architecture and employs the proposed Streaming Self-Attention (SSA) in place of masked self-attention at each layer. During inference, SSA accumulates the attention assigned to
the input source tokens to assess the sufficiency of source information and obtain the translation policy accordingly. During training, DST leverages our proposed constraints to ensure the learning of the SSA mechanism. The details of DST are introduced as follows.

heading: 3.1 Model Architecture
text: We present the model architecture of DST in Figure 2. DST takes the concatenation of the source and target prefixes as input, encoding the positional information of both the source and target prefixes separately. In this way, the expansion of the source prefix does not affect the position of the generated target prefix, preventing the re-encoding of the generated target prefix. In each layer, DST replaces the masked self-attention module (Vaswani et al., 2017) in the conventional Decoder-only architecture with our proposed Streaming Self-Attention (SSA) module. During inference, the SSA module will determine the policy and integrate with the soft-attention mechanism to derive the context vector and predict the next token.

heading: 3.2 Streaming Self-Attention
text: In order to address the SiMT task, DST incorporates Streaming Self-Attention (SSA) in place of masked self-attention. By concatenating the source sentence and the target sentence as input, SSA adopts masked self-attention within the source sentence (Vaswani et al., 2017). For each target token, SSA initially allocates its attention probability to all possible source prefixes and subsequently combines the allocated attention with the soft-attention
mechanism to compute the expected attention probability for source tokens and preceding target tokens. The context vectors of target tokens are computed by utilizing the expected attention. We then provide a detailed explanation of the SSA mechanism as applied to target tokens during training.
Given the source sentence x and the target sentence y, the hidden state sequence input to SSA is denoted as h = (hs1, ..., h s J , h t 1, ..., h t I), where h s j denotes the source hidden state and hti represents the target hidden state. To articulate the allocated attention to the source prefixes, we define pi,j , which signifies the allocated attention of the target token yi to the source prefix x≤j . The calculation of pi,j is computed by utilizing the relationship between the source and target prefixes:
pi,j = sigmoid( h t iU Q(h s jU K)√ d ), (5)
where UQ and UK are learnable parameters, and d denotes the dimension of hidden state. h
t i and
h s j represents the mean pooling of hidden states corresponding to the target prefix y≤i and source prefix x≤j , respectively. Subsequently, we utilize pi,j to obtain the expected attention of each target token towards both source tokens and preceding target tokens.
Taking into account all possible source prefixes when translating the target token yi, the expected attention of target token yi to the source token xj is calculated as:
αsi,j =
J∑
m=j
pi,m exp(e s i,j)∑m
l=1 exp (e s i,l)
, (6)
where esi,j denotes the scaled dot-product of h t i and hsj , as illustrated in Eq.(2). The first summation considers all possible source prefixes that include xj , and the term inside summation reflects the attention probability associated with pi,m and softattention. Correspondingly, the expected attention of yi to the target token yk(k ∈ [1, i]) is:
αti,k = (1− J∑
j=1 pi,j) exp(eti,k)∑i l=1 exp(e t i,l) , (7)
where ∑J
j=1 pi,j represents the total attention to the source tokens and eti,k is the scaled dot-product of hti and h t k. Therefore, we can get the context vector of yi by considering both source tokens and
previous target tokens:
cti = J∑
j=1
αsi,j(h s jW
V ) + i∑
k=1
αti,k(h t kW V ), (8)
where W V denotes the projection parameters. The above is the operational mechanism of SSA during training. During inference, SSA utilizes the input source tokens and the previously generated target tokens to generate the context vector.

heading: 3.3 Inference
text: After introducing SSA, DST employs it to derive a policy, which instructs the model to generate translation. The SiMT policy generally assesses the adequacy of the source information to determine whether to proceed with generating the translation or to read additional source tokens (Zhang and Feng, 2023b). In DST, SSA initially allocates attention to different source prefixes using pi,j , subsequently integrating the allocated attention with the soft-attention. Consequently, ∑m j=1 pi,j signifies the accumulated attention to the current available source prefix x≤m and quantifies the sufficiency of source information. By setting an appropriate threshold δinfer, DST deems the source information sufficient under the condition:
m∑
j=1
pi,j > δinfer, (9)
where m is the number of input source tokens. At this point, DST is capable of generating the translation based on the existing source tokens. Otherwise, the model should continue reading the source tokens until the aforementioned condition is met or the entire sentence is input to our model. It is noteworthy that DST will generate the translation only when most layers meet the conditions.

heading: 3.4 Training Method
text: To facilitate the learning of SSA mechanism, we introduce the training method. The essence lies in the learning of pi,j . Although we may not directly provide an optimization objective for pi,j , we can leverage the characteristics of SiMT tasks (Zhang et al., 2022) and the properties of pi,j to induce its learning. Therefore, in addition to the crossentropy loss Lsimt, we propose three additional constraints and a curriculum learning strategy to aid in the training of DST.
Summation Constraint Based on the inherent property of pi,j , we introduce the summation constraint. As described in the section 3.2, pi,j represents the attention probability allocated to the source prefix x≤j and ∑J j=1 pi,j reflects the total attention on all source tokens. Therefore, it should be equivalent to the sum of attention to source tokens in masked self-attention (Vaswani et al., 2017):
Lsum = I∑
i=1
∥ J∑
j=1
pi,j − βi∥2, (10)
where βi denotes the attention to the source tokens in masked soft-attention. It is computed as:
βi =
∑J j=1 exp(e
s i,j)∑J
j=1 exp(e s i,j) + ∑i k=1 exp(e t i,k) , (11)
where esi,j is the scaled dot-product of target token yi and source token xj , and eti,k represents the scaled dot-product between the target token yi and its preceding target token yk.
Latency Constraint In addition to the summation constraint, we introduce a latency constraint for DST. Without the latency constraint, SSA tends to allocate excessive attention to longer source prefixes during training (Ma et al., 2020b). This tendency encourages the model to read more source tokens during inference, resulting in high latency. According to Zhang and Feng (2022b), the alignment favoring low latency tends to concentrate near the diagonal between source and target sentences. Expressing the attention allocation for source prefixes in sentence pair (x, y) as p = (pi,j)I×J , the latency constraint aims to encourage SSA to allocate more attention to source prefixes closer to the diagonal. To make it, we introduce the cost matrix C = (Ci,j)I×J , where Ci,j is computed as:
Ci,j = 1 max(I, J) max(|j− i× J I | − ϵ, 0), (12)
where |j − i × JI | quantifies the degree of deviation from the diagonal between the target token yi and the source prefix x≤j . Therefore, Ci,j supports attention distributions that are close to the diagonal. The hyperparameter ϵ controls the tolerance level for the deviation. We then obtain the latency constraint as:
Llat = I∑
i=1
J∑
j=1
pi,jCi,j . (13)
Further explanation about latency constraint is provided in the Appendix A.
Consistency Constraint During training, DST utilizes summation and latency constraints to ensure the learning of pi,j in each layer. However, there are variations among different layers (Yang et al., 2020), and our policy is obtained by integrating decisions from different layers. Without constraints on the decisions between different layers, the presence of outlier layers may result in excessively high latency for the learned policy. Therefore, we propose a consistency constraint to ensure consistency across decisions at different layers. Denoting the pi,j in the n-th layer as p (n) i,j , and the consistency constraint is calculated as follows:
Lcon = I∑
i=1
J∑
j=1
N∑
n=1
1
N ∥p(n)i,j − pi,j∥2, (14)
where N is the number of layers and pi,j is computed as follows:
pi,j = 1
N
N∑
n=1
p (n) i,j . (15)
Therefore, we can get the overall training objective of DST:
L = Lsimt + Lsum + Llat + Lcon. (16) Curriculum Learning Strategy If we train DST based on the training objective L directly, the model learns to generate translation using the entire source sentence. However, DST generates translations based on source prefixes during inference, leading to a discrepancy between training and inference (Zhang and Feng, 2022d). To mitigate this problem, we train DST to generate translations based on the source prefix by masking out subsequent source tokens during training. Specifically, by setting the threshold δtrain, DST masks out the source tokens after the shortest prefix x≤m that satisfies the condition ∑m j=1 pi,j > δtrain. Additionally, we introduce curriculum learning (Bengio et al., 2009), which gradually transitions the available source information from the entire sentence to the prefix consistent with the inference. During training, we implement this strategy by adjusting δtrain as:
δtrain = δinfer + (1− δinfer)× exp (− update
T ),
(17) where T is a hyperparameter and update represents the number of updates. During this process, DST gradually adapts to the scenario of generating translations based on prefixes (Guo et al., 2024).

heading: 4 Experiments

heading: 4.1 Datasets
text: We evaluate our method on three translation tasks.
IWSLT152 English→Vietnamese (En→Vi) (Cettolo et al., 2015) Consistent with Ma et al. (2020b), we replace the tokens occurring less than 5 with ⟨unk⟩. We use TED tst2012 and TED tst2013 as the development set and the test set, respectively.
WMT163 English→Romanian (En→Ro) We use newsdev-2016 as validation set and newstest2016 as test set. The source and target languages employ a shared vocabulary. Other settings are consistent with Gu et al. (2018).
WMT154 German→English (De→En) We use newstest2013 as validation set and newstest2015 as test set. Following Ma et al. (2020b), we apply BPE (Sennrich et al., 2016) with 32K subword units and use a shared vocabulary between source and target.

heading: 4.2 System Settings
text: Our experiments involve the following methods. Apart from our approach, other methods all employ Encoder-Decoder architecture.
Full-sentence represents the full-sentence translation of Encoder-Decoder architecture.
Wait-k (Ma et al., 2019) reads k tokens and then alternates between writing and reading a token.
MoE Wait-k (Zhang and Feng, 2021) treats each head as a Wait-k expert and integrates the outputs of all experts to generate translation.
MMA (Ma et al., 2020b) makes each head determine the policy by predicting a Bernoulli variable and generates translation by integrating the results of multiple heads.
PED (Guo et al., 2022) implements the adaptive policy by integrating post-evaluation into the translation policy.
HMT (Zhang and Feng, 2023a) utilizes the Hidden Markov Model to model the policy as hidden events and translation as observed events, and achieves the current state-of-the-art performance.
DST denotes our method described in Section 3. All systems are adapted from Fairseq Library (Ott et al., 2019). Regarding the compared methods, we apply Transformer-Small (6 layers, 4 heads) for En→Vi task and Transform-Base (6 layers, 8 heads) for En→Ro and De→En tasks. For our approach, we set the number N of layers in DST to 16
2https://nlp.stanford.edu/projects/nmt/ 3www.statmt.org/wmt16/ 4www.statmt.org/wmt15/
and utilize the top 8 layers to determine the translation policy. For the latency constraint, we empirically choose ϵ as 1. Other detailed system settings are shown in Appendix B. We obtain the SiMT models under different latency by adjusting δinfer. To accelerate the convergence and achieve better performance, our approach is fine-tuned from conventional Decoder-only architecture with masked self-attention mechanism (Vaswani et al., 2017). We use greedy search during inference and evaluate all methods with latency measured by Average Lagging (AL) (Ma et al., 2019) and quality estimated by BLEU (Papineni et al., 2002).

heading: 4.3 Main Results
text: The performance comparison of our approach with other SiMT methods is illustrated in Figure 3, demonstrating that our approach achieves state-ofthe-art performance on three tasks.
Compared to the Wait-k and MoE Wait-k methods, our method brings significant improvement. Both Wait-k and MoE Wait-k employ fixed policy and rely on heuristic rules to determine the source prefixes for translation (Ma et al., 2019; Zhang and Feng, 2021). This usually leads to the omission or redundancy of source information during trans-
lation (Zhang and Feng, 2022c), resulting in degraded performance. In contrast, our method adaptively determines the translation policy by leveraging the correspondence between the source and target prefixes. This adaptability allows our method to achieve superior tradeoffs. Additionally, our method outperforms PED, MMA, and HMT, which belong to the adaptive policy. Previous adaptive methods are based on the Encoder-Decoder architecture, determining the policy based on extracted features from the source and target prefixes (Ma et al., 2023; Zhang and Feng, 2023a). Our method relies on the Decoder-only architecture and utilizes the accumulated attention to existing source tokens to assess the sufficiency of translation, resulting in more effective policies. Besides, our method surpasses the performance of Full-sentence translation based on the Encoder-Decoder architecture with lower latency. This can be attributed to the advantages of the decoder-only architecture.

heading: 5 Analysis
text: To deepen the understanding of our method, we conduct multiple analyses. The experiments are all performed on the De→En translations task.

heading: 5.1 Ablation Study
text: In order to understand the settings, we conduct ablation studies on the training methods, curriculum learning strategy, and the number of layers.
Table 1 presents ablation experiments on the training method, where each setting contributes to the performance of DST. The latency constraint aids the model in acquiring the low-latency policy (Zhang and Feng, 2022b), and the curriculum learning strategy enables the model to adapt to generating translations based on prefixes (Guo et al., 2024). The consistency constraint effectively addresses the

heading: N δinfer AL BLEU
text: problems of outlier layers, which ensures consistency across decisions made at different layers (Ma et al., 2020b).
We further explore the impact of hyperparameters in the curriculum learning strategy in Table 2. A smaller T indicates a quicker transition of the training environment to the scenario of SiMT. We find that transitioning to the scenario of SiMT at an appropriate pace can achieve better tradeoffs between latency and translation quality.
Additionally, we investigate the relationship between the number of layers in DST and its performance in Table 3. This suggests that the performance of DST shows incremental improvement as the number of layers increases, highlighting a certain level of scalability.

heading: 5.2 Hallucination in Translation
text: To assess the quality of translations generated by different SiMT methods, we evaluate the hallucination in translations. In SiMT, when the model is trained to predict target tokens based on missing essential source information, it is prone to producing hallucinations during inference (Guo et al., 2023a). Therefore, the proportion of hallucinations contained in the generated translations also serves as a reflection of the quality of the learned policy during training.
To quantify the hallucinations, we introduce the hallucination rate (HR) (Chen et al., 2021a), which measures the percentage of tokens generated that are not related to the source sentence. We then provide its detailed definition. Given the source sentence x, we define the corresponding translation
as ŷ. Subsequently, we can get the alignment set h, which is a set of source-target index pairs (j, i) where jth source token xj aligns to the ith target token ŷi.
The token ŷi in ŷ is seen as hallucination (H(i,h)=1) if it can not be aligned to any source token:
H(i,h)=1[{(j, i) ∈ h} = ∅]. (18)
Therefore, the hallucination rate can be defined as:
HR(x, ŷ,h)= 1
|ŷ|
|ŷ|∑
i=1
H(i,h). (19)
As illustrated in Figure 4, our model demonstrates a lower likelihood of generating hallucinations at all latency, indicating increased reliability in generated translations. Compared to the Wait-k approach, our method allows for flexible utilization of essential source information by adjusting its policy and obtains a lower hallucination rate. Moreover, DST considers a more extensive range of translation policies than HMT during training (Zhang and Feng, 2023a). This provides possibilities for DST to acquire more effective policies, thereby achieving better tradeoffs between latency and translation quality.

heading: 5.3 Model Efficiency
text: In addition to evaluating the hallucinations in translation, we also investigate the model efficiency of different SiMT methods. As shown in Table 4, our method obtains superior performance with relatively high efficiency.
Compared to the SiMT models based on the Encoder-Decoder architecture, our model, utilizing a Decoder-only architecture, achieves superior performance with a relatively smaller number of parameters. Combining the results from Table 3, it reflects the parameter efficiency and scalability of the Decoder-only architecture (Fu et al., 2023).
Besides, we evaluate the training and inference efficiency of different SiMT models. All related experiments are conducted on NVIDIA GeForce RTX 3090. In terms of training speed, our approach is lower than Wait-k policy but higher than the adaptive policy HMT. Compared to Wait-k policy, our method involves more computations about attention, therefore requiring more training costs. Furthermore, HMT significantly increases the training complexity by expanding the target sentences several times during training. During inference, while our method is slightly slower than Wait-k, the inference speed of our method can still reach 78 tokens per second, which can fully meet the application requirements.
In addition to comparing the training and inference efficiency of SiMT methods in Table 4, we also consider model inference time and delay caused by waiting for source information concurrently. To evaluate the performance of the SiMT models in scenarios that closely resemble real-world conditions, we compute ComputationAware Average Lagging (CAAL) (Ma et al.,
2020a) for different methods utilizing MuST-C English→German dataset (Di Gangi et al., 2019). Unlike Ma et al. (2020a), our approach does not take direct speech input but rather uses the text corresponding to the speech. Therefore, we use whisper-large-v35 to align the text with the speech, considering both the length of the corresponding speech sequence and the actual machine inference time when calculating CAAL. Our settings use the Transformer-Small architecture and evaluate the performance of SiMT methods using CAAL and SacreBLEU (Post, 2018). The results are shown in Table 5. Although the inference speed of our method is slightly slower than the Wait-k policy in Table 4, it still achieves better performance in more realistic scenarios.
Therefore, our Decoder-only method can achieve state-of-the-art performance with higher parameter utilization, and relatively faster inference and training speed.

heading: 5.4 Analysis of Streaming Self-Attention
text: After evaluating the overall performance of our method, we delve into assessing the effectiveness of the proposed Streaming Self-Attention (SSA) mechanism. SSA initially allocates attention probability to all possible source prefixes and sub-
5https://huggingface.co/openai/ whisper-large-v3
sequently combines the allocated attention with the soft-attention to determine the expected attention for source tokens, referred to as ExpectedAllocation. To demonstrate the effectiveness of SSA, we also explore a Max-Allocation strategy, where the attention probability is allocated solely to the most probable source prefix.
As illustrated in Table 6, the ExpectedAllocation method yields better latency-quality tradeoffs. The Expected-Allocation method integrates the translation policy with translation capability. It aids the model in learning the correlation between source and target prefixes, resulting in enhanced translation policies and overall performance. This underscores the importance of attention probability allocation in SSA.

heading: 6 Related Work
text: Simultaneous machine translation (SiMT) begins generating translation before reading the entire source sentence. To ensure the quality of generated translations and meet the latency requirements, it is necessary to determine the suitable source prefixes for translation. The process of determining the source prefixes for target tokens is referred to as the policy (Zhang et al., 2024). Depending on whether the model decides the policy utilizing the correspondence between source and target prefixes, SiMT methods can be broadly categorized into fixed policy and adaptive policy.
For fixed policy, the SiMT model reads or generates tokens according to the heuristic rules. Ma et al. (2019) proposes Wait-k policy, which reads k source tokens first, and then writes and reads one token alternately. Elbayad et al. (2020) introduces the unidirectional encoder, where the preceding source tokens cannot attend to subsequent words. Zhang and Feng (2021) proposes MoE Wait-k, which assigns a Wait-k policy to each head and combines the results from multiple heads to generate translations. There are some methods that enhance the SiMT capability by modifying the reference (Chen et al., 2021b; Guo et al., 2023b). However, the fixed policy requires the SiMT model to generate tokens even when source information is insufficient, leading to a decline in performance of SiMT.
Unlike fixed policy, adaptive policy can dynamically decide whether to read or generate tokens based on the correspondence between source prefix and target prefix. Zheng et al. (2020) implements the adaptive policy through a composition
of several fixed policies. Miao et al. (2021) proposes a generative framework, which uses a reparameterized Poisson prior to regularise the policy. Zhang and Feng (2022b) models the contribution of each source token to the target tokens through optimal transport and determines the policy by accumulating the contributions of input source tokens. Zhang and Feng (2023a) utilizes the Hidden Markov Model to model the translation policy as the hidden events and the translation as the observed events. Ma et al. (2023, 2024) first proposes to conduct SiMT with non-autoregressive architecture and achieves good results. However, previous SiMT methods all employ the Encoder-Decoder architecture.
With the rise of language models, the Decoderonly architecture has demonstrated superior performance across various tasks (Touvron et al., 2023; Team et al., 2024; Jiang et al., 2024). The Decoderonly architecture has the advantages of good scalability and parameter efficiency. More importantly, its unidirectional encoding nature aligns greatly with streaming input. Therefore, we explore the application of the Decoder-only architecture in SiMT and propose the first Decoder-only SiMT model.

heading: 7 Conclusion
text: In this paper, we propose the first Decoder-only SiMT model, which leverages the characteristics of the Decoder-only architecture to implement the adaptive policy. Experiments show that our method achieves state-of-the-art performance and exhibits excellent scalability and model efficiency.
Limitations
Regarding the system settings, we investigate the impact of the number of layers and training methods on the performance of our DST method due to resource constraints. We think that further exploration of system settings could potentially yield even better results. We leave the exploration for future work.

heading: Acknowledgements
text: This paper is supported by National Natural Science Foundation of China (Grant No. 62376260). We thank all the anonymous reviewers for their valuable feedback and thorough reviews.

heading: A Explanation of Latency Constraint
text: In the Streaming Self-Attention (SSA) mechanism, the model obtains the attention probability allocated to the source prefixes by predicting pi,j . Subsequently, the model accumulates pi,j to determine the total attention on the input source tokens. The model utilizes ∑m j=1 pi,j to assess the sufficiency of the source information, thereby acquiring the translation policy. The model generates target tokens when it deems the source information sufficient. To ensure that the model generates translations tightly after reading the necessary source information, we introduce the latency constraint that encourages the model to utilize as few source tokens as possible to generate translations.
According to Zhang and Feng (2022b), the alignments conducive to latency tend to concentrate near the diagonal between source and target sentences. Therefore, we introduce Llat to encourage SSA to pay more attention to the source prefixes near the diagonal. The intuitive illustration of the cost matrix C = (Ci,j)I×J and the attention allocation matrix p = (pi,j)I×J is shown in Figure 5.

heading: B System Settings
text: The system settings on three translation tasks are shown in Table 7.
Our method is based on fine-tuning full-sentence translation model. During the full-sentence training stage, our model adopts the traditional decoder layer (Vaswani et al., 2017) and is trained with cross-entropy loss. Once we obtain a wellperforming translation model, we then add policyspecific parameters and train the model according to our proposed strategy in Section 3.4.
For more detailed implementation issues, please refer to our code.

heading: C Detailed Results
text: In addition to the performance comparison in Figure 3, we also present the numerical results for our method. Table 8, 9, and 10 respectively describe the performance of DST on IWSLT15 En→Vi, WMT16 En→Ro, and WMT15 De→En translation tasks. Each task is evaluated using latency measured by AL (Ma et al., 2019) and translation quality measured by BLEU (Papineni et al., 2002).
Table 9: Numerical results on WMT16 En→Ro.
δinfer AL BLEU
0.20 3.22 28.57 0.30 4.72 30.55 0.40 5.49 30.89 0.50 7.09 31.85 0.60 9.40 32.22
Table 10: Numerical results on WMT15 De→En.

heading: Future Work
text: We leave the exploration for future work