Self-Modifying State Modeling for Simultaneous Machine Translation (2024)

Abstract: Simultaneous Machine Translation (SiMT) generates target outputs while receiving stream source inputs and requires a read/write policy to decide whether to wait for the next source token or generate a new target token, whose decisions form a decision path. Existing SiMT methods, which learn the policy by exploring various decision paths in training, face inherent limitations. These methods not only fail to precisely optimize the policy due to the inability to accurately assess the individual impact of each decision on SiMT performance, but also cannot sufficiently explore all potential paths because of their vast number. Besides, building decision paths requires unidirectional encoders to simulate streaming source inputs, which impairs the translation quality of SiMT models. To solve these issues, we propose Self-Modifying State Modeling (SM), a novel training paradigm for SiMT task. Without building decision paths, SM individually optimizes decisions at each state during training. To precisely optimize the policy, SM introduces Self-Modifying process to independently assess and adjust decisions at each state. For sufficient exploration, SM proposes Prefix Sampling to efficiently traverse all potential states. Moreover, SM ensures compatibility with bidirectional encoders, thus achieving higher translation quality. Experiments show that SM outperforms strong baselines. Furthermore, SM allows offline machine translation models to acquire SiMT ability with fine-tuning 1.

heading: 1 Introduction
text: Simultaneous Machine Translation (SiMT) (Gu et al., 2017; Ma et al., 2019; Zhang et al., 2020) outputs translation while receiving the streaming source sentence. Different from normal Offline Machine Translation (OMT) (Vaswani et al., 2017),
* Corresponding Author 1Our source code is available at https://github.com/
EurekaForNLP/SM2
SiMT needs a suitable read/write policy to decide whether to wait for the coming source inputs (READ) or generate target tokens (WRITE).
As shown in Figure 1(a), to learn a suitable policy, existing SiMT methods usually require building a decision path (i.e., a series of READ and WRITE decisions made by the policy) to simulate the complete SiMT process during training (Zhang and Feng, 2022c). Methods of fixed policies (Ma et al., 2019; Zhang and Feng, 2021) build the decision path based on pre-defined rules, and only optimize translation quality along the path. Methods of adaptive policies (Zheng et al., 2019; Miao et al., 2021; Zhang and Feng, 2023) dynamically build the decision path and optimize the policy based on the SiMT performance along this path.
However, the current training paradigm based on decision paths faces inherent limitations. First, it can lead to imprecise optimization of the pol-
9781
icy during training. For fixed policies, pre-defined rules cannot ensure optimal decisions at each state. For adaptive policies, there exists a credit assignment problem (Minsky, 1961), which means it is difficult to identify the impact of each individual decision on the global SiMT performance along a path, thus hindering the precise optimization of each decision. Second, due to numerous potential decision paths, existing methods (Zheng et al., 2019; Miao et al., 2021; Zhang and Feng, 2023) often prohibit the exploration of some paths during training, but this insufficient exploration cannot ensure the optimal policy. Third, for building decision paths in training, existing methods require unidirectional encoders to simulate streaming source inputs and avoid the leakage of source future information (Elbayad et al., 2020), which impairs SiMT models’ translation quality (Iranzo-Sánchez et al., 2022; Kim and Cho, 2023).
To address these issues, we propose SelfModifying State Modeling (SM2), a novel training paradigm for SiMT task. As shown in Figure 1(b), instead of constructing complete decision paths, SM2 individually optimizes decisions at all potential states during training. This paradigm necessitates addressing two critical issues: firstly, how to independently optimize each decision based on its own contribution to SiMT performance; and secondly, how to sufficiently explore all potential states during training. To realize the independent optimization, SM2 assesses each decision by estimating confidence values which measure the translation credibility. High confidence means the SiMT model can predict a credible target token at current state and WRITE is beneficial for SiMT performance; otherwise, READ is preferred. Since golden confidence values are unavailable, SM2 introduces Self-Modifying process to learn accurate confidence estimation (DeVries and Taylor, 2018; Lu et al., 2022). Specifically, during training, the SiMT model is allowed to modify its prediction based on the received source prefix with the prediction based on the complete source sentence, and the confidence is estimated to determine whether the modification is necessary to ensure a credible prediction at current state. To sufficiently explore all potential states, SM2 conducts Prefix Sampling to divide all states into groups according to the number of their received source prefix tokens, and sample one group for optimization in each iteration.
Compared to the training paradigm based on decision paths, SM2 presents significant advantages.
First, the Self-Modifying process can assess each decision independently, which realizes the precise optimization of policy without the credit assignment problem. Second, Prefix Sampling ensures sufficient exploration of all potential states, promoting the discovery of the optimal policy. These benefits enable SM2 to learn a more effective policy. Furthermore, without building decision paths in training, SM2 ensures compatibility with bidirectional encoders, thereby improving translation quality. This compatibility also allows OMT models to acquire the SiMT capability via fine-tuning. Our contributions are outlined in the following:
• We propose Self-Modifying State Modeling (SM2), a novel training paradigm that individually optimizes decisions at all states without building complete decision paths.
• SM2 can learn a better policy through precise optimization of each decision and sufficient exploration of all states. With bidirectional encoders, SM2 achieves higher translation quality and compatibility with OMT models.
• Experimental results on Zh→En, De→En and En→Ro SiMT tasks show that SM2 outperforms strong baselines under all latency levels.

heading: 2 Background
text: Simultaneous machine translation For SiMT task, we respectively denote the source sentence as x = (x1, ..., xM ) and the corresponding target sentence as y = (y1, .., yN ). Since the source inputs are streaming, we denote the number of source tokens available when generating yi as gi, and hence the prediction probability of yi is p(yi | x≤gi ,y<i) (Ma et al., 2019). Thus, the decoding probability of y is given by:
p (y | x) = N∏
i=1
p(yi | x≤gi ,y<i) (1)
Decision state and decision path We define the state sij as the condition in which the source prefix x≤j has been received and the target prefix y<i has been generated. At sij , a decision dij ∈ {WRITE,READ} can be made based on the context (x≤j ,y<i) (Grissom II et al., 2014; Gu et al., 2017; Zhao et al., 2023). Specifically, if x≤j is sufficient for the SiMT model to predict yi accurately, dij should be WRITE; otherwise, dij should be READ. As shown in Figure 1(a), a series
of decisions [d00, ..., dNM ] are made in the SiMT process, which forms a decision path from s00 to sNM . Along the decision path, the SiMT model can finish reading the whole x and outputting the complete y (Zhang and Feng, 2022c). Since these concepts are usually used in SiMT methods based on reinforcement learning (RL) (Grissom II et al., 2014; Gu et al., 2017), we compare our method with RL-based methods in Appendix A for clarity.

heading: 3 The Proposed Method
text: We propose Self-Modifying State Modeling (SM2), which individually optimizes decisions at all states. The overview of SM2 is shown in Figure 2. To independently optimize each decision, SM2 learns confidence estimation to assess decisions at each state by modeling the Self-Modifying process (Sec.3.1). To ensure sufficient exploration during training, SM2 conducts Prefix Sampling to traverse all potential states (Sec. 3.2). Then, based on estimated confidence at each state, SM2 can determine whether the received source tokens are sufficient to generate a credible token and make suitable decisions during inference (Sec.3.3).

heading: 3.1 Self-Modifying for Confidence Estimation
text: Intuitively, when a translation model has access to the complete input x (i.e., OMT setting), it can produce credible outputs. Therefore, a prediction made by the translation model at sij (i.e. SiMT setting) is considered credible if it aligns with that in OMT setting. Conversely, if the prediction in
SiMT setting is incredible, it will be modified in OMT setting. Based on this insight and Ask For Hints (DeVries and Taylor, 2018; Lu et al., 2022), we model the Self-Modifying process to assess the translation credibility of each state. Specifically, we provide the SiMT model an option to modify its prediction in SiMT setting with that in OMT setting, and confidence estimation is defined as a binary classification determining whether the current generation requires the modification to ensure a credible prediction. Through measuring translation credibility, decisions at each state can be independently assessed. High confidence means the SiMT model can generate a credible token at sij without modification and the WRITE decision is beneficial for SiMT performance; whereas low confidence indicates the prediction is inaccurate at sij and the READ decision is preferred.
During training, the Self-Modifying process is conducted in two steps: prediction in SiMT setting & OMT setting and confidence-based modification.
For prediction in SiMT setting& OMT setting, the SiMT model outputs different predictions at sij in SiMT setting and OMT setting respectively. These predictions are calculated as follows:
pij = p(yi | x≤j ,y<i) pi = p(yi | x,y<i)
(2)
It is noted that the model parameters in SiMT setting and OMT setting are shared.
For confidence-based modification, an additional confidence net is used to predict the confidence cij
Algorithm 1: Confidence-based Policy Input :Streaming inputs x≤j , Threshold
γ, i = 1, j = 1, y0 ← ⟨BOS⟩ Output :Target outputs y
1 while yi−1 ̸= ⟨EOS⟩ do 2 calculate confidence cij as Eq.(3); 3 if cij ≥ γ then // WRITE 4 generate yi with x≤j ,y<i; 5 i← i+ 1; 6 else // READ 7 wait for next source token xj+1; 8 j ← j + 1; 9 end
10 end
at sij . The confidence net is represented as:
cij = sigmoid(W T · hij + b) (3)
where hij is the hidden representation from the top decoder layer in SiMT setting and θ = {W, b} are trainable parameters. If pij is credible, cij should be close to 1; otherwise, cij should be close to 0. To accurately calibrate cij in the training process, we integrate the modification into the prediction probability as follows:
pmij = cij · pij + (1− cij) · pi (4)
Subsequently, the translation loss is calculated using the modified probability:
Lsij = −yi log(pmij ) (5)
Notably, the SiMT model can enhance the prediction credibility by estimating a lower cij for more modification. However, this manner may cause an over-reliance on pi. To avoid that, an additional penalty term for cij is introduced:
Lcij = − log(cij) (6)
Through Self-Modifying process, SM2 independently optimizes each decision based on their individual effect on the SiMT performance, thus realizing the precise optimization of the policy without credit assignment problem. We provide a gradient analysis of the independent optimization in Appendix B for further explanation.

heading: 3.2 Prefix Sampling
text: To sufficiently explore all potential states during training, Prefix Sampling is conducted in SM2. As shown in Figure 2, states are categorized into groups, and one group is randomly sampled for optimization in each iteration. Specifically, all possible states of (x,y) are divided into M groups according to the number of their received source prefix tokens, and each group comprises N states, which can be formulated as follows:
Sj = {sij | 1 ≤ i ≤ N}, j ∈ [1,M ] (7)
In each iteration, we sample j ∼ U(1,M). Then, SM2 respectively predicts target translation in SiMT setting based on Sj and those in OMT setting based on SM , where the complete source sentence is received. Thus, the modified translation loss and the penalty item of each iteration are computed as follows:
LSj = N∑
i=1
Lsij
LCj = N∑
i=1
Lcij
(8)
Besides, to ensure the pi in OMT setting can provide effective modification, the translation loss in OMT setting is required, which is formulated as:
Lomt = − N∑
i=1
log(pi) (9)
The total training loss is the following:
L = Lomt + LSj + λLCj (10)
where λ is the super parameter. We discuss the effect of λ in Appendix C.
Through Prefix Sampling, SM2 explores all potential states without building any decision paths. Therefore, SM2 can employ bidirectional encoders without the leakage of source future information in the training process.

heading: 3.3 Confidence-based Policy in Inference
text: During inference, SM2 utilizes cij to assess the credibility of current prediction, thus making suitable decisions between READ and WRITE at sij . Specifically, a confidence threshold γ is introduced to serve as a criterion for making decisions. As
shown in Algorithm 1, if cij > γ, SM2 selects WRITE; otherwise, SM2 selects READ. This decision process is constantly repeated until the complete translation is finished. It is noted that we only utilize SiMT setting in the inference process.
By adjusting γ, SM2 can perform the SiMT task under different latency levels. A higher γ encourages the SiMT model to predict more credible target tokens and the latency will be longer. Conversely, a lower γ reduces the latency but may lead to a decrease in translation quality. The values of γ employed in our subsequent experiments are detailed in Appendix D.

heading: 4 Experiments

heading: 4.1 Datasets
text: We conduct experiments on three datasets: Zh→En We use LDC corpus which contains 2.1M sentence pairs as the training set, NIST 2008 for the validation set and NIST 2003, 2004, 2005, and 2006 for the test sets.
De→En We choose WMT15 for training, which contains 4.5M sentence pairs. Newstest 2013 are used as the validation set and newstest 2015 are used as the test set.
En→Ro WMT16 (0.6M) is used as the training set. We choose newsdev 2016 as the validation set and newstest 2016 as the test test.
We apply BPE (Sennrich et al., 2016) for all language pairs. In Zh→En, the vocabulary size is 30k for Chinese and 20k for English. In both De→En and En→Ro, a shared vocabulary is learned with 32k merge operations. Additional experiments on WMT15 En→Vi are provided in Appendix E.

heading: 4.2 System Settings
text: The models used in our experiments are introduced as follows. All baselines are built based on Transformer (Vaswani et al., 2017) with the unidirectional encoder unless otherwise stated. More details are presented in Appendix D.
OMT-Uni/OMT-Bi(Vaswani et al., 2017): OMT model with an unidirectional/bidirectional encoder.
wait-k (Ma et al., 2019): a fixed policy, which first reads k tokens, then writes one token and reads one token in turns.
m-wait-k (Elbayad et al., 2020): a fix policy, which improves wait-k by randomly sampling different k during training.
ITST (Zhang and Feng, 2022b): an adaptive policy, which models the SiMT task as a transport
problem of information from source to target. HMT (Zhang and Feng, 2023): an adaptive policy, which models the SiMT task as a hidden Markov model, by treating the states as hidden events and the predicted tokens as observed events.
SM2-Uni/SM2-Bi: Our proposed method with an unidirectional/bidirectional encoder.

heading: 4.3 Evaluation Metric
text: For SiMT, both translation quality and latency require evaluation. Since existing datasets mainly focus on the OMT task, the metric based on n-gram may cause inaccurate evaluation (Rei et al., 2020). Therefore, we measure the translation quality with both SacreBLEU (Post, 2018) and COMET2 scores. For latency evaluation, we choose Average Lagging (AL) (Ma et al., 2019) as the metric.
Furthermore, to assess the quality of read/write policy in different SiMT models, we follow Zhang and Feng (2022b) and Kim and Cho (2023) to use Satisfied Alignments (SA), the proportion of the ground-truth aligned source tokens received before translating. Specifically, when generating yi, the number of received source tokens gi should be no less than the golden-truth aligned source position ai, so that the alignment between yi and xai can be satisfied in the SiMT process. Thus, SA(↑) can be calculated as:
SA = 1
N
N∑
i=1
I(ai ≤ gi) (11)

heading: 5 Results and Analysis

heading: 5.1 Simultaneous Translation Quality
text: We present the translation quality under various latency levels of different SiMT models in Figure 3 and Figure 4. These results indicate that SM2 outperforms previous methods across three language pairs in terms of both SacreBLEU and COMET scores. With the unidirectional encoder, SM-Uni achieves higher translation quality compared to current state-of-the-art SiMT models (ITST, HMT) at low and medium latency levels (AL∈ [0, 6]), and maintains comparable performance at high latency level (AL∈ [6, 12]). We attribute this improvement to the effectiveness of learning a better policy during training. Furthermore, with the superior capabilities of the bidirectional encoder, SM2-Bi outperforms previous SiMT models more significantly
2Unbabel/wmt22-cometkiwi-da
across all latency levels. All SiMT models with unidirectional encoders can approach the translation quality of OMT-Uni at high latency levels, but only SM2-Bi achieves similar performance to OMT-Bi as the latency increases. These experimental results prove that SM2 achieves better performance than other SiMT methods for learning better policy and improving translation quality. Detailed numerical results are provided in Appendix E, supplemented with additional evidence demonstrating the robustness of SM2 to sentence length variations.
5.2 Superiority of SM2 in Learning Policy
To verify whether SM2 can learn a more effective policy, we compare SA(↑) under various latency levels of different SiMT models. Following Zhang and Feng (2022b) and Kim and Cho (2023), we conduct the analysis on RWTH3, a De→En alignment dataset. The results are presented in Figure 5. Compared with existing methods, both SM2-Uni and SM2-Bi receive more aligned source tokens before generating target tokens under the same latency. Especially at medium latency level (AL∈ [4, 6]), SM2
3https://www-i6.informatik.rwth-aachen.de/ goldAlignment/
can receive about 8% more source tokens than fixed policies (wait-k, m-wait-k) and 3.6% more than adaptive policies (ITST, HMT). We attribute these improvements to the advantages of SM2 in learning policy. Through precise optimization, SM2 can make more suitable decisions at each state, which generates faithful translations once receiving sufficient source tokens and waits for more source inputs when the predicted tokens are incredible. With sufficient exploration, SM2 can investigate all possible situations and reduce unnecessary latency in the SiMT process.

heading: 5.3 Precise Optimization for Each Decision
text: To validate whether the confidence-based policy is precisely optimized at each state, we examine the relationship between estimated confidence cij and the probability of the correct token yi in the prediction, denoted as pcij . Specifically, we employ SM 2 to decode the validation set in a teacher-forcing manner, calculating the cij and pcij for all possible states. Subsequently, a correlation analysis is performed between cij and pcij . The results in Table 1 demonstrate a strong correlation, evidenced by high values in Pearson (0.82) and Spearman (0.84) coefficients, with a slightly moderate but signifi-
cant Kendall’s τ coefficient (0.65). These results suggest a robust linear and monotonic relationship between cij and pcij , indicating the capacity of cij to accurately assess the credibility of the current predicted token. Consequently, this confirms the effectiveness of the confidence-based policy in making precise decisions at each state.

heading: 5.4 Advantage of Sufficient Exploration
text: Existing methods often prohibit the exploration of some paths due to the possible decision paths being numerous (Zheng et al., 2019; Miao et al., 2021; Zhang and Feng, 2023). To investigate the impact of the prohibition on SiMT models and the superiority of SM2 in sufficiently exploring all states, we attempt to train these methods without prohibition, but they fail to converge. Therefore, we analyze the impact by employing the same prohibition in HMT (Zhang and Feng, 2023) and RIL(Zheng et al., 2019) to train SM2, which restricts SM2 to explore states only between wait-k1 and wait-k2 paths in training. As shown in Figure 6(a), we set k1 = 1 and k2 = 10 in our experiments. The performances of SM2 with prohibition (SM2-Uni-P and SM2-Bi-P) are shown in Figure 6(c), indicating a decline in performance. These results suggest that the prohibition causes insufficient exploration, leading to diminished performance. In contrast, SM2 ensures comprehensive exploration, which is shown in Figure 6(b), thereby achieving higher performance. Further analysis of the policy quality is provided in Appendix F.

heading: 5.5 Compatibility with OMT Models
text: SM2 allows for the parallel training of the bidirectional encoder. Due to this compatibility, SM2-Bi achieves superior translation quality than existing SiMT methods with unidirectional encoders (Figure 3,4). To further present the superiority of this compatibility, we propose fine-tuning OMT models according to SM2, so that the translation ability in OMT models can be easily utilized to gain SiMT models. Specifically, two distinct methods are used: fine-tuning all model parameters (SM2FT) and fine-tuning with adapters (SM2-adapter)4. As shown in Figure 7(a), SM2-adapter can achieve comparable performance with current state-of-theart SiMT models, and SM2-FT closely matches the performance of SM2-Bi.
Additionally, we further explore the effect of the OMT models’ translation abilities on the corresponding SiMT abilities after fine-tuning. We conduct the full-parameter fine-tuning on OMT models with Transformer-small, Transformer-base, and Transformer-big respectively. The OMT and SiMT capabilities of these models are illustrated in Table 2 and Figure 7(b), which reveal that models
4We add adapters after the feed-forward networks of each encoder and decoder layer. For each adapter, the input dimension and output dimension are 512, and the hidden layer dimension is 128.
with stronger OMT abilities achieve better SiMT performance after fine-tuning. Besides, the results in Table 2 show that these models’ original OMT abilities are not hurt, indicating that SM2 enables models to support both OMT and SiMT abilities.

heading: 5.6 Ablation Study
text: We conduct ablation studies on SM2 to analyze the effect of Lomt and modification from OMT setting.
Effect of Lomt As shown in Figure 8, the SiMT model withoutLomt drops quickly. We argue this is because training without Lomt may cause a worse modification. The results in Table 3 show that the OMT performance of SM2 trained without Lomt is significantly affected, even worse than its SiMT performance in the high latency levels. This poor OMT ability cannot provide accurate modification, thus disrupting the policy learning process.
Effect of OMT modification Following Ask For Hints (DeVries and Taylor, 2018; Lu et al., 2022), we use the one-hot label as the "hints" to modify the prediction in SiMT setting. Specifically, we denote ti as the ground-truth label of the i-th target token, and hence the modification in SM2 is adjusted as:
pmij = cij · pij + (1− cij) · ti (12)
As shown in Figure 8, the performance of SM2 trained with modification in Eq.(12) also drops.
We argue that the modification from ti cannot reflect the real available gain from the modification after receiving the complete source sentence, thus learning a worse policy.

heading: 6 Related Work
text: Simultaneous Machine Translation Different from offline machine translation (Vaswani et al., 2017; Zhao et al., 2020; Wu et al., 2024), existing SiMT methods are divided into fixed policy and adaptive policy. For fixed policy, Ma et al. (2019) proposed wait-k, which starts translation after receiving k tokens. Elbayad et al. (2020) proposed multipath wait-k, which randomly samples k during training. For adaptive policy, heuristic rules Cho and Esipova (2016) and reinforcement learning Gu et al. (2017) are used to realize the SiMT task. Ma et al. (2020b) integrated multi-head monotonic attention to model the decision process, where each head independently makes decisions. Similarly, Zhang and Feng (2022a) utilized Gaussian multi-head attention to model the alignment, thus improving the decision-making ability of each head. Miao et al. (2021) proposed a generative framework to learn a read/write policy. Zhang and Feng (2022b) measured the information SiMT had received and proposed an information-based policy. Zhang and Feng (2023) used the Hidden Markov model in SiMT task to learn an adaptive policy.
Previous methods based on decision paths are limited in policy learning and model structure. Our proposed SM2 individually explores all states during training, overcoming these limitations.
Confidence Estimation for OMT Confidence estimation is used to measure the models’ credibility. Wang et al. (2019) used Monte Carlo dropout to propose an uncertainty-based confidence estimation. Wan et al. (2020) utilized the confidence score to guide self-paced learning. DeVries and Taylor (2018) evaluated the confidence by measuring the level it asks for hints from the ground-truth label, and Lu et al. (2022) transferred it to OMT to improve the out-of-distribution detection.

heading: 7 Conclusion
text: In this paper, we propose Self-Modifying State Modeling (SM2), a novel training paradigm for SiMT. SM2 eschews the construction of complete decision paths during training, opting to explore all potential states individually instead. By introducing the Self-Modifying process, SM2 independently assesses each state to precisely optimize the read/write policy without the credit assignment problem. Through Prefix Sampling, SM2 ensures sufficient exploration of all potential states. Experimental results across three language pairs validate the superior performance of SM2, and our analyses further confirm that SM2 can learn a more effective read/write policy. More promisingly, SM2 demonstrates the potential to endow OMT models with SiMT capability through fine-tuning.
Limitations
In this paper, we propose SM2, a novel paradigm that individually optimizes decisions at each state. Although our experiments show the superiority of not building decision paths during training, there are still some parts to be further improved. For example, using a more effective way to independently assess the individual effect of each decision on the SiMT performance. Besides, how to leverage other pre-trained encoder-decoder models like BART and T5, to gain SiMT models, is still a promising direction to explore. These will be considered as objectives for our future work.

heading: Acknowledgements
text: This work has been supported by the National Natural Science Foundation of China (NSFC) under Grant No. 62206295 and 62206286.

heading: B Gradient Analysis
text: In this section, we provide a gradient analysis of the independent optimization in SM2. The training loss function L of SM2 is formulated in Eq. (10). During training, this loss function adjusts each decision dij at state sij by changing the value of corresponding confidence cij . Specifically, the gradient of L with respect to cij is calculated as:
∂L ∂cij = ∂Lsij ∂cij + λ ∂Lcij ∂cij
= − yi pmij · ∂pmij ∂cij − λ cij = − yi(pij − pi) cij · pij + (1− cij) · pi − λ cij (15)
It is evident that this gradient does not contain any ci′j′ (i′ ̸= i or j′ ̸= j). Therefore, in the training process, the estimated value of cij is adjusted only based on its current value and the prediction probability of the current state, without being affected by the decisions at other states, thus allowing for the independent optimization of cij .
In contrast, existing SiMT methods usually conduct training on decision paths and can not ensure independent optimization. Taking ITST (Zhang
and Feng, 2022b) as an example, whose loss function L′ is formulated as:
L′ = Lce + Llatency + Lnorm
Llatency = I∑
i=1
J∑
j=1
Tij × Cij
Lnorm = I∑
i=1
∥∥∥∥∥∥ J∑
j=1
Tij − 1 ∥∥∥∥∥∥ 2
(16)
where Lce is the cross-entropy for learning translation ability, and Cij is the latency cost for each state. During training, the decision is dominated by Tij . The gradient of L′ with respect to Tij is calculated as:
∂L′ ∂Tij = ∂Lce ∂Tij + Cij + 2( J∑
j=1
Tij − 1) (17)
It is noted that the gradient of Tij is also affected by the current values of Tij′(j′ = 1, 2, ..., J). These decisions are coupled in the optimization, thus not enabling the independent optimization of each decision. This can trigger mutual interference during training (Zhang and Feng, 2023) and lead to a credit assignment problem.

heading: C Effect of λ
text: We analyze the effect of λ, which is the weight of the penalty during training. We train SM2 with
different λ ranging from 0.1 to 1, in increments of 0.1. As shown in Figure 9(a), the SM2 models trained with different λ show comparable performance across all latency. This indicates that SM2 is robust to variations in hyper-parameters λ. When λ becomes larger, the corresponding γ at the same latency will also increase. Therefore, we further analyze the effect of λ on the applicable latency range of SM2. We denote the "MAX AL" as the latency of SM2 when γ is set as 0.99 during inference. The results are shown in Figure 9(b). When λ becomes larger, "MAX AL" also decreases, which means a smaller applicable latency range. For example, when λ = 1.0 in training, it is hard for SM2 to perform SiMT task under the latency levels where AL is larger than 5.79 since the threshold γ has been close to 1.

heading: D Hyper-parameters
text: The system settings in our experiments are shown in Table 4. We set λ = 0.1 during training. Besides, we follow Ma et al. (2020b) to use greedy search during inference for all baselines. The values of γ we used are 0.3,0.4,0.5,0.55,0.6,0.65 for Zh→En, 0.3,0.4,0.5,0.55,0.6,0.65,0.7 for De→En, and 0.3,0.4,0.5,0.6,0.65,0.7,0.75 for En→Ro.

heading: E Main Results Supplement
text: E.1 Numerical Results Table 5, 6, 7 respectively report the numerical results on LDC Zh→En, WMT15 De→En, WMT16 En→Ro measured by AL, SacreBLEU and COMET. Figure 10, 11 and Table 8 report the results on WMT15 En→Vi with Transformersmall, which also present the superior performance of SM2-Uni and SM2-Bi.
E.2 Robustness of SM2 to Sentence Length To validate the robustness of SM2 to Sentence Length, we conduct additional experiments on De→En SiMT tasks. Specifically, we divide the test set into two groups based on sentence length: LONG group and SHORT group. The average lengths and the number of sentences in each group are shown in Table 9. Then, we test SM2-Bi and SM2-Uni separately on these two groups. The translation quality under different latency levels for SM2-Bi and SM2-Uni are presented in Figure 12. For clearer comparison, we also provide the performances of OMT models (OMT-Bi, OMT-Uni) on LONG and SHORT groups.
The results in Figure 12 indicate that when applied to longer sentences, the performance changes of SM2 are similar to OMT models in both unidirectional and bidirectional encoder settings. Since the performance of OMT models unavoidably drops as the sentences become longer (Neishi and Yoshinaga, 2019; Kang et al., 2020; Ma et al., 2020a; Variš and Bojar, 2021; Zhang et al., 2023), it is not SM2 that triggers the decrease of translation quality. Therefore, SM2 is still effective on long sentences.

heading: F Effect of Prohibition on Policy
text: To further validate that the prohibition of exploration negatively affects the policy. We compare the SA of SM2 with and without the prohibition on RWTH dataset. The results in Figure 13 indicate that the prohibition makes SM2 learn a worse policy. Therefore, we can conclude that the prohibition will hurt the quality of policy. This further presents the advantage of SM2 in sufficiently exploring all states through Prefix Sampling.

heading: Future Work
text: These will be considered as objectives for our future work