Fixed and Adaptive Simultaneous Machine Translation Strategies Using Adapters (2024)

Abstract: Simultaneous machine translation aims at solving the task of real-time translation by starting to translate before consuming the full input, which poses challenges in terms of balancing quality and latency of the translation. The waitk policy offers a solution by starting to translate after consuming k words, where the choice of the number k directly affects the latency and quality. In applications where we seek to keep the choice over latency and quality at inference, the wait-k policy obliges us to train more than one model. In this paper, we address the challenge of building one model that can fulfil multiple latency levels and we achieve this by introducing lightweight adapter modules into the decoder. The adapters are trained to be specialized for different wait-k values and compared to other techniques they offer more flexibility to allow for reaping the benefits of parameter sharing and minimizing interference. Additionally, we show that by combining with an adaptive strategy, we can further improve the results. Experiments on two language directions show that our method outperforms or competes with other strong baselines on most latency values. 1

heading: 1 Introduction
text: Simultaneous machine translation (SiMT) aims at reducing the latency of translation systems. In scenarios with low latency demands, such as conferences or lectures, translating with minimum delay is crucial. In order to reduce the latency, SiMT models start translating before consuming the full input sentence, which improves the latency but affects the quality of the translation, because of limited access to enough source context to make a correct prediction. SiMT techniques design a strategy to decide when to make a READ (i.e. wait for more source tokens) or WRITE (i.e. output
1Code is available at: https://github.com/issam9/ Adapters-SiMT
a new token) action. The strategy has to balance the trade-off between quality and latency by making more READ or WRITE actions. Making more READ actions will lead to improved quality but will hinder the latency, while the opposite is true for making more WRITE actions. Fixed policies design a strategy that is detached from whether there is sufficient context to make a WRITE action (Ma et al., 2019; Elbayad et al., 2020; Zhang and Feng, 2021). For instance, the wait-k policy (Ma et al., 2019) trains the model to make k number of READ actions before every WRITE action. The value of k has a direct impact on the quality and latency of the translation and since it is decided during training, wait-k models have to be trained with latency in mind, which means that in order to support multiple latency levels, we need to train multiple models. The multi-path training (Elbayad et al., 2020) was introduced to solve this issue by sampling the value of k randomly during training, which results in a model that supports multiple latency levels. This technique was shown to benefit the inference at lower wait-k values by improving the results, but it neglects that parameter sharing between all the wait-k values might introduce interference. Zhang and Feng (2021) addressed the interference issue by using Mixture-of-Experts (MoE), where each head of the multi-head attention is treated as an expert and is trained on different wait-k values. This has proven to be a successful technique, but the number of wait-k experts we can introduce depends on the number of heads in the Transformer model, which limits the flexibility in terms of balancing parameter sharing and interference between the wait-k paths. Our method relies on inserting lightweight adapters (Rebuffi et al., 2017; Houlsby et al., 2019) for this purpose. The number of the adapters and their capacity can be easily adjusted depending on the wait-k values we intend to support and the complexity of the language direction.
Dynamic strategies have gained increased atten-
346
tion in recent years (Gu et al., 2017; Zheng et al., 2019, 2020; Ma et al., 2020; Zhang and Feng, 2022; Zhao et al., 2023) due to their effectiveness. Dynamic strategies strive to strike a balance between latency and quality by making as much READ actions as necessary and as much WRITE actions as possible. The decision to read or write is made dynamically based on the context (which can be the received input and the previous target tokens) at each decoding step. Although dynamic strategies achieve state-of-the-art results, they often require specialized training techniques (Gu et al., 2017; Ma et al., 2020; Zhang and Feng, 2022) that can balance between latency and quality when generating READ/WRITE actions, or even require the training of multiple models (Zheng et al., 2020; Ma et al., 2020) to support multiple latency levels. In order to take advantage of the dynamic wait-k strategies, we adopt a strategy that composes multiple wait-k models during inference (we refer to this as Adaptive Wait-k (Zheng et al., 2020)) to work with wait-k adapters instead. This brings efficiency and cost benefits as only one model is required to satisfy multiple latency levels and also improves performance compared to other strong baselines including Adaptive Wait-k.
In summary, our main contributions are the following:
• We introduce lightweight adapters as a flexible solution to balance parameter sharing and interference in multi-path training.
• We show that by combining adapters with a simple adaptive strategy (i.e. Adaptive Waitk) we can further improve the results.
• We show that our technique outperforms or competes with other strong baselines on most latency levels.

heading: 2 Related Works

heading: 2.1 Adapters for Machine Translation
text: Adapters (Rebuffi et al., 2017; Houlsby et al., 2019) are typically small modules that are used in order to efficiently adapt a pre-trained model to a downstream task, where the pre-trained model can be either frozen (Houlsby et al., 2019), or trained jointly with the adapters (Stickland and Murray, 2019).
Adapters have been used for efficient multi-task fine-tuning (Stickland and Murray, 2019), where each set of adapters is trained on a specific task.
Pfeiffer et al. (2021) added AdapterFusion on top of the adapters as a way to compose the representations of different tasks. Pfeiffer et al. (2022) used adapters as language-specific parameters in order to address the curse of multilinguality in multilingual pre-training, where the adapter modules are introduced during pre-training instead of post-hoc.
For Neural Machine Translation (NMT), Bapna and Firat (2019) introduced a simple formulation of adapters to learn language-pair specific parameters, where they showed that it improves performance on high resource languages in Multilingual Translation. Chronopoulou et al. (2023) trained languagefamily adapters to address negative interference while allowing for parameter sharing between similar languages, which improved performance on low resource languages. Zhao and Calapodescu (2022) fine-tuned adapters on multimodal noise, then added a fusion layer in order to improve generalization to other types of noise. Adapters were also explored for other motivations like Zero-shot NMT and unsupervised domain adaptation (Philip et al., 2020; Malik et al., 2023).

heading: 2.2 Simultaneous Machine Translation
text: SiMT systems can be divided into fixed and adaptive policies. Fixed policies rely on predefined rules for READ/WRITE decisions. Ma et al. (2019) proposed the wait-k policy, where the model starts by reading k tokens then alternates between reading and writing one token. Elbayad et al. (2020) introduced multi-path training, where one model is trained to support multiple wait-k values by sampling k randomly during training. Zhang and Feng (2021) addressed interference in multi-path training by using Mixture-of-Experts. Zhang et al. (2021) used Knowledge Distillation from a Full-Sentence Transformer to embed future information into the SiMT model. For adaptive policies, Gu et al. (2017) trained a Reinforcement Learning agent to decide READ/WRITE actions, where the reward function is designed to consider both quality and latency. Zheng et al. (2019) generated supervised READ/WRITE actions then trained a classification model to predict the action based on encoder and decoder representations. Zheng et al. (2020) introduced a heuristic strategy to compose wait-k models into an adaptive policy based on their uncertainty. Zhang and Zhang (2020) trained a sentence segmentation model to predict complete sentences and feed them through a full-sentence translation
model. Arivazhagan et al. (2019) introduced MILK, where they modified the attention mechanism to learn a Bernoulli variable to decide READ/WRITE actions. Ma et al. (2020) adapted MILK to the transformer architecture. Zhang and Feng (2022) proposed ITST, which quantifies the transported information from source to target then generates a token when the quantity is deemed sufficient. Zhao et al. (2023) trained a supervised policy network based on automatically generated divergence between the predicted distribution of partial and full sentence input.
The majority of the techniques outlined require training multiple models to accommodate different latency levels. Our approach focuses on the efficient training of a single model that can support various latency levels at inference time.

heading: 3 Background

heading: 3.1 Adapters
text: Adapters are lightweight modules that can be inserted into a model for the purpose of task or domain adaptation (Houlsby et al., 2019; Bapna and Firat, 2019). They offer an efficient solution for fine-tuning the model and limiting catastrophic forgetting (Houlsby et al., 2019).
Formally, for a set of N tasks and a model M , the adapter parameters A are introduced. We assume that for each task we have a dataset Dn. The model parameters can be frozen or jointly trained with the adapters. For a frozen model, the model M is pre-trained and the objective function for task n ∈ {1, ..., N} can be defined as:
An ← argmin An Ln(Dn;M,An) (1)
The parameters An are randomly initialized for each task, then they are trained on the dataset Dn in order to minimize the loss function Ln. This results in N adapters that can specialize the model representations to each task n.
In the case of jointly training the model and the adapters, the model parameters M can be randomly initialized or frozen. The objective function can be defined as:
M ′ ← argmin M,A
( N∑
n=1
Ln(Dn;M,An)
) (2)
where M ′ is both the parameters of the model M and the adapters An for n ∈ {1, ..., N}. The parameters An are activated during training depending on the task n.
3.2 Wait-k Policy The wait-k policy (Ma et al., 2019) trains a model to start translating after receiving k source tokens. The model then alternates between writing and reading a new token. It is a fixed policy, where the k value has to be chosen during training and inference. The model reads gk(t) number of source tokens from the source sentence x = (x1, ..., xm) when generating the target token yt, where gk(t) is defined as:
gk(t) = min{|x|, t+ k − 1} (3)
Instead of training the model for a specific wait-k value, Elbayad et al. (2020) introduced the multipath training, which samples k uniformly from [1, ..., |x|] for each batch during training. This enables the model to support multiple wait-k values and allows for information sharing between different wait-k paths. While it was shown that the multi-path training improves the results over the wait-k policy, it does not offer a solution to balance between parameter sharing and interference that we aim at solving by introducing adapters.

heading: 4 Method
text: Our method is composed of two steps: first we train a single model that can support multiple fixed waitk values by using wait-k adapters, then we rely on the probability that the model assigns to the most likely token in order to build an adaptive strategy, where we decide a READ or WRITE action based on a predefined probability threshold.

heading: 4.1 Multi-path Training with Adapters
text: Multi-path training is highly advantageous as an efficient alternative to the wait-k policy, where we need to train multiple models to support more than one latency at inference, but might introduce interference between wait-k paths due to parameter sharing. In order to provide the ability to balance between parameter sharing and interference, we introduce adapters into each decoder layer and we activate adapters according to the wait-k paths they are meant to support. Figure 1 shows an illustration of this. During training, the wait-k value for each batch is sampled uniformly from [1, ..., |x|] following the multi-path training (Elbayad et al., 2020) and based on that, the model decides which adapter will be activated. We set the adapter lagging KA as a list of equally spaced positive integers in increasing order, where each integer speci-
fies the minimum wait-k value supported by each adapter. We insert one adapter for each value in KA. Since the train wait-k is randomly sampled from [1, . . . , |x|], we train each adapter on values starting from its minimum wait-k up until the minimum wait-k of the next adapter. For example, we can set KA = {1, 5, 9, 13} and this will indicate adding 4 adapters, where each adapter will handle 4 wait-k values (starting from each integer in KA until the next), except the fourth adapter (kA = 13), which will handle values starting from 13 up until the length of the input sequence |x|. We follow Bapna and Firat (2019) implementation and insert the residual adapter modules after the feed-forward layer. Algorithm 1 shows the pseudo-code for computing the decoder hidden states at decoding step t using Adapters Wait-k, where H0 is considered to be the input embeddings of the decoder, and gk(t) is computed based on equation 3.
Algorithm 1 Adapters Wait-k Policy Input: Encoder output Z, Decoder hidden states Ht, Adapter lagging KA, Test lagging ktest Output: Hidden states HLt if is_training then
k ← Sample from [1, . . . , |Z|] else
k ← ktest end if for kA in KA do
if k ≥ kA then Al = AlkA for l ∈ [1, . . . , L]
end if end for for l← 1 to L do
H lt = Decoder l(H l−1t , Z≤gk(t)) H lt = A l(H lt) +H l t
end for Return HLt

heading: 4.2 Adaptive Adapters
text: We follow Zheng et al. (2020) to build an adaptive strategy by using adapters instead of different models for each wait-k value, which can be computationally expensive and less efficient. At each decoding step, we activate one adapter based on the lagging behind the current generation step, which is calculated as k = |x| − |y|, where |x| is the number of input tokens and |y| is the number of generated tokens. At the beginning of generation,
|x| = 1 and |y| = 0, which means k starts from 1. Then, we rely on the probability of the most likely token to decide whether to write or read a new token. If the probability is less than a threshold ρk, we read a new token, otherwise, we write. The possible values of k are between kmin and kmax that we determine during inference. If k is lower than kmin, we force the model to read, if it is higher or equal to kmax, we force the model to write, which means that the choice of kmin and kmax also impacts the trade-off between latency and quality (as we analyze in Section 6.1). When the whole input sequence is consumed (i.e. x|x| = 
), we set k to kmax and generate the rest of the target sequence. Algorithm 2 shows the pseudo-code of this method using adapters.
Algorithm 2 Uncertainty based Adaptive Policy Input: Two integers kmin and kmax and a sequence
of thresholds ρk for kmin ≤ k ≤ kmax. Output: Predicted sequence y
while x|x| ̸= 
 and y|y| ̸= 
 do k ← |x| − |y| if k < kmin then
x← x ◦ READ() ▷ READ action else
ytop, ptop ← Pk(M,Ak, x, y) if k < kmax and ptop < ρk then
x← x ◦ READ() ▷ READ action else
y ← y ◦ ytop ▷ WRITE action end if
end if end while while y|y| ̸= 
 do
ytop, ptop ← Pkmax(M,Akmax , x, y) y ← y ◦ ytop ▷ WRITE action
end while return y

heading: 5 Experiments
text: In this section, we describe the datasets we used to evaluate the models and the baselines that we compare against along with the evaluation setup. We also provide the main results of our experiments.

heading: 5.1 Datasets
text: We evaluate our method on two public datasets: the En-Vi dataset for Transformer-Small and De-En for both Transformer-Base and Transformer-Big.
IWSLT152 English → Vietnamese (133K pairs) (Cettolo et al., 2015). We follow the settings of Raffel et al. (2017) and Ma et al. (2020). We use TED tst2012 (1553 pairs) as the validation set and TED tst2013 (1268 pairs) as the test set. We replace tokens with frequency less than 5 with < unk >. The final vocabulary sizes are 17K and 7.7K for English and Vietnamese respectively.
WMT153 German→ English (4.5M pairs) We follow the settings of Ma et al. (2019). We use newstest2013 (3000 pairs) as the validation set and newstest2015 (2169 pairs) as the test set. We apply BPE (Sennrich et al., 2016) with 32K merge operations jointly on the source and target to construct a shared vocabulary.

heading: 5.2 System Settings
text: We conduct experiments on the following systems:
Full Sentence: (Vaswani et al., 2017) Standard Transformer model that takes the full sentence as input before starting to translate.
Wait-k: (Ma et al., 2019) A simple policy that waits for k source tokens before starting to alternate between writing a target token and reading a source token.
Multi-path Wait-k: (Elbayad et al., 2020) Trains a model to support multiple wait-k policies by randomly sampling k during training, then the k value is fixed during inference.
Adaptive Wait-k: (Zheng et al., 2020) It is a method for composing multiple wait-k models during inference in order to build an adaptive strategy. The model is selected based on the lagging behind the generation step, and the decision to write or read is based on the output probabilities.
2nlp.stanford.edu/projects/nmt/ 3www.statmt.org/wmt15/
MoE Wait-k: (Zhang and Feng, 2021) Mixtureof-Experts Wait-k is similar to Multipath Wait-k but applies experts to learn different wait-k policies to avoid interference.
MMA: (Ma et al., 2020) Monotonic multi-head attention (MMA) jointly learns a Bernoulli variable that is used to decide READ/WRITE action.
Adapters Wait-k: Our method as described in Section 4.1.
Adaptive Adapters: Our method as described in Section 4.2.
All implementations are based on the original Transformer architecture (Vaswani et al., 2017) and are using the Fairseq library (Ott et al., 2019). We apply Transformer-Small (4 heads) for En-Vi and both Transformer-Base (8 heads) and TransformerBig (16 heads) for De-En. The encoder is made unidirectional to avoid encoding the source input each time a new token is added.
The evaluation is performed using BLEU (Papineni et al., 2002) for translation quality and Average Lagging (AL)4 (Ma et al., 2019) for latency. AL measures by how many tokens the system is lagging behind an ideal policy (a wait-k policy with k = 0). Given g(t), AL is computed as:
ALg(x, y) = 1
τg(|x|)
τg(|x|)∑
t=1
g(t)− (t− 1)|y|/|x| (4)
where x and y are the source and target sentences respectively, while τg(|x|) = min{t | g(t) = |x|} is the decoding step where the source sentence finishes.
We set the adapter lagging to KA = {1, 3, 5, 7, 9, 11, 13, 15} for our experiments,
4github.com/SimulTrans-demo/STACL
which means that 8 adapters are inserted into the model and we specify the adapter bottleneck size as 64. In Table 1, we report the number of parameters of each method and the number of models required to achieve the latency levels reported in the results section. Adapters Wait-k policy introduces 79.94M parameters into Transformer-Big, but still has the advantage of using one model to support multiple latency levels. In Section 6.3, we experiment with other settings of KA in order to shed light on how much sharing is best between wait-k values during the multi-path training.
The adaptive strategy requires three parameters to be specified at inference, namely, kmin, kmax, and the probability threshold ρk. For En-Vi experiments, kmin and kmax are set to 1 and 9 respectively, while for De-En, we lower kmax to 5, which we have found to improve the results in low latency. We analyze this effect in Section 6.1. ρk decreases as a function of the lagging k, since we want the model to be more aggressive when k is low and more conservative when k is high. We set ρkmin and ρkmax and compute the threshold as: ρk = ρkmin − d.(k − 1), where kmin ≤ k ≤ kmax and d = (ρkmin − ρkmax)/(kmax − kmin). In order to vary the latency, we test the following values of ρkmin and ρkmax : ρkmin ∈ {0.2, 0.4, 0.6, 0.8, 1.}, ρkmax = 0., and ρkmin = 1., ρkmax ∈ {0.2, 0.4, 0.6, 0.8}.

heading: 5.3 Main Results
text: In Figure 2, we compare our methods to previous adaptive and fixed strategies on two language directions. We find that our method improves or competes with other strategies while using a single model. MMA, Wait-k, and Adaptive Wait-k require the training of multiple models in order to support different latency levels (as seen in Table
1), while our method is more efficient in this regard. Adapters Wait-k is competitive with other strong fixed strategies like MoE Wait-k and Multipath Wait-k and it brings further improvements to combine it with the adaptive strategy.
Our method does not support higher latency on De-En because we are using a kmax value of 5 (as seen in Figures 2b and 2c), which we have found to improve results for low latency. However, we show the results for higher kmax and compare them with Adaptive Wait-k on De-En in Section 6.1.
Using adapters alone is competitive with other methods, especially on En-Vi (as seen as in Figure 2a). Compared to Multi-path Wait-k, our method achieves better results on most latency levels, which shows the importance of minimizing interference between different lagging values. Combining our method with an adaptive strategy further improves the results, especially in low latency. In comparison to Adaptive Wait-k, where wait-k policy models are trained and composed during inference, we find that our method is better in all latency levels while being more efficient.
Compared to MoE Wait-k, which also aims at minimizing interference introduced by multi-path training (Zhang and Feng, 2021), we find that our method is better in all latency levels on En-Vi and De-En with Transformer-Big (as seen in Figures 2a and 2c), while achieving competitive results when using Transformer-Base (as seen in Figure 2b). Our method is more flexible in terms of balancing the trade-off between parameter sharing and interference, as we can choose the number of wait-k values supported by each adapter and we can also manipulate the capacity of the adapters by adjusting the bottleneck size. This can bring further improvements but requires more experimentation to find the appropriate hyperparameters.

heading: 6 Analysis
text: In this section, we look into how the performance changes in response to varying the value of kmax, then we provide a wall-clock time comparison between Adapters Wait-k and Multi-path Wait-k. Moreover, we experiment with how balancing between parameter sharing and interference by adjusting the adapter lagging impacts the performance, and also experiment with varying the bottleneck size in order to discern the impact of the complexity of the adapters. At last, we analyze the L2-norm of the adapter representations to discover which
adapter layers are involved in the prediction.

heading: 6.1 Ablation
text: We found that lowering the value of kmax for the adaptive strategy improves the results in low latency, which we believe is the priority in SiMT, but a lower kmax value also limits the ability of supporting high latency. In Figure 3, we show that by increasing the value of kmax we can support high latency and get better quality translations. We compare to Adaptive Wait-k and show that we still achieve better results for all the values of kmax. A lower kmax forces the model to be more aggressive, which in some cases can improve the results in lower latency. The fact that forcing the model to be more aggressive improves the performance signifies that the adaptive strategy decides to wait in cases where the model is able to make a correct prediction, which suggests that the adaptive strategy based on the probability threshold can still be improved by a better strategy.

heading: 6.2 Inference Time
text: Although our method has more parameters than the baseline Multi-path Wait-k due to the additional adapters, the effect on the inference time is not proportional to the number of adapters because only one adapter is activated at a time. To illustrate this, we compare the wall-clock inference time (averaged over 5 runs) of Adapters Wait-k and Multipath Wait-k in Figure 4. It seems that adapters are faster in low k values which could be due to over generation by the Multi-path model (where the model generates longer sequences than it should), while starting from a k value of 7, Multi-path Waitk is better and the difference fluctuates between 0.29s and 0.66s.

heading: 6.3 Adapter Lagging
text: The adapter lagging KA specifies the number of wait-k values that one single adapter will support
and also the number of adapters that we will use. We vary the adapter lagging window between 1 and 5, while maintaining the range between 1 and 16. The results are shown in Figure 5. The wait-k values supported by an adapter controls the amount of sharing and interference between the values. For example, for KA = {1, 5, 9, 13}, adapter A1 will be trained on k ∈ {1, 2, 3, 4}. We note that although it has more parameters, a window of 1 achieves the worst results, which signifies that parameter sharing between wait-k values is crucial. Adapter lagging with window 4 and 5 are competitive especially in low latency, which indicates that lower wait-k values benefit more from sharing. This is consistent with the fact that wait-k models achieve better results when tested on lower wait-k values (Zhang and Feng, 2021).

heading: 6.4 Adapter Bottleneck
text: The adapter’s bottleneck size can be used to tune the representation capacity of the adapters and can be interesting to tune depending on the language pair and the adapter lagging. In Figure 6, we experiment with doubling the adapter’s bottleneck size from 8 to 128, which can be regarded as increasing the representation capacity of the adapter network. We found that the bottleneck size impacts the performance but not in a consistent way - as in larger size results in better performance - but it seems to interact with other hyperparameters (e.g. adapter lagging) to improve or hinder the performance, especially in high latency, where the gap in performance is larger.

heading: 6.5 Adapter Representation Norm
text: We compute the L2-norm of the adapter representations in order to discover which adapter layers are involved in the representations (Liu et al., 2020; Zhu et al., 2021). We measure the L2-norm during inference for kmin = 1 and kmax = 9 while varying the value of ρkmin and ρkmax , as described in Section 5.2. As depicted in Figure 7, the norm for all layers except layer 6 decreases as we increase ρkmin or ρkmax , which correlates with making the adaptive strategy more conservative because the threshold for making a write action is higher. This shows that the adapters are more involved in the prediction when the model is forced to be more aggressive. Only layer 6 is stably invested in adapting the model representations at all the threshold values, which seems to indicate that only low thresh-
old predictions are complex enough to recruit all the adapter layers. Based on this observation, we experiment with inserting adapters only in the last layer (i.e. layer 6). We show in Figure 8 the results of comparing between inserting adapters in all layers and inserting the adapters only in the last layer, where we see a drop in performance only in lower latency levels. This shows that we can make the model more efficient by removing lower layer adapters with a small drop in performance.

heading: 7 Conclusion
text: In this paper, we employ adapters to build a SiMT model that can support multiple latency levels at inference. We use the multi-path training and show that by adding wait-k adapters we can flexibly balance parameter sharing and interference between the wait-k paths. Furthermore, we adopt a simple adaptive strategy and show that it further improves the results. By comparing against strong adaptive and fixed strategies, we find that our method achieves better or competitive results on most latency levels.

heading: 8 Limitations
text: The two datasets we used are common in SiMT research and were selected to compare against other baselines, but evaluating on only two language directions can be a limiting factor for the generalization of our results. Although Vietnamese is from a different language family, it deploys a similar word order (i.e. Subject-Verb-Object) to English and German and we believe that more challenges might emerge when dealing with language directions with a different word order. Additionally, we evaluate latency using common SiMT latency metrics such
as AL, which are sentence-level and do not reflect the nature of a streaming scenario (Iranzo-Sánchez et al., 2021). Furthermore, in this work, we only evaluated on offline data, while evaluating on real interpretation data might offer more realistic results (Zhao et al., 2021).

heading: Acknowledgements
text: The research presented in this paper was conducted as part of VOXReality project5, which was funded by the European Union Horizon Europe program under grant agreement No. 101070521.

heading: A Hyperparameters
text: We list the hyperparameters of our experiments in Table 2.

heading: B Numeric Results
text: In Tables 3, 4 and 5, we report the numeric results of our methods. We report the BLEU score for quality, while for latency we used Average Lagging (AL), Consecutive Wait (CW) (Gu et al., 2017), Average Proportion (AP) (Cho and Esipova, 2016) and Differentiable Average Lagging (DAL) (Arivazhagan et al., 2019). Below we provide the definition of CW, AP and DAL. g(i) constitutes the number of tokens read when predicting yi, while |x| and |y| refer to the number of source and target tokens respectively.
Consecutive Wait (CW) Computes the average number of consecutive tokens read between two predicted tokens.
CW = ∑|y| i=1(g(i)− g(i− 1))∑|y| i=1 Ig(i)−g(i−1)>0
(5)
Average Proportion (AP) Computes the proportion of tokens read to make every prediction.
AP = 1
|x||y|
|y|∑
i=1
g(i) (6)
Differentiable Average Lagging (DAL) Is a differentiable version of the Average Lagging metric.
g′(i) =
{ g(i) if i = 1
max ( g(i), g′(i− 1) + |x||y| ) if i > 1
(7)
DAL = 1
|y|
|y|∑
i=1
g′(i)− i− 1|x|/|y| (8)

heading: Future Work
text: (2021) used Knowledge Distillation from a Full-Sentence Transformer to embed future information into the SiMT model. For adaptive policies, Gu et al. (2017) trained a Reinforcement Learning agent to decide READ/WRITE actions, where the reward function is designed to consider both quality and latency. Zheng et al. (2019) generated supervised READ/WRITE actions then trained a classification model to predict the action based on encoder and decoder representations. Zheng et al. (2020) introduced a heuristic strategy to compose wait-k models into an adaptive policy based on their uncertainty. Zhang and Zhang (2020) trained a sentence segmentation model to predict complete sentences and feed them through a full-sentence translation
model. Arivazhagan et al. (2019) introduced MILK, where they modified the attention mechanism to learn a Bernoulli variable to decide READ/WRITE actions. Ma et al. (2020) adapted MILK to the transformer architecture. Zhang and Feng (2022) proposed ITST, which quantifies the transported information from source to target then generates a token when the quantity is deemed sufficient. Zhao et al. (2023) trained a supervised policy network based on automatically generated divergence between the predicted distribution of partial and full sentence input. The majority of the techniques outlined require training multiple models to accommodate different latency levels. Our approach focuses on the efficient training of a single model that can support various latency levels at inference time