Average Token Delay: A Latency Metric for Simultaneous Translation
Yasumasa Kano1 , Katsuhito Sudoh1 , Satoshi Nakamura1
1

Nara Institute of Science and Technology, Japan{kano.yasumasa.kw4, sudoh, s-nakamura}@is.naist.jp

Abstract

Wait-3
SRC SRC SRC

SRC
TGT

In simultaneous translation, translation begins before thespeaker has finished speaking. In its evaluation, we have toconsider the latency of the translation in addition to the quality,with latency preferably as small as possible for users to comprehend what the speaker says with a small delay. Existing latency
metrics focus on when the translation starts but do not consider
adequately when the translation ends. This means such metrics
do not penalize the latency caused by a long translation output,
which delays user comprehension. In this work, we propose
a novel latency evaluation metric called Average Token Delay(ATD) that focuses on the end timings of partial translations insimultaneous translation. We discuss the advantage of ATD using simulated examples and investigate the differences between
ATD and Average Lagging with simultaneous translation experiments.
Index Terms: simultaneous translation, latency evaluation

1. Introduction
Machine translation (MT) has evolved rapidly using recent neural network techniques and is now widely used both for written
and spoken languages. MT for speech is a very attractive application for translating various conversations, lecture talks, etc.
For smooth real-time speech communication across languages,
speech MT should run in real-time and incrementally without
waiting for the end of an input utterance as in consecutive interpretation. While the translation quality can improve by waiting
for later inputs as the context, it should result in longer latency.
This quality-latency trade-off is the most important issue in simultaneous MT.
Most recent simultaneous MT studies use BLEU [1] forevaluating the quality and Average Lagging [2] for the latency.
AL is based on the number of input words that have become
available when starting a translation and measures its averageover all the timings of generating partial translations. It is very
suitable for wait-k [2] that waits for k input tokens before starting the translation and then repeats to read and write one token
alternately.
This work focuses on a problem of AL. AL does not sufficiently consider the cases where chunk-level outputs are generated at a time and give smaller latency values for them than
one-by-one cases like wait-k. In addition, AL has a critical flaw
that AL can be negative when the chunk-level outputs become
longer, although the latency should not be negative.
Suppose we have a pair of seven input tokens and sevenoutput tokens and apply two different simultaneous policies tothis pair as shown in Figure 1: wait-3 and chunk-3 that writes
three tokens after reading three tokens. Figure 2 illustrates asituation where one input or output token requires one unit time

SRC
TGT

Chunk-3
SRC SRC SRC

TGT

SRC
SRC
TGT
TGT TGT TGTSRC SRC SRCTGT TGT TGT

1

2

3

4

5

6

SRC
TGT TGT TGT

7

8

9

10

11

12

TGT
13

14

Figure 1: wait-3 and chunk-3 in a step-wise view.
Wait-3
SRC SRC SRC / SRC / SRC / SRC / SRC /TGT / TGT / TGT / TGT / TGT TGT TGT
Chunk-3
SRC SRC SRC / SRC SRC SRC / SRC
TGT TGT TGT / TGT TGT TGT / TGT
1

2

3

4

5

6

7

8

9

10

Figure 2: wait-3 and chunk-3 in a time-synchronous view.

to speak, ignoring computation time for simplicity. These policies are equivalent to each other, from the viewpoint of latency
in this situation. However, AL for wait-3 is 15= 3 and that
5≈
1.857;
theequationofALis explainedfor chunk-3 is 13
7
in Section 2. AL tends to give a small latency value for sucha long chunk-level output. However, a long output delays thestart of the translation of later parts and makes the listener feelthe delay. This problem becomes more severe when we needspeech outputs because the speech outputs must be sequential.
Therefore, the length of a translation output has a large effect ondelay and should be included in the latency measurement. Thisobservation suggests the need of another latency metric to copewith such situations.
We propose a novel latency metric called Average Token
Delay (ATD) 1 that focuses on the end timings of partial translations. ATD generalizes the latency measurement for simultaneous translation both with speech and text outputs and works intuitively for chunk-based outputs that are not properly handled
by AL as presented above. We present some simulation results
to show the characteristics of ATD clearly and also demonstrate
its effectiveness through simultaneous translation experiments.

2. Existing Latency Metric for
Simultaneous Translation
Gu et al.[3] proposed a latency metric called Consecutive Wait
(CW) to measure the local delay. Since CW is not originallyan average measure of the latency, Ma et al.[2] used a step-wise
CW. The average CW result in the same value for the follow1 ATD will be implemented in https://github.com/
facebookresearch/SimulEval

ing two extreme cases: (1) all the output tokens are generated
after reading all input tokens and (2) all the output tokens are
generated after reading only the first input token. Cho et al.[4]proposed Average Proportion (AP). AP differs with the changeof the sequence length for the same simultaneous translation
policy and is not intuitive for the latency metric.
Ma et al.[2] proposed Average Lagging (AL). AL is denoted
as follows:1
ALg (x, y) =τg (|x|)τg (|x|) 

X

τ =1

τ −1
g(τ ) −
r


.

(1)

x is the input sentence and y is its translation. g(τ ) is amonotonic non-decreasing function representing the number ofsource words read to predict the τ -th target word. r is the length
ratio defined as |y|/|x|. τg (|x|) is the cut-off step defined asfollows:
τg (|x|) = min{τ | g(τ ) = |x|}(2)
meaning the index of the output token predicted right after theobservation of the entire source sentence.
AL solves the problem of AP mentioned above by focusing on the difference from the ideal policy based on the length
ratio. However, AL still suffers from unintuitive latency measurement because AL can be negative when the model finishes
the translation before reading the entire input due to the subtraction term. Ma et al.[5] modified AL by changing the calculationof the length ratio r based on the length of the reference translation. Papi et al.[6] proposed Length-Adaptive Average Lagging(LAAL) that uses the longer one between the reference and output. These modifications do not completely avoid negative values when the model generates a long partial output faster than
the ideal policy. Arivazhagan et al.[7] proposed another variant,Differentiable Average Lagging (DAL), for optimizing simultaneous translation model. Iranzo-Sánchez et al.[8] proposed amethod to calculate AL for a streaming input in a segmentationfree manner.

3. Proposed Metric: Average Token Delay
We propose a novel latency metric called Average Token Delay(ATD). We start from the latency measurement using ATD incase of simultaneous speech-to-speech MT and then generalize
it for speech-to-text and text-to-text cases.
3.1. ATD for simultaneous speech-to-speech translation
Figure 3 illustrates a step-by-step workflow of simultaneous
speech-to-speech MT. The white boxes represent fixed-length
speech segments, the orange ones represent the processing timeto encode prefix inputs and to judge whether prefix translations
can be predicted there, and the blue ones represent decoding
time. Intuitively, ATD is the average time difference between
the points of the same color (black, red, and white).
Suppose we have an input x and an output y segmented
into C chunks x = x1 , x2 , ..., xC and y = y 1 , y 2 , ..., y C ,respectively. An input chunk xc is what is observed after theprevious judgment for the prefix translation y c−1 and is used
to predict y c . In the case of text, each chunk consists of subsegments with words, subwords, or characters. In the case ofspeech, we divide each chunk into sub-segments of 0.3 seconds
long from the beginning of the chunk, assuming one word isuttered in 0.3 seconds. Through this segmentation, the input andoutput sentences can be represented as a series of sub-segments
x = x1 , ..., x|x| , y = y1 , ..., y|y| respectively.

src
tgt

1. Policy : READ

src
tgt

2. Policy : WRITE

src
tgt

3. Decode and Output
speech segment

src
tgt

4. Policy : WRITE

src
tgt

5. Decode and Output
speech segment

src
tgt

6 . Policy : READ

src
tgt

7 . Policy : WRITE

src

T (xa(1)) T (xa(2)) T (xa(3))

8 . Decode and Output
speech segmenttgt
T (y1)T (y2)T (y3)

Figure 3: Step-by-step example of simultaneous speech-tospeech MT

ATD is defined as follows:
|y|


1 X
T (yt ) − T (xa(t) )|y| t=1

(3)

(
s(t)
if s(t) ≤ Lacc (xc(t) )a(t) =c(t)Lacc (x ) otherwise(4)s(t) = t − max(Lacc (y c(t)−1 ) − Lacc (xc(t)−1 ), 0)(5)

ATD(x, y) =where

T (·) in Eq. (3) represents the ending time of each token,
which is shown as colored points in Figure 3. a(t) representscthePc indexjof the input token corresponding to yt . Lacc (x ) =j=1 |x | is the cumulative length up to the c-th chunk, andLacc (x0 ) = 0. Lacc (y c ) is defined similarly. c(t) denotes the
chunk number c to which yt belongs. As shown in Eq. (5), if theprevious translation prefix is longer than the previous input prefix, s(t) becomes smaller than the output index t, which means
the previous long output makes the time difference between the
input token and the corresponding output token larger. ATD isguaranteed not to become negative in any conditions due to thenature of these equations.
ATD is the average delay of output sub-segments against
their corresponding input sub-segments, considering the latency
required for inputs and outputs. Although the input-output correspondence does not necessarily mean semantic equivalence,
especially for language pairs with large differences in their word
order and the numbers of tokens, we use this simplified formulation for the latency measurement as same as AL.
Figure 4 shows examples to explain the term in max operator in Eq. (5). In Figure 4a, Suppose we measure the token
delay on y5 . y5 is in the second output chunk, so c(5) = 2.
Since Lacc (y 1 ) = Lacc (x1 ) = 3, we obtain a(5) = s(5) =5 − 0 = 5 ≤ Lacc (x2 ) = 5, and therefore y5 corresponds
to x5 . In Figure 4b, Suppose we measure the token delayon y2 . y2 is in the second output chunk, so c(2) = 2. Since
Lacc (y 1 ) − Lacc (x1 ) = 1 − 3 < 0, we obtain a(2) = s(2) =2 − 0 = 2 ≤ Lacc (x2 ) = 5, therefore y2 corresponds to x2 . In
Figure 4c, the first output chunk is longer: Lacc (y 1 ) = 4. This
results in a(5) = s(5) = 5 − 1 = 4 ≤ Lacc (x2 ) = 5, so y5
corresponds to x4 .

← Lacc (x1 ) →x1
x1 x2 x3x

← Lacc (x1 ) →x1
x1 x2 x3

2

x4
y1

x5
y2
y3
y1
← Lacc (y 1 ) →

y4

y5

y1

y2 y3
y1
← Lacc (y 1 ) →y2(a) Lacc (y c(t)−1 ) − Lacc (xc(t)−1 ) = 0 (t = 5)

← Lacc (x1 ) →x1
x1 x2 x3x(a) s(t) ≤ Lacc (y c(t) ) (t = 3)

← Lacc (x1 ) →x1
x1 x2 x3

2

x4
y1
y1

x5
y2

y3

y4

y5

y3 y4
y1
←− Lacc (y 1 ) −→← Lacc (y 1 )→(b) Lacc (y c(t)−1 ) − Lacc (xc(t)−1 ) < 0 (t = 2)

← Lacc (x1 ) →x1
x1 x2 x3

y1

y2

y2

(b) s(t) > Lacc (y c(t) ) (t = 4)

Figure 5: Example for the explanation of Eq. (4)
x2
x4
y1

x5
y2

y3 y4
y1
←− Lacc (y 1 ) −→

y5

y6
y2

(c) Lacc (y c(t)−1 ) − Lacc (xc(t)−1 ) > 0 (t = 5)

Figure 4: Examples for the explanation of Eq. (5)Figure 5 shows examples to explain Eq. (4). In Figure 5a,
we measure the token delay on y3 . Here, we obtain s(t) = 3 ≤Lacc (x1 ) = 3, so y3 corresponds to x3 . Figure 5b, we measure
the token delay on y4 . Here, we obtain s(t) = 4 > Lacc (x1 ) =3, so y4 corresponds to x3 .
3.2. ATD for simultaneous {speech,text}-to-text translation
Figure 6 illustrates the latency measurement for speech-to-text
simultaneous translation, where the output duration can be ignored. Figure 7 is the one for text-to-text simultaneous translation. We reserve input duration here because input for text-totext simultaneous translation comes from speech via automatic
speech recognition (ASR), in most cases. The input durationreflects ASR computation time.
3.3. Non-computation-aware ATD
We sometimes use the latency measurement independent of thecomputation time for estimating ideal situations that are not influenced by the performance of computers and the efficiency of
implementations. In Figures 3, 6, and 7, we remove the orange,
blue and yellow parts and only include the duration of speechsegments to calculate delay. However, this means all the terms
in text-to-text translation become 0. We follow the conventional
step-wise latency measurement as CW and AP by letting each
input and output word spend one step as shown in Figure 8.
Also, we assume the model can read the next input and output
the partial translation in parallel as shown in Figure 2.

4. Simulation
Before presenting the latency measurement experiments using
real data, we show simulation results comparing AL and ATD
in different conditions in simultaneous text-to-text translation.

Figure 6: Summary view for latency measurement for simultaneous speech-to-text translation

4.1. Comparison of Wait-k and Chunk-k
We assume that the numbers of tokens of the input and output
are both 40. We compare the latency measurement by AL and
ATD where the hyperparameter k for wait-k and chunk-k varies
from 1 to 40. Here, for simplicity, we assume the length of input
and output chunks are the same for chunk-k until the predictionof the end-of-sentence token.
Figure 9 indicates the gap between wait-k and chunkk by AL mentioned in section 1, while ATD results in thesame values for them as shown in Figure 10. One serious
problem raised here is the large jump in AL at k = 40.
For example, in the case of chunk-39, AL uses r = 1,
τgchunk-39 (|x|) = 40, gchunk-39 (τ ) = 39 (1 ≤ τ ≤ 39), andgchunk-39) = 40 (τ = 40). Then the AL value becomes
 P(τ
39
781
1
τ =1 τ + 1 = 40 = 19.525. However, in the case40of chunk-40, gchunk-40 (τ ) = 40 for all τ and τgchunk-40 (|x|) = 1
according to Equation 2. As a result, the AL value becomes
40. This phenomenon comes from the definition of the cut-off
step described in Section 4.2 by Ma et al.[2], where they assume
later outputs derive no further delays.
4.2. Translation example
Figure 11 shows examples of chunk-based simultaneous translation for the input I bought a pen. by three different models.
Model 1 waits for the whole input sentence and results inthe largest delay and the highest translation quality.
Model 2 has a smaller delay than Model 1 because it cansegment the input after observing I. The segmentation enables
the model to generate a partial translation but causes quality
degradation due to the lack of context.
Model 3 works similarly to Model 2 for the input segmentation but outputs a longer segment for I. This causes large quality
degradation by over-translation.
Regarding the latency of these three models, AL is theFigure 7: Summary viewfor latency measurement
for simultaneous text-to-texttranslation

Figure 8: Summary viewfor non-computation-aware
latency measurement for
simultaneous
text-to-texttranslation

55
50
45
40
35
30
25
20
15
10
5
0

55
50
45
40
35
30
25
20
15
10
5
0

0

5

10

15

20
k

25

30

35

40

Figure 9: Latency measurement by AL

20220307変更
Model 2 AL 1

Model 3
ATD:4.1, AL:0.8, Quality: 3rd
I / bought a pen. /私で ござい ます 。/ペン を 買っ た 。
1 2
3 4
5
6 7
8 9
10 11

Figure 11: Translation example

wait-kchunk-k
0

5

10

15

20
k

25

30

35

40

Figure 10: Latency measurement by ATD

30

30

25

25

20
PA-0
PA-1
wait-k
MU-0
MU-1
ICLP

15
10
0.0

smallest for Model 3 and decreases largely from Model 1 toModel 2; they are not intuitive as discussed so far. In contrast,
ATD results in reasonable latency values considering the delaycaused by the outputs.

2.5

5.0

7.5

10.0
AL

12.5

15.0

BLEU

wait-kchunk-kModel 2
ATD:3.4, AL:1.6, Quality: 2nd
I / bought a pen . /私。
/ ペン を 買っ た 。
1 2
3 4 5 6
7 8
9 10

BLEU

ATD

AL

Model 1
ATD:5.4, AL:5.0, Quality: 1st
I bought a pen . /私 は ペン を 買っ た 。
12
3 4 5 6 7 8
9 10 11 12

20

10

17.5

Figure 12: Latency measurement by AL

PA-0
PA-1
wait-k
MU-0
MU-1
ICLP

15

4

6

We conducted analyses on actual simultaneous translation results to investigate the effectiveness of ATD. Note that the analyses here were conducted on simultaneous text-to-text translation for simplicity.

14

16

18

PA-0
PA-1
wait-k
MU-0
MU-1
ICLP

1.7
1.6

Ratio

10
12
ATD

Figure 13: Latency measurement by ATD

1.5

5. Analyses

8

1.4
1.3
1.2
1.1
1.0
0.0

2.5

5.0

7.5

10.0
AL

12.5

15.0

17.5

Figure 14: Length ratio and AL

5.1. Data
We used the data from the IWSLT evaluation campaign forEnglish-to-Geman simultaneous translation. We used WMT2014 training set (4.5 M sentence pairs) for pre-training and
IWSLT 2017 training set (206 K sentence pairs) for fine-tuning.
The development set consists of dev2010, tst2010, tst2011 andtst2012 (5,589 sentence pairs in total), and the evaluation set is
tst2015 (1,080 sentence pairs).
We compared wait-k [2], Meaningful Unit [9], Incremental Constitute Label Prediction [10], and Prefix Alignment [11],following the the experimental settings in the literature [11].
5.2. Results
As shown in Figures 12 and 13, ATD demonstrated clear differences in delay among models compared to AL. MU and ICLP
were affected by the change in the latency metric. We analyzed their results in detail and found this degradation was due
to over-translation as suggested by the observations of length
ratio results shown in Figure 14. This phenomenon is the sameas what happened with Model 3 in subsection 4.2. MU and
ICLP generated long translations exceeding the length ratio of
1.0 when they worked with small latency. One interesting finding here is the correlation between BLEU and ATD by MU;
larger latency did not always result in better BLEU. It is because over-translation increases ATD, but decreases BLEU at
the same time. In contrast, wait-k is a strategy that generates

one output token at a time and can avoid such an issue. PA also
worked well with the latency measurement by ATD because it
fine-tunes the translation model to avoid over-translation.

6. Conclusion
We proposed a novel latency metric ATD for simultaneous machine translation, which addresses the problem in the latency
evaluation for a chunk-based model by taking the output length
into account. ATD gives a large latency value to a long output
based on the assumption that the output also causes a delay, different from AL. We revealed the effectiveness of ATD throughthe analyses of simulation and actual translation results compared with AL.
Future work includes studies on semantics-oriented latency
measurement not just focusing on timing information without
any consideration about the delivery of contents.

7. Acknowledgements
Part of this work was supported by JSPS KAKENHI Grant
Numbers JP21H05054 and JP21H03500.

8. References[1] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method
for automatic evaluation of machine translation,” in Proceedingsof the 40th Annual Meeting of the Association for Computational
Linguistics. Philadelphia, Pennsylvania, USA: Association for
Computational Linguistics, Jul. 2002, pp. 311–318. [Online].
Available: https://aclanthology.org/P02-1040[2] M. Ma, L. Huang, H. Xiong, R. Zheng, K. Liu, B. Zheng,
C. Zhang, Z. He, H. Liu, X. Li, H. Wu, and H. Wang,“STACL: Simultaneous translation with implicit anticipationand controllable latency using prefix-to-prefix framework,” inProceedings of the 57th Annual Meeting of the Association for
Computational Linguistics. Florence, Italy: Association for
Computational Linguistics, Jul. 2019, pp. 3025–3036. [Online].
Available: https://aclanthology.org/P19-1289[3] J. Gu, G. Neubig, K. Cho, and V. O. Li, “Learning totranslate in real-time with neural machine translation,” inProceedings of the 15th Conference of the European Chapterof the Association for Computational Linguistics: Volume 1,
Long Papers. Valencia, Spain: Association for Computational
Linguistics, Apr. 2017, pp. 1053–1062. [Online]. Available:
https://aclanthology.org/E17-1099[4] K. Cho and M. Esipova, “Can neural machine translation dosimultaneous translation?” arXiv preprint arXiv:1606.02012.,
2016.
[5] X. Ma, M. J. Dousti, C. Wang, J. Gu, and J. Pino, “SIMULEVAL:
An evaluation toolkit for simultaneous translation,” in Proceedings of the 2020 Conference on Empirical Methods in Natural
Language Processing: System Demonstrations. Online: Association for Computational Linguistics, Oct. 2020, pp. 144–150. [Online]. Available: https://aclanthology.org/2020.emnlp-demos.19[6] S. Papi, M. Gaido, M. Negri, and M. Turchi, “Overgeneration cannot be rewarded: Length-adaptive average
lagging for simultaneous speech translation,” in Proceedingsof the Third Workshop on Automatic Simultaneous Translation.
Online: Association for Computational Linguistics, Jul. 2022,
pp. 12–17. [Online]. Available: https://aclanthology.org/2022.
autosimtrans-1.2[7] N. Arivazhagan, C. Cherry, W. Macherey, C.-C. Chiu, S. Yavuz,
R. Pang, W. Li, and C. Raffel, “Monotonic infinite lookback
attention for simultaneous machine translation,” in Proceedingsof the 57th Annual Meeting of the Association for Computational
Linguistics. Florence, Italy: Association for Computational
Linguistics, Jul. 2019, pp. 1313–1323. [Online]. Available:
https://aclanthology.org/P19-1126[8] J. Iranzo-Sánchez, J. Civera Saiz, and A. Juan, “Stream-level
latency evaluation for simultaneous machine translation,” inFindings of the Association for Computational Linguistics:
EMNLP 2021. Punta Cana, Dominican Republic: Association
for Computational Linguistics, Nov. 2021, pp. 664–670. [Online].
Available: https://aclanthology.org/2021.findings-emnlp.58[9] R. Zhang, C. Zhang, Z. He, H. Wu, and H. Wang, “Learning
adaptive segmentation policy for simultaneous translation,” inProceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing (EMNLP). Online: Association
for Computational Linguistics, Nov. 2020, pp. 2280–2289.
[Online]. Available: https://aclanthology.org/2020.emnlp-main.
178[10] Y. Kano, K. Sudoh, and S. Nakamura, “Simultaneous neuralmachine translation with constituent label prediction,” inProceedings of the Sixth Conference on Machine Translation.
Online: Association for Computational Linguistics, Nov. 2021,
pp. 1124–1134. [Online]. Available: https://aclanthology.org/
2021.wmt-1.120[11] ——, “Simultaneous neural machine translation with prefixalignment,” in Proceedings of the 19th International Conferenceon Spoken Language Translation (IWSLT 2022). Dublin,
Ireland (in-person and online): Association for Computational
Linguistics, May 2022, pp. 22–31. [Online]. Available: https://aclanthology.org/2022.iwslt-1.3