Average Token Delay:
A Latency Metric for
Simultaneous Translation

Yasumasa Kano, Katsuhito Sudoh, Satoshi Nakamura
Nara Institute of Science and Technology, Japankano.yasumasa.kw4@is.naist.jp

Simultaneous Translation
Standard translationIbought

apen
私はペン

買った

Simultaneous translationIbought
私

aはpen
ペン

買った2

Latency of Simultaneous Translation
Case 1:I

Case 2:
I

Avg. EVS = (1 + 2+ 1 )/3 = 1.3
bought

apen

私は買った

(Smaller Latency)
EVS(Ear-Voice-Span)ペン

Avg. EVS = (1 + 4+ 1 )/3 = 2.0
boughta

私はpenペン買った3

Quality of Simultaneous Translation
Case 1
Subject

Verb

Ibought
私

ObjectaはSubject

Case 2

Japanese word order:
SOV (Subject-Object-Verb)

pen
買った

ペン

Verb

ObjectUngrammatical inJapanese(Higher quality)

SubjectVerb

Ibought私
Subject

Object

aは

pen
ペン

買った

ObjectVerb

4

Quality-Latency Trade-off
Latency: Small Quality: Low

Case 1
Ibought

apen

私

は買った

ペン

Verb

ObjectSubject

Case 2I

Ungrammatical in
Japanese

Latency: Large Quality: Highboughta

私はSubject

pen
ペン

買った

ObjectVerb

5

Previous Latency Metric: AL (Average Lagging)$(|𝒙|)

1𝑡−1
𝐴𝐿 =* 𝑔 𝑡 −τ(|𝒙|)
γ[Ma+, 2019]!"#Delay𝒙 = 𝑥! , 𝑥" , … , 𝑥|𝒙| : Input tokens
𝒚 = 𝑦, 𝑦" , … , 𝑦|𝒙| : Output tokens
|𝒚|
γ=|𝒙|

Catch-up

Lagging

Average Lagging

𝑡 : Target token index
𝑔(𝑡) : Number of input tokens read to output 𝑡 th target token
τ 𝑿= min 𝑡 𝑔 𝑡 = |𝒙|)6

Calculation of AL (Ideal Strategy)
Target (Catch-up)

私はペンTime synchronous view

買ったsource:target:私Iboughtaはペン買ったpenIboughtSource(Delay)a

Ideal Strategy(AL = 0)pen

7Calculation of AL (Case 1)
私はペンTime synchronous view (Case 1)

買ったsource:IItarget:boughtboughtapen

私はペン

買った

Ideal Strategy(AL = 0)apen

AL = (1 + 1 + 1 + 1 ) / 4 = 1

Real Strategy

8Case 3: Long Translation Output
Case 3:I

Avg. EVS = (1 + 7+ 4 )/3 = 4.0
boughtapen

私はですね(Larger Latency)

え〜

ペン

買った

ReviewCase 1:
I

Avg. EVS = (1 + 2+ 1 )/3 = 1.3
boughtapen

私は買った

ペン

9

Calculation of AL (Case 3)
Time synchronous view (Case3)

Iboughtapen

私はですねえ〜ペン買った
Real Strategy

Ibought

Ideal Strategy(AL = 0)apen

AL = (1.0 + 0.5 + 0.0 – 0.5 – 1.0 – 1.5 + 1.0 ) / 7= -0.0710

Problems of AL• Longer translation output → Smaller latency• Negative latency value
Propose latency metrics to solve the problems

11

Proposed Metric: ATD (Average Token Delay)|𝒚|

1𝐴𝑇𝐷 =* 𝑇 𝑦! − 𝑇(𝑥7(!) )|𝒚|!"#Target token time𝒙 = 𝑥! , 𝑥" , … , 𝑥|𝒙| : Input tokens
𝒚 = 𝑦, 𝑦" , … , 𝑦|𝒙|

Source token time

Token Delay: Output tokens

𝑡 : Target token index
𝑎(𝑡) : Source token index corresponding
to target token index 𝑡𝑇(∗) : Ending time of each token

Average Token Delay

12

Calculation of ATD (Inspired by EVS)
Case 1:

ATD = (1 + 1 + 1 + 1)/4 = 1.0
Ibought
私Case 3:aはAvg. EVS: 1.3
Token Delay:
Without Semantic
Correspondence

pen
買った

EVS:
With Semantic
Correspondence

ペン

ATD = (1 + 2 + 3 + 4 + 5 + 6 +6 + 6)/8 = 4.1
Iboughtapen

私はですねえ〜Avg. EVS: 4.0

ペン

買った13

Comparison of AL and ATD• AL• Longer translation output → Smaller Latency• Negative latency value• ATD• Close to EVS• Longer translation output → Larger Latency• Non-negative latency value

ATD solves the problems of AL
14

Experiment of Simultaneous Translation
1.Wait-k [Ma+, 2019] k=2

To adjust latency:
k = [2, 4, 6, …, 30]

2Iboughtapen

私は買った

ペン

Larger k,Larger latency

2.Fixed-size segmentation k=2
2I

2boughtapen

私はペン買った

15

Result
Result (En-De)
BLEU

BLEU
[Papineni+,2022]

AL
Larger Delay,
Higher Quality

ATD
Larger Delay,
Lower Quality
16

Analysis: Length ratio
𝑝𝑟𝑒𝑑𝑖𝑐𝑡𝑒𝑑 𝑡𝑟𝑎𝑛𝑠𝑙𝑎𝑡𝑖𝑜𝑛 𝑙𝑒𝑛𝑔𝑡ℎ𝑙𝑒𝑛𝑔𝑡ℎ 𝑟𝑎𝑡𝑖𝑜 =𝑟𝑒𝑓𝑒𝑟𝑒𝑛𝑐𝑒 𝑙𝑒𝑛𝑔𝑡ℎUnnecessarywords

LargerLengthRatio

Latency

Quality
Omissionk

ATD reflects this characteristic

Smaller17

Conclusion• Problems of AL• Longer translation output → Smaller latency• Negative latency value• Proposed latency metric: ATD• Solved the problems above• Future work• Investigate correlation between latency metrics and latency scoresevaluated by human18