SlowGuess/NAACL_2022_OCR huggingface dataset:
Empirical total time cost of EISL training As discussed in the experiments in the paper, we first pretrain the model with the CE loss until convergence, and then finetune with the EISL loss. Here we report the total time cost of each stage, based on the WMT-18 translation setting as described in Section 4.1. The results are shown in Table 7. As
Figure 12: Results of training and inference time. EISL-  $n$  represents  $n$ -gram EISL loss and EISL-12 represents the combination of 1-gram and 2-gram EISL loss.
the data size increases, the convergence time of both pretraining and finetuning grows. The time cost of the finetuning stage is less than half of that of the pretraining stage.
A.7 Hyperparameters
Regarding which  $n$ -grams to use and their weights  $w_{n}$  in the EISL loss, we found in our experiments that the default values largely following the standard BLEU metric (i.e., maximum  $n = 4$  with equal weights) work well. Specifically, we use  $n \in \{2,3,4\}$  and equal weights  $w_{n} = 1 / 3$  as our default values. Most of our experiments adopt the default values which achieve consistent substantial improvement over CE and other rich baselines as shown in our experiments. (except for the synthetic experiment where we show the effect of different  $n$ -grams including those selected using the validation set).
Besides, in our experiments, we first pretrain the model with the CE loss (i.e., EISL with  $n = T^*$  and teacher forcing, see Section 3.3) and then finetune with the EISL loss. We simply do the CE pretraining until convergence before switching to the EISL finetuning. Therefore, there is no need of tuning for the training iterations of pretraining.
A.8 Analysis of Efficient Implementation
In order to validate the efficiency and accuracy of our approximation (for autoregressive models)
discussed in Section 3.2, we conduct the analysis experiments, showing that the approximate (and efficient) EISL loss values are very close to exact (but expensive) EISL value. We use the same setting as section 4.1, and finetune the model with our efficient approximate EISL loss on Multi30k. Throughout the course of training, we record the loss values of both the exact implementation and our approximate implementation. As shown in Figure 13(a) and (b), the tendency of two losses is very close to each other. We also plot the absolute difference of the two losses as shown in Figure 13(c). We can see the difference decreases as training proceeds. The observations validate the effectiveness of our approximate implementation.
We note that training the model with the exact loss is costly, which necessitates our approximation. Specifically, for  $n$ -gram loss, we need to run the forward pass of the decoder  $(T - n)^2$  times, and keep the whole computation graph for backpropagation, which will consume much more time and memory. Even for only loss evaluation (without the backward pass), we found the runtime of the exact loss is about 15 times longer than that of the efficient approximate implementation based on convolution operator.
Data Size	PreTraining Time (CE)	Finetuning Time (EISL)
1M	1h 40min 57s	49min 33s
2M	5h 56min 57s	1h 35min 10s
4M	8h 55min 18s	3h 57min 44s
Table 7: Convergence time of pretraining and finetuning stages.
Figure 13: The change of loss values during training. The x-axis represents the training step. a) gives the loss curve of exact implementation; b) gives the loss curve of efficient approximate implementation as we discussed in section 3.2; and c) gives the absolute difference between the two implementations.
Source (de)	ein junger mann nimmt an einem lauf teil und derjenige , der dies aufzeichnet , lächelt .
Target (en)	a young man participates in a career while the subject who records it smiles .
SC = 3	CE	young man is running on a a and the other man is smiling .
PG	young man is running on a track and the other man is smiling .
EISL	young man is running in a dirt course and the other is smiling .
SC = 6	CE	young man is running a race and the other is smiling .
PG	young man taking a race and the other smiling . a
EISL	young man is running a race and the other guy is smiling .
SC = 9	CE	young man . a the is running up and up hill smiling taking
PG	young man takes on a slope and thejenige , the the smiles . a
EISL	young man is on a hillside smiling and the others , who is smiling .
RR = 15%	CE	young man is running on a track and the other is smiling .
PG	young man is running on a track and the other is smiling .
EISL	young man is running in a race and the runner is smiling .
RR = 30%	CE	young man man is is running on a track track and the other is is smiling smiling .
PG	young man man is is running on a track track and the other man man who is is smiling .
EISL	young man is running in a race and the other is smiling at him . .
RR = 50%	CE	a young young man man is is smiling smiling at at a window window while another smiles smiles at him him . .
PG	a young man man is is napping napping on on a grassy grassy field field and and some people people are are smiling smiling . .
EISL	young man running in a race and the other is smiling at the action . .
BR = 20%	CE	young man unk unk a run and the unk is smiling .
PG	young man is running in a race and the one who is looking at him is smiling .
EISL	young man is running in a race with the runner who is up .
BR = 35%	CE	young man unk unk a unk , and the unk is smiling unk
PG	young man unk unk track unk others unk .
EISL	young man unk is un in a race and the other un is un at the finish .
BR = 45%	CE	young unk is unk on a unk unk and the unk smiles unk
PG	young man unk a unk tel unk unk .
EISL	young unk un is un in a race , the other is smiling back .
NL = 5	CE	young man is running a race and the one who is running is smiling .
PG	young man is running a race and the one scoring is smiling .
EISL	young man is running a race and one of the runners is up to him .
NL = 15	CE	young man is unk unk a unk and the other man is smiling .
PG	young man is on a unk smiling at thejenige . .
EISL	young man is in a race , the other smiling .
NL = 20	CE	a young man is unk unk a unk and unk is smiling at him .
PG	young smiles on in aid and thejenige smile on . . .
EISL	young man unk unk a ladder and unk , who is unk smiling .
Table 8: Example 1.
Source (de)	15 große hunde spielien auf einem eingezäunten grundstückIBEINEMHaus.
Target (en)	15 large dogs playing in a fenced yard beside a house .
SC = 3	CEPGEISL	large dogs play on a dirt path next to a house .15 large dogs play on an earthen platform next to a house .large dogs are playing on a dirt path next to a house .
SC = 6	CEPGEISL	large dogs play on a play area next to abandoned house .15 large dogs playing on a eingezäunten group stage next to a house .group of dogs play on a abandoned path next to a house .
SC = 9	CEPGEISL	large dogs play a . on a field next to abandoned house dogs play on a snowy grundstück next to a house .15 large .15 large dogs play on an abandoned hillside next to a house .
RR = 15%	CEPGEISL	large dogs are playing on a fenced in area next to a house .large dogs are playing on a fenced in area next to a house .large dogs are playing on a fenced track next to a house .
RR = 30%	CEPGEISL	large dogs dogs play on on a dirt track near a house house .large dogs dogs play on a fenced-in area area next to a house .large dogs play on a fenced walkway next to a house .
RR = 50%	CEPGEISL	small dogs dogs play on on a grassy grassy field field next next to to a house house . .15 large dogs dogs are playing playing on on a grassy grassy field field next next to to a house house . .15 large dogs playing on a fenced terrain next to a house .
BR = 20%	CEPGEISL	large dogs play in a fenced yard next to a house .large dogs are playing on an overcast walk next to a house .large dogs are playing in a fenced area near to a house .
BR = 35%	CEPGEISL	unk dogs play unk a unk unk by a house .large dogs unk a unk path unk unk house .large dogs unk play in a fenced area next to a house .
BR = 45%	CEPGEISL	unk dogs unk on a unk unk next to unk house .large dogs unk a unk unk .large unk un are un in a fenced-out game next to a house .
NL = 5	CEPGEISL	large dogs are playing on a fenced in area next to a house .large dogs are playing on a fenced in area next to a house .large dogs are playing on a fenced backwalk next to a house .
NL = 15	CEPGEISL	large dogs are playing on a unk grassy field next to a house .large dogs playing on a unk next to a house . . .large dogs play on a covered piece of furniture next to a house .
NL = 20	CEPGEISL	large dogs are playing on on a a a grassy grassy field next to a house .large play in autenck in a house . . .large dogs play on a unk unk next to a house . .
Table 9: Example 2.
Source (de)	ein afroamerikanischer mann spielt irgendwo in der stadt gitarre und singt
Target (en)	an african american man playing guitar and singing in an urban setting .
SC = 3	CE	african american man is playing the guitar and singing in the city .
PG	african american man is playing the guitar in the city and singing
EISL	african american man is playing the guitar in the city and singing .
SC = 6	CE	african-american man is playing guitar in the a and singing city .
PG	african american man playing irgendwo in the city guitar singing
EISL	african american man is playing the guitar in the city
SC = 9	CE	african-american man playing guitar in the a and singing city
PG	african americanischer man plays irgendwo in the city guitar singing . a
EISL	african american man is playing the guitar in the city and singing
RR = 15%	CE	african american american man plays guitar guitar in the city city .
PG	african american man is playing guitar in the city and singing .
EISL	african american man is playing guitar in the city and singing .
RR = 30%	CE	african american man plays guitar guitar in in the city city while singing .
PG	african american man plays guitar guitar in the city city and songs .
EISL	an african american man playing guitar in the city and singing . .
RR = 50%	CE	african african american american man playing guitar guitar in in the the city city and singing singing .
PG	african american american man is is playing playing guitar guitar in in the the city city . .
EISL	an african american man playing guitar in the city and singing . .
BR = 20%	CE	african american man plays guitar unk sings unk
PG	african american man is playing guitar and singing in the city .
EISL	african american man is playing the guitar and singing .
BR = 35%	CE	african american man unk unk guitar unk singing unk
PG	african american man unk guitar unk singing unk
EISL	african american unk is un a guitar and singing in the city .
BR = 45%	CE	african american unk unk playing unk guitar in unk city unk
PG	afroamerikanischer man unk irgendwo unk unk
EISL	af unk un playing some sort of guitar in the city and singing .
NL = 5	CE	african american man plays guitar and songs somewhere in the city .
PG	african american man is playing guitar and singing in the city .
EISL	african american man is playing guitar and singing somewhere in the city .
NL = 15	CE	african american man is playing the guitar in the city and singing .
PG	afroamerikanischer man is irgendwo in the city guitarre .
EISL	african american man playing some sort of guitar in the city and singing .
NL = 20	CE	african american american man is playing the guitar in the city unk
PG	afroamerikanischer singt in the city guitarre singt .
EISL	african american man plays unk unk in the city unk
Table 10: Example 3.
Source (de)	ein strandaufsichtgebäude steht im sand , es ist ein bewölkter tag .
Target (en)	a lifeguard building is on the sand on a cloudy day .
SC = 3	CE	beach a is standing in the sand on a beautiful day .
PG	beachfront building is standing in the sand on a beautiful day .
EISL	beach view building is standing in the sand on a cloudy day .
SC = 6	CE	beach a is in the sand building on a beautiful day .
PG	beach viewgeb building standing in sand on a beautiful day .
EISL	beach view building is standing in the sand on a beautiful day .
SC = 9	CE	beach a in the sand . a cloudy day stands beach
PG	beachaufsichtge building stands in sand , the is a beautiful day . a
EISL	. a beachfront building standing in the sand is a beautiful day .
RR = 15%	CE	beachfront building is standing in the sand on a cloudy day .
PG	beachfront building is standing in sand , it is a cloudy day .
EISL	beach building is standing in the sand , it is a cloudy day .
RR = 30%	CE	beachfront beachfront building building is standing standing in the sand
PG	sand on a cloudy day .
EISL	beachfront beachfront building building is standing in sand sand on a cloudy day .
RR = 50%	CE	beachfront beachfront building building is is standing standing in in the
PG	sand , it looks like it is a beach resort resort .
EISL	a beachfront beachfront building building is is standing standing in in sand
BR = 20%	CE	a beach view building is in the sand , it is a cloudy day .
PG	beachfront building is standing in sand on a cloudy day unk
EISL	beachfront building is standing in sand on a cloudy day .
BR = 35%	CE	beach view building is standing in the sand , it is a cloudy day .
PG	beach unk unk standing in sand on a cloudy day unk
EISL	beach unk building unk unk sand unk a cloudy day .
BR = 45%	CE	unk unk is standing unk the sand unk it is a beautiful day unk
PG	beachaufsichtgebäude unk unk sand unk .
EISL	beach unk un is un in the sand , this is a cloudy day .
NL = 5	CE	beachfront view building is standing in the sand on a cloudy day .
PG	beachfront view building is standing in sand on a cloudy day .
EISL	beachfront building is standing in the sand , it is a cloudy day .
NL = 15	CE	beach unk unk is standing in the sand unk it is a sunny day .
PG	beach unk is in sand on a snowy day .
EISL	beach building is in the sand , it is a cloudy day .
NL = 20	CE	beach unk unk is standing in the sand unk it is a sunny sunny day .
PG	beachaufsichtgebäude steht in sand , es is a day .
EISL	beach unk stands in sand unk it is a sunny day .
Table 11: Example 4.
Source (de)	zwei hunde haben beim spielten dasselbe holzstück im maul .
Target (en)	two dog is playing with a same chump on their mouth .
SC = 3	CEPGEISL	dogs are two playing with . pieces of wood in their mouths two dogs are playing with pieces of black wood in their mouths . two dogs are playing with pieces of wood in their mouths .
SC = 6	CEPGEISL	dogs are two . playing with sticks in their mouths two dogs have been playing with pieces of wood in their mouths . two dogs are playing with pieces of wood in their mouths .
SC = 9	CEPGEISL	two dogs their . are playing with sticks in muzzled dogs haben beim play pieces in their mouth . two . two dogs have been playing with sticks in their mouth .
RR = 15%	CEPGEISL	two dogs are playing with a piece piece of wood in their mouth . dogs are playing with white wooden blocks in their mouth . two dogs are playing with some pieces of wood in their mouths .
RR = 30%	CEPGEISL	two dogs dogs are playing with a piece piece of wood in their mouths . dogs dogs are playing with white wooden blocks blocks in their mouth . two dogs are playing with pieces of wood in their mouths .
RR = 50%	CEPGEISL	two dogs dogs are playing playing with with plastic plastic sticks sticks in in their their mouth mouth . two dogs dogs are playing with with plastic holsters holsters in in their maul maul . two dogs have playing with some white wood in their mouths .
BR = 20%	CEPGEISL	dogs unk unk pieces of wood in their mouths . dogs are playing with wet wood in their mouths . dogs are playing with wet pieces of wood in their mouths .
BR = 35%	CEPGEISL	unk have unk pieces of unk in their mouths . two dogs unk unk piece of wood unk their mouth . two dogs unk playing with some piece of wood in their mouth .
BR = 45%	CEPGEISL	dogs are playing with unk unk in unk mouth unk dogs unk unk piece of unk holzstück unk . dogs unk un are un while play with some wood pieces in their mouth .
NL = 5	CEPGEISL	two dogs are playing with the same piece of wood in their mouths . dogs have pieces of wood in their mouths . two dogs are playing with the same piece of wood in their mouths .
NL = 15	CEPGEISL	two dogs are are playing with unk unk in their mouths . dogs haben on a game unk unk . . . two dogs have been playing with a piece of wood in their mouth .
NL = 20	CEPGEISL	two dogs are are playing with unk unk in their mouths . dogs haben in a playenselbeck in their mouth . two dogs are playing with unk sticks in their mouths . .
Table 12: Example 5.