SlowGuess/NAACL_2022_OCR huggingface dataset:
5. Each test set is decoded 30 times on different hosts, and we report the mean p90 latency with its  $95\%$  confidence interval. Alignment-based vocabulary selection includes the top  $k$  most frequently aligned BPE tokens for each source token based on a fast_align model trained on the same data as the translation model. NVs include all tokens that are scored above the threshold  $\lambda$ . All vocabulary selection methods operate at the BPE level.
Evaluation Human Evaluations and COMET / BLEU use full precision (FP32) inference outputs. We decided to use FP32 for human evaluation as we wanted to evaluate the quality of the underlying model independent of whether it gets used on CPU or GPU and the output differences between FP16/FP32/INT8 being small. We report mean and standard deviation of SacreBLEU (Post, 2018) and COMET (Rei et al., 2020) scores on detokenized outputs for three runs with different random seeds. For human evaluations, bilingual annotators see a source segment and the output of a set of 4 systems at once when assigning an absolute score to each output. The size of the evaluation set was 350 for EN-DE and EN-RU and 200 for DE-EN and RU-EN for newstest2020. We used the full sets of sentences differing between NVS  $\lambda = 0.9$ , align  $k = 200$  for the ITDS test set (309 for EN-DE and 273 for DE-EN).
Adaptation For domain adaptation, we fine-tune the NVS model for 10 epochs using a learning rate of 0.0001 and a batch size of 2048 target tokens. To adapt the alignment-based vocabulary selection model, we include the adaptation data as part of the training data for the alignment model. We upsample the adaptation data by a factor of 10 for a comparable setting with NVS fine-tuning.
B Positive class weight ablation
Based on preliminary experiments we had used a weight for the positive class  $(\lambda_p)$  of  $100k$  in the experiments in  $\S 4$ . Here the positive class refers to tokens being present on the target side and the negative class to tokens being absent from the target side. For a Machine Translation setting there are many more words that are not present than are present on the target side. The negative class therefore dominates the positive class. This can be
$^{8}$ BLEU+case.mixed+lang.en-de+numrefs.1+smooth.exp+tok.13a+version.1.4.14.
pos. weight	BLEU	COMET
EN-DE	auto x = 10	34.4	0.459
auto x = 1	34.1	0.458
100k	34.4	0.461
10k	34.2	0.460
1k	34.4	0.463
100	34.2	0.456
10	32.5	0.295
1	15.9	-0.498
DE-EN	auto x = 10	40.9	0.644
auto x = 1	40.8	0.640
100k	40.8	0.642
10k	41.0	0.645
1k	40.8	0.643
100	40.8	0.638
10	40.2	0.558
1	25.3	-0.608
EN-RU	auto x = 10	23.6	0.524
auto x = 1	23.5	0.524
100k	23.6	0.524
10k	23.7	0.528
1k	23.6	0.526
100	23.3	0.497
10	20.6	0.128
1	5.6	-1.509
RU-EN	auto x = 10	35.6	0.564
auto x = 1	35.6	0.563
100k	35.6	0.557
10k	35.8	0.565
1k	35.4	0.556
100	35.5	0.551
10	34.2	0.452
1	20.1	-0.622
Table 8: Translation quality in terms of BLEU and COMET on newstest2020 with different weights for the positive class. auto  $x$  refers to setting the weight according to the ratio of the negative class to the positive class with a factor  $x$ .
counteracted by using a large value for the positive weight  $\lambda_{p}$ .
Instead of setting  $\lambda_{p}$  to a fixed weight one can also define it as
$$
\lambda_ {p} = x \frac {n _ {n}}{n _ {p}}
$$
with  $n_p$  as the number of unique target words,  $n_n = V - n_p$  as the number of remaining words and  $x$  being a factor to increase the bias towards recall. This way the positive class and negative class
are weighted equally. Table 8 shows the result of different positive weights, including the automatic setting according to the ratio (auto). We see that not increasing the weight of the positive class results in large quality drops. For positive weights  $>1000$  the quality differences are small. The auto setting provides an alternative that is easier to set than finding a fixed positive weight.
C Additional vocabulary size vs. recall plots
Figures 6 and 7 provide results for the proverbs and ITDS test sets, respectively. We see the same trend across all test sets of NVS offering higher recall at the same vocabulary size compared to alignment-based vocabulary selection. For the proverbs test set this is true both for the literal and the idiomatic translations.
Figure 6: Vocabulary size (speed) vs. recall of reference tokens (quality) for proverbs test set. For NVS, the values correspond to setting  $\lambda \in [0.99, 0.9, 0.5, 0.1, 0.01, \dots, 0.000001]$ . For align, the values correspond to setting  $k \in [100, 200, \dots, 1000, 2000, \dots, 10000]$ .
Figure 7: Vocabulary size (speed) vs. recall of reference tokens (quality) for ITDS test set. For NVS, the values correspond to setting  $\lambda \in [0.99, 0.9, 0.5, 0.1, 0.01, \dots, 0.000001]$ . For align, the values correspond to setting  $k \in [100, 200, \dots, 1000, 2000, \dots, 10000]$ .