SlowGuess/NAACL_2021_OCR huggingface dataset:
Kenneth Rose. 1998. Deterministic annealing for clustering, compression, classification, regression and related optimization problems. IEEE, 86(11).
Chinnadhurai Sankar and Sujith Ravi. 2019. Deep reinforcement learning for modeling chit-chat dialog with discrete attributes. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, pages 1-10, Stockholm, Sweden. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), Berlin, Germany.
Amr Sharaf and Hal Daumé III. 2017. Structured prediction via learning to search under bandit feedback. In Proceedings of the 2nd Workshop on Structured Prediction for Natural Language Processing, pages 17-26, Copenhagen, Denmark. Association for Computational Linguistics.
Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Minimum risk training for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1683-1692, Berlin, Germany. Association for Computational Linguistics.
Artem Sokolov, Julia Kreutzer, Stefan Riezler, and Christopher Lo. 2016. Stochastic structured prediction under bandit feedback. In Advances in Neural Information Processing Systems (NeurIPS), Barcelona, Spain.
Artem Sokolov, Julia Kreutzer, Kellen Sunderland, Pavel Danchenko, Witold Szymaniak, Hagen Fürstenau, and Stefan Riezler. 2017. A shared task on bandit learning for machine translation. In Proceedings of the Second Conference on Machine Translation (WMT), Copenhagen, Denmark.
Richard S. Sutton and Andrew G. Barto. 1998. Reinforcement Learning: An Introduction. MIT Press.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Chaojun Wang and Rico Sennrich. 2020. On exposure bias, hallucination and domain shift in neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3544-3552, Online. Association for Computational Linguistics.
John Wieting, Taylor Berg-Kirkpatrick, Kevin Gimpel, and Graham Neubig. 2019. Beyond BLEU:training neural machine translation with semantic similarity. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages
4344-4355, Florence, Italy. Association for Computational Linguistics.
Ronald J. Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn., 8(3-4):229-256.
Lijun Wu, Fei Tian, Tao Qin, Jianhuang Lai, and TieYan Liu. 2018. A study of reinforcement learning for neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3612-3621, Brussels, Belgium. Association for Computational Linguistics.
Yilin Yang, Liang Huang, and Mingbo Ma. 2018. Breaking the beam search curse: A study of (re-)scoring methods and stopping criteria for neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3054-3059, Brussels, Belgium. Association for Computational Linguistics.
Adams Wei Yu, Hongrae Lee, and Quoc Le. 2017. Learning to skim text. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1880-1890, Vancouver, Canada. Association for Computational Linguistics.
Domain	Train	Dev	Test
WMT15	3898886	8496	2902
IWSLT14	159392	7245	6750
Table 3 lists the sizes of data split for the parallel datasets from WMT15 (Bojar et al., 2015) and IWSLT $^{14^3}$  used in the experiments. The two datasets are preprocessed using scripts from the Moses toolkit. $^{4}$  The preprocessing pipeline contains the following steps:
- Tokenization with tokenizer.perl
- Lowercasing with lowercase.perl
- Filtering using clean-corpus-n.perl. Sentences with more than 80 words are removed from the dataset
Additionally, we applied Byte-Pair-Encoding (Sennrich et al., 2016) using subword-nmt<sup>5</sup> to create subword units.
B Model Configurations
Table 8 contains the hyperparameters as Joey NMT configurations for the pretrained models, Table 9 the modified hyperparameters for PG, and Table 10 the modified hyperparameters for MRT. Random seeds for the three runs were set to 42, 8 and 64.
C Sample Efficiency
Fig. 2 shows that both MRT and Policy Gradient need a comparable amount of steps (with a batch size of 256 tokens) to reach their optimum. However, the performance of MRT is more stable over the course of training, while Policy Gradient shows higher variance. MRT learns from  $n = 5$  outputs and rewards per step (compared to Policy Gradient with  $n = 1$ ), which stabilizes the updates.
Figure 2: Dev BLEU/steps for cross-domain.
Table 3: Training, dev, and test sizes for WMT15 and IWSLT14 de-en data in number of sentences.
Model	Dev BLEU	Test BLEU
Pretraining	34.26	34.12
PG	34.37 ± 0.12	34.24 ± 0.04
PG + average bl	34.75 ± 0.06	34.40 ± 0.03
PG + τ = 1.2	34.46 ± 0.10	34.15 ± 0.02
PG + τ = 0.8	34.47 ± 0.12	34.26 ± 0.11
PG + constant	34.26 ± 0	34.13 ± 0.01
PG + scaled	34.55 ± 0.10	34.30 ± 0.10
Table 4: PG variants in-domain adaptation (IWSLT14), beam size=5.
D Additional Considerations
Learned Baseline A reward baseline can also be learned by formulating it as a regression problem, but like Wu et al. (2018) we found no empirical gains, thus excluded it from the experiments reported in this paper.
Scaling Rewards We found that selecting max and min over all previous rewards led to deteriorating BLEU scores. This is why we recompute them for each batch.
Gold Tokens in MRT Shen et al. (2016) add the gold sequence to the sample space. However, Edunov et al. (2018) find that this destabilizes training, so Choshen et al. (2020) and Wang and Sen-rich (2020) choose to omit it, and so do we.
E Development Results
Tables 4 and 5 report results on the development set that were used for tuning the models. They show stable results across different held-out sets.
F Absolute Peakiness
Tables 7 and 6 contain the absolute values for the change peakiness that were used to compute percentages for the main paper results.
3https://sites.google.com/site/ iwsltevaluation2014/mt-track 4https://github.com/moses-smt/ mosesdecoder/tree/master/scripts 5https://github.com/rsennrich/ subword-nmt
Model	Dev BLEU	Test BLEU
Pretraining	18.86	20.35
PG	21.63 ± 0.39	22.62 ± 0.56
PG + average bl	23.31 ± 0.32	24.53 ± 0.26
PG + τ = 1.2	20.45 ± 0.18	21.58 ± 0.21
PG + τ = 0.8	22.16 ± 0.08	23.71 ± 0.27
PG + constant	18.86 ± 0	20.35 ± 0
PG + scaled	22.40 ± 0.10	23.50 ± 0.15
Table 5: PG variants cross-domain adaptation (WMT15 to IWSLT14), beam size=5
Model	Δtop10	Δmode	Δgold
PG	0.294±0.001	0.400±0.001	0.095±0.002
PG + scaled	0.290±0.002	0.397±0.006	0.098±0.001
PG + average bl	0.300±0.001	0.428±0.001	0.103±0.002
PG + τ = 1.2	0.260±0.003	0.357±0.006	0.082±0.001
PG + τ = 0.8	0.312±0.001	0.428±0.002	0.102±0.001
PG + constant	0.323±0.001	0.521±0.002	-0.123±0.004
MRT	0.307±0.002	0.482±0.004	0.126±0.004
Table 6: Absolute changes in peakiness after cross-domain adaptation (WMT15 to IWSLT14).
Model	Δtop10	Δmode	Δgold
PG	0.099±0.002	0.199±0.003	0.076±0.001
PG + scaled	0.097±0.002	0.162±0.005	0.074±0.002
PG + average bl	0.108±0.002	0.185±0.006	0.083±0.002
PG + τ = 1.2	0.052±0.002	0.112±0.004	0.060±0.001
PG + τ = 0.8	0.123±0.001	0.198±0.003	0.083±0.003
PG + constant	0.012±0.007	0.017±0.007	0.003±0.001
MRT	0.112±0.003	0.217±0.004	0.090±0.004
Table 7: Absolute changes in peakiness for in-domain (IWSLT14) adaptation.
Model	IWSLT14	WMT15
Parameter	Setting	Setting
initializer	"xavier"	"xavier"
embed initializer	"xavier"	"xavier"
embed init gain	1.0	1.0
init gain	1.0	1.0
bias initializer	"zeros"	"zeros"
tied embeddings	True	True
tied softmax	True	True
encoder type	Transformer	Transformer
encoder embeddings dim	256	128
encoder hidden size	256	128
encoder dropout	0.3	0.3
encoder num layers	6	6
encoder num heads	4	4
encoder ff_size	1024	512
decoder type	Transformer	Transformer
decoder embeddings dim	256	128
decoder hidden size	256	128
decoder dropout	0.3	0.3
decoder num layers	6	6
decoder num heads	4	4
decoder ff_size	1024	512
optimizer	"adam"	"adam"
normalization	"tokens"	"tokens"
adam_betas	[0.9, 0.999]	[0.9, 0.999]
scheduling	"plateau"	"plateau"
patience	5	5
decrease_factor	0.7	0.7
loss	"crossentropy"	"crossentropy"
learning rate	0.0003	0.0003
learning rate_min	0.00000002	0.00000002
weight decay	0.0	0.0
label smoothing	0.1	0.1
batch size	2048	4096
batch type	"token"	"token"
epochs	100	100
Table 8: Pretraining model parameters
Parameter	In-domain	Cross-domain
learning rate	0.00001	0.0001
batch size	128	256
Table 9: Policy Gradient parameters
Parameter	In-domain	Cross-domain
learning rate	0.00001	0.0001
batch size	32	64
batch multiplier	4	4
eval batch size	128	128
samples	5	5
alpha	0.005	0.005
Table 10: MRT parameters