SlowGuess/ACL_2020_OCR huggingface dataset:
Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P. Adams, and Nando de Freitas. 2016. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 104:148-175.
Niranjan Srinivas, Andreas Krause, Sham Kakade, and Matthias Seeger. 2010. Gaussian process optimization in the bandit setting: No regret and experimental design. In Proceedings of the 27th International Conference on International Conference on Machine Learning, ICML'10, pages 1015-1022, USA. Omnipress.
Sander Tars and Mark Fishel. 2018. Multi-domain neural machine translation. CoRR, abs/1805.02282.
Brian Thompson, Huda Khayrallah, Antonios Anastasopoulos, Arya D. McCarthy, Kevin Duh, Rebecca Marvin, Paul McNamee, Jeremy Gwinnup, Tim Anderson, and Philipp Koehn. 2018. Freezing subnetworks to analyze domain adaptation in neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 124-132, Brussels, Belgium. Association for Computational Linguistics.
Yulia Tsvetkov, Manaal Faruqui, Wang Ling, Brian MacWhinney, and Chris Dyer. 2016. Learning the curriculum with bayesian optimization for task-specific word representation learning. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 130-139, Berlin, Germany. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998-6008. Curran Associates, Inc.
Rui Wang, Masao Utiyama, Lemao Liu, Kehai Chen, and Eiichiro Sumita. 2017. Instance weighting for neural machine translation domain adaptation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1482-1488. Association for Computational Linguistics.
Rui Wang, Masao Utiyama, and Eiichiro Sumita. 2018a. Dynamic sentence sampling for efficient training of neural machine translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 298-304, Melbourne, Australia. Association for Computational Linguistics.
Wei Wang, Isaac Caswell, and Ciprian Chelba. 2019a. Dynamically composing domain-data selection with clean-data selection by "co-curricular learning" for neural machine translation. In Proceedings of the
7721
57th Conference of the Association for Computational Linguistics, pages 1282-1292, Florence, Italy. Association for Computational Linguistics.
Wei Wang, Taro Watanabe, Macduff Hughes, Tetsuji Nakagawa, and Ciprian Chelba. 2018b. Denoising neural machine translation training with trusted data and online data selection. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 133-143. Association for Computational Linguistics.
Xinyi Wang, Hieu Pham, Paul Michel, Antonios Anastasopoulos, Graham Neubig, and Jaime Carbonell. 2019b. Optimizing data usage via differentiable rewards. arXiv preprint arXiv:1911.10088.
Xinyi Wang, Yulia Tsvetkov, and Graham Neubig. 2020. Balancing training for multilingual neural machine translation. arXiv preprint arXiv:2004.06748.
Marlies van der Wees, Arianna Bisazza, and Christof Monz. 2017. Dynamic data selection for neural machine transaltion. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1400-1410.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyls, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144.
Yinfei Yang, Gustavo Hernandez Abrego, Steve Yuan, Mandy Guo, Qinlan Shen, Daniel Cer, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2019. Improving multilingual sentence embedding using bidirectional dual encoder with additive margin softmax. CoRR, abs/1902.08564.
Xuan Zhang, Gaurav Kumar, Huda Khayrallah, Kenton Murray, Jeremy G Winnup, Marianna J. Martindale, Paul McNamee, Kevin Duh, and Marine Carpuat. 2018. An empirical exploration of curriculum learning for neural machine translation. CoRR, abs/1811.00739.
Xuan Zhang, Pamela Shapiro, Gaurav Kumar, Paul McNamee, Marine Carpuat, and Kevin Duh. 2019. Curriculum learning for domain adaptation in neural machine translation. In 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics.
A Cross-lingual Embedding Model Parameters
The sentence encoder has a shared 200k token multilingual vocabulary with 10k OOV buckets. For each token, we also extract character n-grams  $(n = [3,6])$  hashed to 200k buckets. Word token items and character n-gram items are mapped to 320 dim. character embeddings. Word and character n-gram representations are summed together to produce the final input token representation. The encoder is a 3-layer Transformer with hidden size of 512, filter size of 2048, and 8 attention heads. We train for 40M steps using an SGD optimizer with batch size  $K = 100$  and learning rate 0.003. During training, the word and character embeddings are scaled by a gradient multiplier of 25.
B Transformer-Big Results
We replicate experiments with the Transformer-Big architecture. Table 11 shows the Transformer-Big results that correspond to the RNN results in Table 3. These results show that the multi-domain curriculum meets the performance requirement (i) (Section 4.1) using the Transformer architecture. Table 12 shows the Transformer-Big results corresponding to RNN results in Table 4. They show that the proposed multi-domain curriculum meets the performance requirement (ii) using Transformer.
Curri.	N14	T15	PA	D15	Avg
P1: B	34.1	36.3	34.2	32.3	34.2
P2: C6-feats	39.6	40.2	50.6	37.7	42.0
W1: B	40.8	39.9	46.0	37.8	41.1
W2: C6-feats	41.8	41.2	48.1	38.8	42.5
Table 11: Transformer Big SacreBLEU: English  $\rightarrow$  French multi-domain curriculum improves no-curriculum baseline (B) over all testsets, using Transformer-Big. P: Paracrawl training data. W: WMT14 training data.
C An Explanation: Noisy Data Useful in Low-Resource Setup
With noisy, limited data (e.g., 100k pairs), we can train a model A on all data, or a model B on the filtered subset (e.g., 10k). We can also fine-tune A on the filtered data, to produce model C. C could be better than A due to use of higher-quality data or better than B due to use of more
7722
Curri.	N14	T15	PA	D15	Avg
P1: B	34.1	36.3	34.2	32.3	34.2
P3: CdN	33.7	36.1	32.7	32.5	33.8
P4: CdT	35.3	37.7	32.8	34.0	35.0
P5: CBERT	39.2	40.1	49.7	37.5	41.6
P6: CqN	38.9	39.8	48.9	36.9	41.1
P7: CqT	37.3	40.4	44.7	36.2	39.7
P2: C6-feats	39.6	40.2	50.6	37.7	42.0
P2-P*	+0.4	-0.2	+0.9	+0.2	+0.3
W1: B	40.8	39.9	46.0	37.8	41.1
W3: CdN	40.9	39.2	44.4	37.6	40.5
W4: CdT	39.8	39.6	43.3	37.3	40.0
W5: CBERT	40.5	39.2	45.7	38.3	40.9
W6: CqN	41.1	40.0	47.6	38.0	41.7
W7: CqT	41.1	41.4	47.7	38.5	42.2
W2: C6-feats	41.8	41.2	48.1	38.8	42.5
W2-W*	+0.7	-0.2	+0.4	+0.3	+0.3
Table 12: Transformer Big SacreBLEU: English  $\rightarrow$  French multi-domain curriculum (P2, W2) vs. single-domain curricula (P3-7, W3-7). BLEU scores over 4 testsets and their average. Frame boxes mark best pertestset BLEU  $(\mathrm{W}^{*},\mathrm{P}^{*})$  over all single-domain curricula. Bold color denotes multi-domain curriculum has best BLEU  $(\mathrm{W2 - W^{*}}\geq 0)$  . P: ParaCrawl training data. W: WMT14 training data.