SlowGuess/NAACL_2021_OCR huggingface dataset:
H. J. Scudder. 1965. Probability of error of some adaptive pattern-recognition machines. IEEE Trans. Information Theory, 11(3):363-371.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86-96, Berlin, Germany.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and TieYan Liu. 2019. Mass: Masked sequence to sequence pre-training for language generation. In Proceedings of the 36th International Conference on Machine Learning, pages 5926-5936, Long Beach, California, USA.
Haipeng Sun, Rui Wang, Kehai Chen, Masao Utiyama, Eiichiro Sumita, and Tiejun Zhao. 2019. Unsupervised bilingual word embedding agreement for unsupervised neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1235-1245, Florence, Italy.
Haipeng Sun, Rui Wang, Kehai Chen, Masao Utiyama, Eiichiro Sumita, and Tiejun Zhao. 2020a. Knowledge distillation for multilingual unsupervised neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3525-3535, Online.
Haipeng Sun, Rui Wang, Kehai Chen, Masao Utiyama, Eiichiro Sumita, and Tiejun Zhao. 2020b. Unsupervised neural machine translation with cross-lingual language representation agreement. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:1170-1182.
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. 2010. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. Journal of Machine Learning Research, 11:3371-3408.
Jiawei Wu, Xin Wang, and William Yang Wang. 2019. Extract and edit: An alternative to back-translation for unsupervised neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1173-1183, Minneapolis, Minnesota, USA.
Zhen Yang, Wei Chen, Feng Wang, and Bo Xu. 2018. Unsupervised neural machine translation with
weight sharing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 46-55, Melbourne, Australia.
David Yarowsky. 1995. Unsupervised word sense disambiguation rivaling supervised methods. In 33rd Annual Meeting of the Association for Computational Linguistics, pages 189-196, Cambridge, Massachusetts, USA.
Jiajun Zhang and Chengqing Zong. 2016. Exploiting source-side monolingual data in neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1535-1545, Austin, Texas, USA.
A.1 Quantity Ratio Analysis
We investigated the effect of quantity ratio  $\epsilon$  on UNMT performance for the En-Fr translation task during the first epoch of our proposed self-training methods. As shown in Fig. 1,  $\epsilon$  ranging from  $1\%$  to  $100\%$  all enhanced UNMT performance and the performance was similar when the quantity ratio  $\epsilon$  was greater than  $10\%$ . The UNMT model converged faster with less data. Therefore, we selected  $10\%$  as the quantity ratio  $\epsilon$  for our proposed self-training methods.
improvement decreases as additional epochs are added. Considering the computational cost of synthetic data generation, we trained the UNMT model for only two epochs.
Figure 1: Effect of the quantity ratio  $\epsilon$  on UNMT performance for the En-Fr translation tasks.
A.2 Epoch Number Analysis
epoch size
epoch size
Figure 2: Effect of the number of epochs on UNMT performance for the En-Fr and En-Et translation tasks.
In Figure 2, we empirically demonstrate how the number of epochs affects the UNMT performance on the En-Fr and En-Et translation tasks. We found that the use of additional epochs has little influence on the baseline system. In contrast, increasing the number of epochs for our proposed strategies can improve performance because the quality of the synthetic data used by the UNMT model is better after more epochs; however, the