SlowGuess/NAACL_2022_OCR huggingface dataset:
Philipp Koehn, Graham Neubig, Juan Pino, and Hassan Sajjad. 2019. Findings of the first shared task on machine translation robustness. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 91-102, Florence, Italy. Association for Computational Linguistics.
Xuechen Li, Florian Tramer, Percy Liang, and Tatsunori Hashimoto. 2021. Large language models can be strong differentially private learners. arXiv preprint arXiv:2110.05679.
Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Xiaodong Fan, Bruce Zhang, Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Winnie Wu, Shuguang Liu, Fan Yang, Daniel Fernando Campos, Rangan Majumder, and Ming Zhou. 2020. Xglue: A new benchmark dataset for cross-lingual pre-training, understanding and generation. In EMNLP.
Bill Yuchen Lin, Chaoyang He, ZiHang Zeng, Hulin Wang, Yufen Huang, M. Soltanolkotabi, Xiang Ren, and S. Avestimehr. 2021. Fednlp: A research platform for federated learning in natural language processing. In arXiv.cs.CL 2104.08815.
Dianbo Liu and Tim Miller. 2020. Federated pretraining and fine tuning of bert using clinical notes from multiple silos. ArXiv, abs/2002.08562.
Ming Liu, Stella Ho, Mengqi Wang, Longxiang Gao, Yuan Jin, and He Zhang. 2021. Federated learning meets natural language processing: A survey. arXiv preprint arXiv:2107.12603.
Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. 2017. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR.
Paul Michel and Graham Neubig. 2018. Mtnt: A testbed for machine translation of noisy text. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.
Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow. 2020. On the stability of fine-tuning bert: Misconceptions, explanations, and strong baselines. In International Conference on Learning Representations.
Toan Q Nguyen and Julian Salazar. 2019. Transformers without tears: Improving the normalization of self-attention. arXiv preprint arXiv:1910.05895.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the
40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc.
Matt Post. 2018. A call for clarity in reporting bleu scores. arXiv preprint arXiv:1804.08771.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683.
Swaroop Ramaswamy, Rajiv Mathews, Kanishka Rao, and Françoise Beaufays. 2019. Federated learning for emoji prediction in a mobile keyboard. arXiv preprint arXiv:1906.04329.
Victor Sanh, Lysandre Debut, Julien Chaumont, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108.
Sebastian Urban Stich. 2019. Local sgd converges fast and communicates little. In ICLR 2019-International Conference on Learning Representations, CONF.
Asa Cooper Stickland, Xian Li, and Marjan Ghazvininejad. 2020. Recipes for adapting pre-trained monolingual and multilingual models to machine translation. arXiv preprint arXiv:2004.14911.
Joel Stremmel and Arjun Singh. 2020. Pretraining federated text models for next word prediction. arXiv preprint arXiv:2005.04828.
Dianbo Sui, Yubo Chen, Jun Zhao, Yantao Jia, Yuantao Xie, and Weijian Sun. 2020. Feded: Federated learning via ensemble distillation for medical relation extraction. In EMNLP.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771.
Chuhan Wu, Fangzhao Wu, Ruixuan Liu, Lingjuan Lyu, Yongfeng Huang, and Xing Xie. 2021. Fedkd: Communication efficient federated learning via knowledge distillation. ArXiv, abs/2108.13323.
Xing Wu, Zhaowang Liang, and Jianjia Wang. 2020. Fedmed: A federated learning framework for language modeling. Sensors, 20(14):4048.
Timothy Yang, Galen Andrew, Hubert Eichner, Haicheng Sun, Wei Li, Nicholas Kong, Daniel Ramage, and Françoise Beaufays. 2018. Applied federated learning: Improving google keyboard query suggestions. arXiv preprint arXiv:1812.02903.
Michal Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. 2016. The united nations parallel corpus v1. 0. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 3530-3534.
Each LM experiment ran for approximately a day each on a 6 GPU cluster of RTX 6000 GPUs with 24GB of memory per GPU. The MT experiments took approximately 12 hours each and the TC experiments took around 3 hours each, all on the same cluster.
We use the AdamW optimizer (Loshchilov and Hutter, 2017; Kingma and Ba, 2014) for all experiments (shown to be effective for FL in Lin et al. 2021). Each client goes through a full epoch of local learning before synchronizing with the server.
For MT, we report results using the 5e-5 learning rate, as we found in initial results (as have others also, see Appendix B of Stickland et al. (2020) as one example) that MT experiments are generally consistent over learning rates when fine-tuning. For language modeling and text classification, we use three different learning rates (1e-4, 5e-5, 1e-5). All models were selected using the best performing version on the validation set, for the given model and training setting. For both tasks, we use early stopping (5 epochs of no improvement for MT and TC, 10 epochs for LM).
We use the standard sacreBLEU settings: nrefs:1, mixed case, eff:no, tok:13a, smooth:exp, and version 2.0.0. For Ja and Zh we use their respective tokenizers.
B Randomly Initialized MT
We do not report results for randomly initialized training of MT systems, as large neural MT systems generally need large amounts of data to be effective. We ran experiments for the MTNT dataset from
random initializations, running for twice as many epochs. Resulting models appeared to converge by loss but had extremely low BLEU scores. Thus, we only include pretrained results in Table 2.
C MTNT Data Preprocessing for M2M-100
M2M-100 was trained using scripts that removed input with "excess punctuation." We follow this in preparing MTNT training data. We use all  $\mathrm{En} \rightarrow$  Ja data (consisting of approximately 6k instances) and take the corresponding  $\mathrm{En} \rightarrow$  Fr instances, randomly sampling additional instances until there are the same number of instances in each direction. We sample an equal number of training instances as we are testing the effects of multilingual data, rather than unequal dataset sizes. We then remove the training instances with excess punctuation (or sentences less than 3 characters) following the M2M-100 script. This leaves 5605 instances in each direction for training. We use the standard MTNT dev and test sets, as-is, consisting of approximately 1k data points.
D Full LM Results
We show the full results of the LM experiments, with standard deviations over five random seeds in Tables 4 and 5.
M	En	Cs	Lt	Es	Europarl
Pl	Fi	Pt	De	Avg
B	26.2 ± 2.4	34.8 ± 1.8	40.1 ± 2.3	20.0 ± 1.3	20.0 ± 1.3	26.6 ± 1.4	25.5 ± 2.0	22.1 ± 1.8	26.9 ± 1.8
C	19.3 ± 1.5	4.5 ± 0.4	3.9 ± 0.3	8.3 ± 0.7	4.7 ± 0.3	4.9 ± 0.3	7.0 ± 0.6	10.8 ± 0.8	7.9 ± 0.6
I	26.6 ± 1.7	5.4 ± 0.4	4.3 ± 0.3	11.2 ± 0.9	5.8 ± 0.4	5.7 ± 0.3	8.9 ± 0.7	15.1 ± 1.1	10.4 ± 0.7
N	50.6 ± 2.9	7.1 ± 0.5	11.9 ± 0.9	16.0 ± 1.2	17.7 ± 1.2	12.1 ± 0.7	35.6 ± 2.8	21.7 ± 1.4	21.6 ± 1.4
C	12.1 ± 0.9	3.7 ± 0.3	3.3 ± 0.2	13.9 ± 1.2	4.7 ± 0.4	4.0 ± 0.2	4.8 ± 0.4	6.8 ± 0.6	6.7 ± 0.5
I	10.5 ± 0.9	4.0 ± 0.3	4.2 ± 0.3	6.1 ± 0.5	3.8 ± 0.3	4.5 ± 0.3	5.6 ± 0.4	6.9 ± 0.5	5.7 ± 0.4
N	8.8 ± 0.6	3.7 ± 0.3	3.9 ± 0.3	6.0 ± 0.5	3.8 ± 0.3	4.4 ± 0.3	5.6 ± 0.4	6.7 ± 0.5	5.4 ± 0.4
Table 4: Results for the LM FL experiments on the Europarl Corpus. Bold scores indicate the best in the column for the given section. Scores are measured in perplexity (lower is better). The top row (B) is a baseline using the pretrained model with no further tuning. The middle rows are trained from randomly-initialized models while the bottom rows tune the pretrained model on task data. Due to space we abbreviate: C for Centralized, I for IID FL, and N for non-IID FL.
M	En	Es	Fr	UN Ru	Zh	Ar	Avg
B	22.3 ± 2.0	15.0 ± 0.9	17.2 ± 1.1	9.8 ± 0.6	18.1 ± 1.0	14.7 ± 0.8	16.2 ± 1.0
C	9.0 ± 0.6	5.2 ± 0.3	8.2 ± 0.5	3.9 ± 0.3	4.3 ± 0.2	4.6 ± 0.3	5.9 ± 0.4
I	9.1 ± 0.7	5.2 ± 0.3	8.4 ± 0.4	3.7 ± 0.3	3.9 ± 0.2	4.5 ± 0.2	5.8 ± 0.3
N	12.8 ± 0.9	11.5 ± 0.7	14.6 ± 0.8	9.3 ± 0.7	8.2 ± 0.5	8.3 ± 0.4	10.8 ± 0.6
C	7.0 ± 0.5	4.1 ± 0.2	4.9 ± 0.3	2.9 ± 0.2	3.3 ± 0.2	3.6 ± 0.2	4.3 ± 0.3
I	6.5 ± 0.5	3.9 ± 0.2	5.7 ± 0.3	2.8 ± 0.2	3.2 ± 0.2	3.5 ± 0.2	4.3 ± 0.3
N	7.1 ± 0.5	4.5 ± 0.3	6.2 ± 0.3	3.2 ± 0.2	4.2 ± 0.2	4.0 ± 0.2	4.9 ± 0.3
Table 5: Results for the LM FL experiments on the UN Corpus. Bold scores indicate the best in the column for the given section. Scores are measured in perplexity (lower is better). The top row (B) is a baseline using the pretrained model with no further tuning. The middle rows are trained from randomly-initialized models while the bottom rows tune the pretrained model on task data. Due to space we abbreviate: C for Centralized, I for IID FL, and N for non-IID FL.