SlowGuess/NAACL_2022_OCR huggingface dataset:
Our baseline results are summarized in Table 2. The evaluation metric used in all experiments is BLEU (Papineni et al., 2002) computed by Sacre-BLEU (Post, 2018). As expected, models work accurately on in-domain data but perform poorly on other domains, specially if the domain is significantly different from training samples, e.g. the PHP model's BLEU score is zero when translating WMT test sentences.
In order to remedy the poor quality for out-domain data, we train a central model and adapt it to all other domains with two techniques of data combination and chained fine-tuning. In data combination, we simply concatenate all corpora to create a much larger training set. We initialize the central model with WMT parameters and retrain it for extra 50K steps with the new dataset. In chained fine-tuning, we do not combine datasets but instead fine-tune the central model sequentially using each domain's training set for additional 50K
$^{2}$ http://statmt.org/wmt14/  
translation-task.html  
 $^{3}$ https://github.com/moses-smt/  
mosesdecoder  
 $^{4}$ The same sets used in Roosta et al. (2021)
<sup>5</sup>https://github.com/mjpost/sacrebleu
WMT	OS	TED	PHP	UB
WMT	33.66	18.57	29.22	8.04	12.41
OS	13.66	23.58	24.22	7.84	13.83
TED	12.09	13.59	29.32	6.67	10.15
PHP	0.00	0.26	0.26	34.48	0.00
UB	0.28	0.78	0.75	2.30	30.15
Table 2: Baseline results for the De→En direction. Models are trained for 100K steps. The first column indicates which training set is used to train the engine and other columns show models' performance on different test sets, e.g.  $[OS][TED] = 24.22$  indicates the BLEU score of a model trained on the OS training set and tested on the TED test set. The best BLEU score for each test set is bold-faced.
steps (10K for each), i.e. we start from the WMT model, then sequentially fine-tune it over UB, OS, and other datasets one after another. This sort of fine-tuning could suffer from catastrophic forgetting, so we ran different experiments to figure out the best order of fine-tuning.
The chained fine-tuning strategy could provide relevant baselines for FL experiments. Imagine a scenario where the central model is shipped to a client environment and it is updated there with multiple local datasets. In data combination, we assume that all data is accessible at training time (a fully-observable environment with full access to all domains' data) whereas chained fine-tuning pictures a more realistic scenario by forcing to update the central model gradually on the client side.
Table 3 summarizes results for these two finetuning methods. Exposing the central model to other domains' data yields much better quality. Data combination clearly outperforms and it shows the impact of having direct access to data; the privilege that we do not have in settings such as chained training and FL.
Technique	WMT	OS	TED	PHP	UB
chained	18.26	23.51	28.19	16.14	23.05
combination	33.50	21.82	31.51	37.56	35.61
Table 3: Domain adaptation results in centralized settings.
5.3 Federated Learning Results
Models reported in the previous section (specially in Table 3) are high-quality engines that are trained
in a centralized fashion and provide acceptable performance for all domains. Fine-tuning addressed the problem of poor quality for out-domain data, but as discussed previously, centralized fine-tuning and having access to out-domain data might not be always possible. Therefor, in this section we try to train comparable alternatives in an FL setting.
Our setting has one server and five clients (one for each dataset). In the interest of fair comparison between the FL and centralized approaches, we initialize all the clients with WMT parameters. Each client updates its model with local data and shares it with the server in each round. We Pull client updates after 10K steps of fine-tuning for aggregation and repeat this process for 5 rounds. In total, each model is fine-tuned for 50K steps which is identical to the setting we used for centralized training. Results for this experiment are summarized in Table 4.
WMT	OS	TED	PHP	UB
S	33.97	19.17	30.8	37.32	47.9
WMTc	32.07	18.28	29.55	9.55	13.75
OSc	19.05	23.39	27.85	13.57	18.58
TEDc	17.37	16.05	34.30	11.83	17.05
PHPc	4.07	4.33	7.48	45.07	10.90
UBc	0.77	4.27	5.66	14.98	49.51
Table 4: FL results with 1 server  $(S)$  and 5 clients (indicated with the  $c$  superscript), e.g.  $OS^c$  is a client initialized with WMT parameters and updated with its own data (OS training samples).
The first row in Table 4 belongs to the server. FL affects model training quite positively and provides significantly better BLEU scores, specially for those low-performing models such as PHP. The average BLEU score of the server over different domains is 33.83, which is 1.83 points higher than that of the best model reported in Table 3. This means, even though FL does not access clients' data, it is more impactful in fusing information and training mixed-domain engines. This outcome for a complex task such as NMT was unexpected.
After the final FL round, the server parameters are pushed back to clients so they can also benefit from the result of aggregation. At this point, each client can decide to run another phase of fine-tuning with local data over the server parameters. This is a trade-off between being domain specific and remaining generic. Results for this process
are listed from the second to last rows in Table 4, e.g.  $PHP^{c}$  after the last Push step has access to all server parameters so its BLEU score on in-domain data is 37.32, similar to that of the server. It is also able to translate other domains with the average BLEU score of 32.96. However, as the fifth row shows, if it decides to fine-tune the last parameter set it received from the server with its own local data, its BLEU score increases from 37.32 to 45.07 on the in-domain test set, but at the same time it loses its generalization over other domains.
We also ran an ablation study to evaluate how the number of FL rounds impact model quality. Two important factors in FL settings are the number of clients and training rounds. We can pass over the first one as we have a cross-silo setting with a limited number of clients, but Table 5 and Figure 3 provide additional information on the second hyperparameter.
WMT	OS	TED	PHP	UB
S5	33.97	19.17	30.8	37.32	47.9
S10	31.90	19.63	31.04	38.33	48.63
S50	31.05	20.83	32.27	43.99	51.07
Figure 3: The learning curve of the UB model for rounds  $\in \{5,10,50\}$ .
Results from the figure/table above show that the number of rounds and model performance increase proportionally in low-quality clients such as UB or PHP. This was expected since with a higher number of rounds clients are updated more frequently with rich information from the central server. However, it comes at a price as it increases communication load. It also delays local model
training, because in each round the client has to suspend training to read server values and updates its internal model. For other high-quality clients such as WMT, higher rounds lead to some degradation since they receive external updates from less-accurate peers and have to compensate for their low quality. The choice of the number of rounds is a trade-off between quality and bandwidth.
5.4 Dynamic Pulling Results
We proposed a novel technique for better pulling and mentioned that it is able to reduce the communication load yet maintain model quality. Table 6 reports related results to support our claims.
Table 5: FL results for rounds  $\in$  {5,10,50}.rounds = 50 means aggregation occurs 50 times between checkpoints  $100\mathrm{K}$  (where FL training starts) and  $150\mathrm{K}$  (where FL training ends).
WMT	OS	TED	PHP	UB
default	33.97	19.17	30.8	37.32	47.90
DPc l	29.28	19.17	30.88	36.33	45.74
DPc g	30.74	18.28	27.61	13.69	32.16
random	24.58	19.16	30.57	35.33	42.50
Table 6: The impact of different communication techniques on model quality. The first row is copied from Table 4 for easier comparison. The bold-faced numbers are the best results obtained by DP-based techniques.
The average BLEU scores for the default,  $DP_{l}^{c}$ , and  $DP_{g}^{c}$  methods on different domains are 33.83, 32.28, and 24.49, respectively. This means the assumption that less active tensors are responsible for domain adaption could be true and highly-fluctuating tensors should only be kept for learning in-domain knowledge. We also provide results from our random configuration, in which we exchange the same number of parameters as in  $DP_{l}^{c}$  and  $DP_{g}^{c}$  but those parameters are selected randomly. The comparison between random and other alternatives shows that the selection criterion directly affects model quality.
Although there is a gap by 1.55 points between default and  $DP_{l}^{c}$  (which is meaningful in NMT),  $DP_{l}^{c}$  could still be a strong candidate when training large models in the context of FL, because the number of parameters exchanged in each pulling step is 45,724,160 for default whereas this number is only 22,863,104 (50% less) for  $DP_{l}^{c}$ . It should also be noted that pulling occurs not once but for multiple rounds and saving 50% each time is a significant gain. Moreover,  $DP_{l}^{c}$  still performs on par with data combination which is a strong but centralized and not private baseline.
5.5 Comparison to Controllers
As previously discussed in Section 3.1, Controllers (Roosta et al., 2021) only exchange 4 layers (2 from the encoder and 2 from the decoder in a 12-layer Transformer) with the server in the Push and Pull phases, which means the communication bandwidth they reduce is around  $66\%$  ( $\frac{4}{12} \approx 33\%$  of the layers are only exchanged between the server and clients). In our case,  $DP$  only affects Pull which leads to  $25\%$  bandwidth reduction in client-to-server exchanges. $^6$
We challenged our model to see if we can also save  $66\%$  in bandwidth by sending/receiving the same number of parameters as in Controllers. More precisely, we applied our threshold-based strategy in both Push and Pull and modified the value of the threshold such that it only accepts  $33\%$  of the parameters to exchange with the server. Table 7 summarizes results of this experiment.
WMT	OS	TED	PHP	UB
DPc l	29.52	19.25	31.53	36.19	45.40
DPc g	29.09	19.07	30.93	34.47	41.57
6E6D (0-3)	31.13	19.19	30.95	33.79	32.85
8E8D (0-6)	31.79	20.02	30.6	32.43	33.41
Table 7: Dynamic parameter selection versus Controllers.
In our comparison, we selected the two best performing Controller models reported in Roosta et al. (2021). The 6E6D (0-3) configuration is a Transformer with a 6-layer encoder and 6-layer decoder whose first and fourth layers are selected to act as Controllers. In the 8E8D (0-6) configuration, instead of using the original encoder/decode layers as Controllers four additional layers (two for the encoder and two for the decoder) are defined which are placed after layers 0 and 6. This means, in an 8-layer encoder/decoder the first and seventh layers are Controllers and the rest are ordinary layers.
Results show that our model could be a reliable alternative for communication-efficient FL, even though we aggressively limited the number of parameters exchanged in this new setting. Moreover, in our model we do not need to define additional layers. Unlike Controllers, we also do not have to deal with finding the correct position to place Con
troller layers. According to Roosta et al. (2021), the final model performance is directly impacted by misplacing Controllers and our solution solves that problem.
In this paper, we reported a set of benchmarking result for NMT in the context of FL. We also proposed an effective technique to reduce the communication bandwidth. Our solution tries to determine a subset of parameters that are responsible for learning out-domain knowledge and only exchanges them with the server.
In future work, we are interested in  $i)$  adding more languages to train multilingual engines,  $ii)$  improving communication protocols even further,  $iii)$  comparing other FL algorithms in the presence of  $DP$ , and  $iv)$  investigating NMT in cross-device settings.
We would like to thank our anonymous reviewers for their valuable comments as well as Liling Tan, Grant Strimel, and Sriram Venkatapathy (from Amazon) for providing feedback on the first version of this paper.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
Ankur Bapna, Naveen Arivazhagan, and Orhan First. 2019. Simple, scalable adaptation for neural machine translation. arXiv preprint arXiv:1909.08478.
Duc Bui, Kshitiz Malik, Jack Goetz, Honglei Liu, Seungwhan Moon, Anuj Kumar, and Kang G Shin. 2019. Federated user representation learning. arXiv preprint arXiv:1909.12535.
Mingqing Chen, Rajiv Mathews, Tom Ouyang, and Françoise Beaufays. 2019. Federated learning of out-of-vocabulary words. arXiv preprint arXiv:1903.10635.
Chenhui Chu and Rui Wang. 2018. A survey of domain adaptation for neural machine translation. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1304-1319, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep
${}^{6}$  Pull is only 50% of the communication where we excluding half the parameters using the threshold,so  ${50}\%  \times$ ${50}\%  = {25}\%$  .
bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Suyu Ge, Fangzhao Wu, Chuhan Wu, Tao Qi, Yongfeng Huang, and Xing Xie. 2020. Fedner: Privacy-preserving medical named entity recognition with federated learning. arXiv e-prints, pages arXiv-2003.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional sequence to sequence learning. In International Conference on Machine Learning, pages 1243-1252. PMLR.
Jonas Geiping, Hartmut Bauermeister, Hannah Droge, and Michael Moeller. 2020. Inverting gradients—how easy is it to break privacy in federated learning? arXiv preprint arXiv:2003.14053.
Robin C Geyer, Tassilo Klein, and Moin Nabi. 2017. Differentially private federated learning: A client level perspective. arXiv preprint arXiv:1712.07557.
Andrew Hard, K. Rao, Rajiv Mathews, F. Beaufays, S. Augenstein, Hubert Eichner, Chloeé Kiddon, and D. Ramage. 2018. Federated learning for mobile keyboard prediction. ArXiv, abs/1811.03604.
Corentin Hardy, Erwan Le Merrer, and Bruno Sericola. 2019. Md-gan: Multi-discriminator generative adversarial networks for distributed datasets. In 2019 IEEE international parallel and distributed processing symposium (IPDPS), pages 866-877. IEEE.
Florian Hartmann, Sunah Suh, Arkadiusz Komarzewski, Tim D Smith, and Ilana Segall. 2019. Federated learning for ranking browser history suggestions. arXiv preprint arXiv:1911.11807.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pages 2790-2799. PMLR.
Shaoxiong Ji, Wenqi Jiang, Anwar Walid, and Xue Li. 2020. Dynamic sampling and selective masking for communication-efficient federated learning. arXiv preprint arXiv:2003.09603.
Shaoxiong Ji, Shirui Pan, Guodong Long, Xue Li, Jing Jiang, and Zi Huang. 2019. Learning private neural language modeling with attentive aggregation. In 2019 International Joint Conference on Neural Networks (IJCNN), pages 1-8. IEEE.
Heejae Kim, Taewoo Kim, and Chan-Hyun Youn. 2020. On federated learning of deep networks from non{iid} data: Parameter divergence and the effects of hyperparametric methods.
David Leroy, Alice Coucke, Thibaut Lavril, Thibault Gisselbrecht, and Joseph Dureau. 2019. Federated learning for keyword spotting. In ICASSP 2019-2019
IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6341-6345. IEEE.
Qinbin Li, Yiqun Diao, Quan Chen, and Bingsheng He. 2021. Federated learning on non-iid data silos: An experimental study. arXiv preprint arXiv:2102.02079.
Qinbin Li, Zeyi Wen, Zhaomin Wu, Sixu Hu, Naibo Wang, Yuan Li, Xu Liu, and Bingsheng He. 2019. A survey on federated learning systems: vision, hype and reality for data privacy and protection. arXiv preprint arXiv:1907.09693.
Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. 2020. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine, 37(3):50-60.
Yujun Lin, Song Han, Huizi Mao, Yu Wang, and William J Dally. 2018. Deep gradient compression: Reducing the communication bandwidth for distributed training. ICLR.
Pierre Lison and Jörg Tiedemann. 2016. OpenSubtitles2016: Extracting large parallel corpora from movie and TV subtitles. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16).
Dianbo Liu and Tim Miller. 2020. Federated pretraining and fine tuning of bert using clinical notes from multiple silos. arXiv preprint arXiv:2002.08562.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. 2017. Communication-efficient learning of deep networks from decentralized data. In Artificial Intelligence and Statistics, pages 1273-1282. PMLR.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318.
Peyman Passban, Puneeth SM Saladi, and Qun Liu. 2020. Revisiting robust neural machine translation: A transformer case study. arXiv preprint arXiv:2012.15710.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proc. of NAACL.
Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. 2020. Adapterhub: A framework for adapting transformers. arXiv preprint arXiv:2007.07779.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on
Machine Translation: Research Papers, pages 186-191, Belgium, Brussels. Association for Computational Linguistics.
Hanchi Ren, J. Deng, and Xianghua Xie. 2021. Grnn: Generative regression neural network - a data leakage attack for federated learning.
Tanya Roosta, Peyman Passban, and Ankit R. Chadha. 2021. Communication-efficient federated learning for neural machine translation. ArXiv, abs/2112.06135.
Andreas Rückle, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. 2020. Adapterdrop: On the efficiency of adapters in transformers. arXiv preprint arXiv:2010.11918.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany. Association for Computational Linguistics.
Joel Stremmel and Arjun Singh. 2020. Pretraining federated text models for next word prediction. arXiv e-prints, pages arXiv-2005.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. arXiv preprint arXiv:1409.3215.
Jörg Tiedemann. 2012. Parallel data, tools and interfaces in opus. In Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC'12), Istanbul, Turkey. European Language Resources Association (ELRA).
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. arXiv preprint arXiv:1706.03762.
Su Wang, Mengyuan Lee, Seyyedali Hosseinalipour, Roberto Morabito, Mung Chiang, and Christopher G Brinton. 2021. Device sampling for heterogeneous federated learning: Theory, algorithms, and implementation. arXiv preprint arXiv:2101.00787.
Xing Wu, Zhaowang Liang, and Jianjia Wang. 2020. Fedmed: A federated learning framework for language modeling. Sensors, 20(14):4048.
Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. 2019. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 10(2):1-19.
Zhuangdi Zhu, Kaixiang Lin, and Jiayu Zhou. 2020. Transfer learning in deep reinforcement learning: A survey. arXiv preprint arXiv:2009.07888.