SlowGuess/NAACL_2022_OCR huggingface dataset:
Jinhyun So, Başak Güler, and A Salman Avestimehr. 2021a. Codedprivateml: A fast and privacy-preserving framework for distributed machine learning. IEEE Journal on Selected Areas in Information Theory, (1).
Jinhyun So, Başak Güler, and A Salman Avestimehr. 2021b. Turbo-aggregate: Breaking the quadratic aggregation barrier in secure federated learning. IEEE Journal on Selected Areas in Information Theory, (1).
Joel Stremmel and Arjun Singh. 2020. Pretraining federated text models for next word prediction. *ArXiv*.
Dianbo Sui, Yubo Chen, Jun Zhao, Yantao Jia, Yuantao Xie, and Weijian Sun. 2020. FedED: Federated learning via ensemble distillation for medical relation extraction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2118-2128, Online. Association for Computational Linguistics.
Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. NewsQA: A machine comprehension dataset. In Proceedings of the 2nd Workshop on Representation Learning for NLP, pages 191-200, Vancouver, Canada. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008.
Hongyi Wang, Kartik Sreenivasan, Shashank Rajput, Harit Vishwakarma, Saurabh Agarwal, Jy-yong Sohn, Kangwook Lee, and Dimitris S. Papailiopoulos. 2020a. Attack of the tails: Yes, you really can backdoor federated learning. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris S. Papailiopoulos, and Yasaman Khazaeni. 2020b. Federated learning with matched averaging. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing:
System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. 2019. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST), (2).
T. Yang, G. Andrew, Hubert Eichner, Haicheng Sun, W. Li, Nicholas Kong, D. Ramage, and F. Beaufays. 2018a. Applied federated learning: Improving google keyboard query suggestions. ArXiv.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018b. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369-2380, Brussels, Belgium. Association for Computational Linguistics.
Ligeng Zhu, Zhijian Liu, and Song Han. 2019. Deep leakage from gradients. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 14747-14756.
Xinghua Zhu, Jianzong Wang, Zhenhou Hong, and Jing Xiao. 2020. Empirical studies of institutional federated learning for natural language processing. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 625-634, Online. Association for Computational Linguistics.
A FL+NLP
Many realistic NLP services heavily rely on users' local data (e.g., text messages, documents and their tags, questions and selected answers, etc.), which can be located at either personal devices or larger data-silos for organizations. These local data are usually regarded as highly private and thus not directly accessible by anyone, according to many data privacy regulations; this makes it difficult to train a high-performance model to benefit users. Federated learning aims to solve machine learning under such a privacy-preserving use case, thus offering a novel and promising direction to the community: FL+NLP.
Apart from the goal of learning a shared global model for all clients, FL also provides a new perspective for many other interesting research questions in NLP. One related direction is to develop personalized models for NLP applications, which requires both protection of data privacy and transferred ability on users' own input feature distribution caused by language styles, interested topics and so on. The recent concerns on adversarial attacks and safety issues of NLP models are also highly related to  $\mathrm{FL + NLP}$ . We thus believe  $\mathrm{FL + NLP}$  is of vital importance for applying NLP technologies in realistic use cases and could benefit many relevant research areas.
A.1 Challenges of Applying FL in NLP
Given the promising benefits of studying FL+NLP, however, this research direction is currently blocked by the lack of a standardized platform providing fundamental building blocks: benchmark datasets, NLP models, FL methods, evaluation protocols, etc. Most of the current FL platforms either focus on unifying various FL methods and use computer vision models and datasets for their experiments, but lack the ability to connect the study of pre-trained language models, the most popular NLP, and realistic NLP applications of various task formulations.
The first challenge in developing a comprehensive and universal platform for  $\mathrm{FL + NLP}$  is to deal with various task formulations for realistic NLP applications, which have different input and output
format (Section B). As the non-IID data partition over clients is the major feature of FL problems, it is also a challenge to simulate the realistic non-IID partition for existing NLP datasets (Section 3.2). Finally, a platform also must integrate various FL methods with the Transformer-based NLP models for a variety of task types, and thus a flexible and extensible learning framework is needed. In particular, the conventional trainer component of Transformers now needs to be modified for efficient and safe communications towards federated learning (Section F).
B Basic Formulations of NLP Tasks
There are various types of NLP applications, but many of them share a similar task formulation (i.e., input-and-put formats). We show four common task formulations that can cover most of the mainstream NLP applications: text classification, sequence tagging, question answering, sequence-to-sequence generation.
Text Classification (TC) The input is a sequence of words,  $x = [w_{1}, w_{2}, \ldots]$ , and the output is a label  $y$  in a fixed set of labels  $\mathcal{L}$ . Many NLP applications can be formulated as text classification tasks. For example, we can use TC models for classifying the topic of a news article to be political, sports, entertainment, etc., or analyzing movie reviews to be positive, negative or neutral.
Sequence Tagging (ST) The input is a sequence of words,  $x = [w_{1}, w_{2}, \ldots, w_{N}]$ , and the output is a same-length sequence of tags  $y = [t_{1}, t_{2}, \ldots, t_{N}]$ , where  $t_{i}$  is in a fixed set of labels  $\mathcal{L}$ . The main difference between TC and ST is that ST learns to classify the label of each token in a sentence, which is particularly useful in analyzing syntactic structures (e.g., part-of-speech analysis, phrase chunking, and word segmentation) and extracting spans (e.g., named entity recognition).
Question Answering (QA) Given a passage  $P = [w_{1}, w_{2}, \ldots, w_{N}]$  and a question  $q$  as input, the task is to locate a span in the passage as the answer to the question. Thus, the output is a pair of token index  $(s, e)$  where  $s, e \in \{1, 2, \ldots, N\}$  for denoting the begin and end of the span in the passage. This particular formulation is also known as reading comprehension.
Natural Language Generation (NLG) Both in
Figure 8: The probability density of quantity of training examples in each of the 100 clients on the 20News dataset with different  $\beta$ . When  $\beta$  is larger, then all clients share more similar numbers of examples; when  $\beta$  is smaller, then the range of the quantity is much wider — i.e., the larger differences between clients in terms of their sizes of datasets.
put and output are sequence of words,  $x = [w_1^i, w_2^i, \ldots, w_N^i]$ ,  $y = [w_1^o, w_2^o, \ldots, w_M^o]$ . It is shared by many realistic applications such as summarization, response generation in dialogue systems, machine translation, etc.
Language Modeling (LM) The left-to-right language modeling task considers a sequence of words as the input  $x = [w_{1}, w_{2}, \ldots, w_{n}]$  and a token  $y = w_{n+1}$  as the output. The output token is expected to be the most plausible next word of the incomplete sentence denoted as  $x$ . Although the direct application of LM is limited, a high-performance pre-trained language model can benefit a wide range of NLP applications (as above) via fine-tuning. It also serves as an excellent test bed as it requires no human annotations at all.
Others. There are some other applications that not are covered by the above four basic formulations, and our extensible platform (detailed in Section F) enables users to easily implement their specific tasks. For each task formulation, we show which datasets are used in FedNLP and how we partition them in Section 3.
Non-III. Label Distribution Note that this might cause a few clients not to have enough examples to sample for particular labels if they are
already used up. Prior works choose to stop assigning early and remove such clients, but it consequently loses the other unused examples and also causes the inconsistency of client numbers. Thus, to avoid these issues, we propose a dynamic reassigning method which complement the vacancy of a label by filling in the examples of other labels based on their current ratio of remaining unassigned examples.
C.1 The FedNLP Training Pipeline: Security and Efficiency
Under the definition of federated learning in Algorithm 1, we design a training system to support the research of NLP in the FL paradigm. We highlight its core capabilities and design as follows.
Supporting diverse FL algorithms. FedNLP aims to enable flexible customization for future algorithmic innovations. We have supported a number of classical federated learning algorithms, including FedAvg (McMahan et al., 2017a), FedOPT (Reddi et al., 2021), and FedProx (Li et al., 2020b). These algorithms follow the same framework introduced in Algorithm 1. The algorithmic APIs are modularized: all data loaders follow the same format of input and output arguments, which are compatible with different models and algorithms and are easy to support new datasets; the method of defining the model and related trainer is kept the same as in centralized training to reduce the difficulty of developing the distributed training framework. For new FL algorithm development, worker-oriented programming reduces the difficulty of message passing and definition. More details are introduced in Appendix F.3.
Enabling secure benchmarking with lightweight secure aggregation. In particular, FedNLP enhances the security aspect of federated training, which is not supported by existing non-NLP-oriented benchmarking libraries (e.g., TFF, LEAF). This is motivated by the fact that model weights from clients may still have the risk of privacy leakage (Zhu et al., 2019). To break this barrier, we integrate secure aggregation (SA) algorithms to the FedNLP system. NLP researchers do not need to master security-related knowledge and also benefit from a secure distributed training environment. To be more
specific, FedNLP supports state-of-the-art SA algorithms LightSecAgg, SecAgg (Bonawitz et al., 2017), and  $\mathrm{SecAgg + }$  (Bell et al., 2020). At a high-level understanding, SA protects the client model by generating a single random mask and allows their cancellation when aggregated at the server. Consequently, the server can only see the aggregated model and not the raw model from each client. In this work, our main effort is to design and optimize these SA algorithms in the context of the FedNLP system. We provide an algorithmic performance comparison in Appendix F.5.
Realistic evaluation with efficient distributed system design. FedNLP aims to support distributed training in multiple edge servers (e.g., AWS EC2) or edge devices (e.g., IoTs and smartphones). To achieve this, the system is designed with three layers: the application layer, the algorithm layer, and the infrastructure layer. At the application layer, FedNLP provides three modules: data management, model definition, and a single-process trainer for all task formats; at the algorithm layer, FedNLP supports various FL algorithms; at the infrastructure layer, FedNLP aims at integrating single-process trainers with a distributed learning system for FL. Specifically, we make each layer and module perform its own duties and have a high degree of modularization. We refer readers to Appendix F for a detailed description of the system architecture and design philosophy.
D More Related Works
Federated Learning Methods. Federated Learning (FL) is a widely disciplinary research area that mainly focuses on three aspects: statistical challenge, trustworthiness, and system optimization. Numerous methods have been proposed to solve statistical challenges, including FedAvg (McMahan et al., 2017b), FedProx (Li et al., 2020b), FedOPT (Reddi et al., 2021), FedNAS (He et al., 2020a,d), and FedMA (Wang et al., 2020b) that alleviate the non-IID issue with distributed optimization, and new formulations, MOCHA (Smith et al., 2017), pFedMe (Dinh et al., 2020), perFedAvg (Fallah et al., 2020), and Ditto (Li et al., 2021b), that consider
personalization and fairness in federated training.
For trustworthiness, security and privacy are the two main research directions that are mainly concerned with resisting data or model attacks, reconstruction, and leakage during training (So et al., 2021b,a, 2020; Prakash et al., 2020; Prakash and Avestimehr, 2020; Elkordy and Avestimehr, 2020; Prakash et al., 2020; Wang et al., 2020a; Lyu et al., 2020). Given that modern deep neural networks are over-parameterized and dominate nearly all learning tasks, researchers also proposed algorithms or systems to improve the efficiency and scalability of edge training (He et al., 2020b,c, 2019, 2021). We refer readers to the canonical survey (Kairouz et al., 2019) for details.
Although tremendous progress has been made in the past few years, these algorithms or systems have not been fully evaluated on realistic NLP tasks introduced in this paper.
E Future Directions
Minimizing the performance gap. In the FL setting, we demonstrate that federated fine-tuning still has a large accuracy gap in the non-IID dataset compared to centralized fine-tuning. Developing algorithms for Transformer models with NLP tasks is of the highest priority.
Improving the system efficiency and scalability. Transformer models are usually large, while resource-constrained edge devices may not be able to run large models. Designing efficient FL methods for NLP tasks is thus a practical problem worth solving. How to adopt a reasonable user selection mechanism to avoid stragglers and speed up the convergence of training algorithms is also a pressing problem to be solved.
Trustworthy and privacy-preserving NLP. We argue that it is an important future research direction to analyze and assure the privacy-preserving ability of these methods, although our focus in this paper is the implementation and performance analysis of the FL methods for NLP tasks. It is now an open problem for both FL and NLP areas, while it is an orthogonal goal for improving the trustworthy of decentralized learning, and it is only possible to study privacy preservation when we have an existing FL+NLP platform. This is also part of our motivation in
proposing FedNLP, and we believe our framework provides a set of flexible interfaces for future development to analyze and improve the privacy-preserving ability of FL methods for NLP tasks and beyond.
Personalized FedNLP. From the perspective of the data itself, user-generated text is inherently personalized. Designing personalized algorithms to improve model accuracy or fairness is a very promising direction. In addition, it is also an interesting problem to adapt the heterogeneous model architecture for each client in the FL network. We show that it is feasible to only fine-tune a small amount of the parameters of LMs, so it is promising to adapt recent prefix-tuning methods (Li and Liang, 2021) for personalizing the parameters of NLP models within the FedNLP framework.
F The System Design of FedNLP
The FedNLP platform consists of three layers: the application layer, the algorithm layer, and the infrastructure layer. At the application layer, FedNLP provides three modules: data management, model definition, and single-process trainer for all task formats; At the algorithm layer, FedNLP supports various FL algorithms; At the infrastructure layer, FedNLP aims at integrating single-process trainers with a distributed learning system for FL. Specifically, we make each layer and module perform its own duties and have a high degree of modularization.
F.1 Overall Workflow
The module calling logic flow of the whole framework is shown on the left of Figure 9. When we start the federated training, we first complete the launcher script, device allocation, data loading, and model creation, and finally call the API of the federated learning algorithm. This process is expressed in Python-style code (see Alg. 2).
F.2 The Application Layer
Data Management. In data management, What DataManager does is control the whole workflow from loading data to returning trainable features. To be specific, DataManager is set up for reading h5py data files and driving a preprocessor to convert raw data to features. There are four
types of DataManager according to the task definition. Users can customize their DataManager by inheriting one of the DataManager class, specifying data operation functions, and embedding a particular preprocessor. Note that the raw data's H5Py file and the non-IID partition file are preprocessed offline, while DataManager only loads them in runtime.
Model Definition. We support two types of models: Transformer and LSTM. For Transformer models, to dock with the existing NLP ecology, our framework is compatible with the HuggingFace Transformers library (Wolf et al., 2020), so that various types of Transformers can be directly reused without the need for re-implementation. Specifically, our code is compatible with the three main classes of Tokenizer, Model, and Config in HuggingFace. Users can also customize them based on HuggingFace's code. Although LSTM has gradually deviated from the mainstream, we still support LSTM to reflect the framework's integrity, which may meet some particular use cases in a federated setting.
NLP Trainer (single process perspective). As for the task-specific NLP Trainer, the most prominent feature is that it does not require users to have any background in distributed computing. Users of FedNLP only need to complete single-process code writing. A user should inherit the Trainer class in the application layer
Figure 9: The overall workflow and system design of the proposed FedNLP platform.
to implement the four methods as shown in the figure: 1. the get_model_parameters() interface allows the algorithm layer to obtain model parameters and transmit them to the server; 2. the set_model_parameters() interface obtains the updated model from the server's aggregation and then updates the model parameters of the local model; 3. the programming of the train() and test() function only needs to consider the data of a single user, meaning that the trainer is completely consistent with the centralized training.
F.3 The Algorithm Layer
In the design of the algorithm layer, we follow the principle of one-line API. The parameters of the API include model, data, and single-process trainer (as shown in Algorithm 2). The algorithms we support include:
Centralized Training. We concatenate all client datasets and use the global data  $\mathcal{D}_G$  to train a global model — i.e., the conventional protocol for
learning an NLP model on a dataset.
FedAvg (McMahan et al., 2017a) is the de facto method for federated learning, assuming both client and server use the SGD optimizer for updating model weights.
FedProx (Li et al., 2020b) can tackle statistical heterogeneity by restricting the local model updates to be closer to the initial (global) model with L2 regularization for better stability in training.
FedOPT (Reddi et al., 2021) is a generalized version of FedAvg. There are two gradient-based optimizers in the algorithm: ClientOpt and ServerOpt (please refer to the pseudo code in the original paper (Reddi et al., 2021)). While ClientOpt is used to update the local models, SerevrOpt treats the negative of aggregated local changes  $-\Delta^{(t)}$  as a pseudo-gradient and applies it on the global model. In our FedNLP framework, by default, we set the ClientOpt to be AdamW (Loshchilov and Hutter, 2019) and the
SerevrOpt to be SGD with momentum (0.9) and fix server learning rate as 1.0.
Each algorithm includes two core objects, ServerManager and ClientManager, which integrate the communication module ComManager from the infrastructure layer and the Trainer of the training engine to complete the distributed algorithm protocol and edge training. Note that users can customize the Trainer by passing a customized Trainer through the algorithm API.
F.4 The Infrastructure Layer
The infrastructure layer includes three modules:
1) Users can write distributed scripts to manage GPU resource allocation. In particular, FedNLP provides the GPU assignment API (map_process_to_gpu() in Algorithm 2) to assign specific GPUs to different FL Clients.
2) The algorithm layer can use a unified and abstract ComManager to complete a complex algorithmic communication protocol. Currently, we support MPI (Message Passing Interface), RPC (Remote procedure call), and MQTT (Message Queuing Telemetry Transport) communication backend. MPI meets the distributed training needs in a single cluster; RPC meets the communication needs of cross-data centers (e.g., cross-silo federated learning); MQTT can meet the communication needs of smartphones or IoT devices.
3) The third part is the training engine, which reuses the existing deep learning training engines by presenting as the Trainer class. Our current version of this module is built on PyTorch, but it can easily support frameworks such as TensorFlow. In the future, we may consider supporting the lightweight edge training engine optimized by the compiler technology at this level.
F.5 Enhancing Security with Secure Aggregation (SA)
FedNLP supports state-of-the-art SA algorithms LightSecAgg, SecAgg (Bonawitz et al., 2017), and  $\mathrm{SecAgg + }$  (Bell et al., 2020). Here, we provide a short performance comparison of these three algorithms. In general, LightSecAgg provides the same model privacy guarantees as SecAgg (Bonawitz et al., 2017) and SecAgg+ (Bell et al., 2020)) while substantially reducing the aggregation (hence run-time) complexity (Figure
??). The main idea of LightSecAgg are that each user protects its local model using a locally generated random mask. This mask is then encoded and shared with other users, in such a way that the aggregate mask of any sufficiently large set of surviving users can be directly reconstructed at the server. Our main effort in FedNLP is integrating these algorithms, optimizing its system performance, and designing user-friendly APIs to make them compatible with NLP models and FL algorithms.