Federated Learning for Semantic Parsing: Task Formulation, Evaluation Setup, New Algorithms
Year: 2023

Abstract:
This paper studies a new task of federated learning (FL) for semantic parsing, where multiple clients collaboratively train one global model without sharing their semantic parsing data. By leveraging data from multiple clients, the FL paradigm can be especially beneficial for clients that have little training data to develop a data-hungry neural semantic parser on their own. We propose an evaluation setup to study this task, where we re-purpose widely-used single-domain text-to-SQL datasets as clients to form a realistic heterogeneous FL setting and collaboratively train a global model. As standard FL algorithms suffer from the high client heterogeneity in our realistic setup, we further propose a novel LOss Reduction Adjusted Reweighting (Lorar) mechanism to mitigate the performance degradation, which adjusts each client’s contribution to the global model update based on its training loss reduction during each round. Our intuition is that the larger the loss reduction, the further away the current global model is from the client’s local optimum, and the larger weight the client should get. By applying Lorar to three widely adopted FL algorithms (FedAvg, FedOPT and FedProx), we observe that their performance can be improved substantially on average (4%-20% absolute gain under MacroAvg) and that clients with smaller datasets enjoy larger performance gains. In addition, the global model converges faster for almost all the clients.1

heading: 1 Introduction
text: Semantic parsing aims to translate natural language utterances into formal meaning representations such as SQL queries and API calls and can be applied to build natural language interfaces that enable users to query data and invoke services without
∗Work started during the internship at IBM T. J. Watson Research Center and continued at OSU.
1Our code and data are publicly available at https://github.com/OSU-NLP-Group/ FL4SemanticParsing
programming (Berant et al., 2013; Thomason et al., 2015; Su et al., 2017; Campagna et al., 2017). Neural semantic parsers have achieved remarkable performance in recent years (Wang et al., 2020a; Rubin and Berant, 2021; Scholak et al., 2021). However, they are data-hungry; bootstrapping a neural semantic parser by annotating data on a large scale can be very challenging for many institutions, as it requires the annotators to have intimate knowledge of formal programs. One natural thought is to leverage data from different institutions and train a unified model that can be used for all institutions. However, in practice, institutions such as hospitals, banks, and legal firms are prohibited from sharing their data with others, due to privacy concerns. Therefore, for institutions that only have very limited data, it is extremely hard to build their own neural semantic parsers.
Federated learning (FL) (Konečnỳ et al., 2016; McMahan et al., 2017; Yang et al., 2018) has turned out to be a popular training paradigm where multiple clients can collaboratively train a global model without exchanging their own data. In this paper, we study a new task of federated learning for semantic parsing. Through FL on the data scattered on different clients (e.g., institutions), we aim to obtain a global model that works well for all clients, especially those that have insufficient data to build their own neural models.
Towards that end, we propose an evaluation setup by re-purposing eight existing datasets that are widely adopted for text-to-SQL parsing, such as ATIS (Srinivasan Iyer and Zettlemoyer, 2017) and Yelp (Navid Yaghmazadeh and Dillig, 2017). These datasets demonstrate great heterogeneity, in terms of dataset sizes, language usage, database structures, and SQL complexity, as they were collected from the real life by different researchers, at different times, and for different purposes. Therefore, we use this collection to simulate a realistic scenario where eight clients with very different
12149
data participate in the FL paradigm to jointly train a neural semantic parser.
Heterogeneity, where the data distributions and dataset sizes on different clients are different, is recognized as one of the biggest challenges in FL (McMahan et al., 2017; Reddi et al., 2020; Li et al., 2020a, 2021; Shoham et al., 2019; T Dinh et al., 2020). Existing work either uses synthetic data (Li et al., 2020a) or splits a classification dataset based on Dirichlet distribution (Lin et al., 2022) to simulate the non-IID federated learning setting, while we propose a more realistic setup to study this setting for semantic parsing. Pre-trained language models such as T5 (Raffel et al., 2020) have been shown as a powerful unified model for various semantic parsing tasks (Xie et al., 2022; Rajkumar et al., 2022), which can be leveraged to save us the efforts for client-specific model designs. Specifically, we adopt T5-base as our backbone semantic parser in the FL paradigm, and conduct extensive experiments and analysis using three widelyadopted FL algorithms: FedAvg (McMahan et al., 2017), FedOPT (Reddi et al., 2020) and FedProx (Li et al., 2020a).
As standard FL algorithms suffer from the high client heterogeneity in our realistic setup, we further propose a novel re-weighting mechanism for combining the gradient updates from each client during the global model update. The high-level idea is shown in Figure 1. Our intuition is that, for each client, the reduction of training loss during each round can signalize how far the current global model is away from the local optimum. By giving larger weights to those clients that have larger training loss reduction, the global model update can accommodate those clients better, thus mitigating potential performance degradation caused by high heterogeneity. We formulate this intuition as a re-weighting factor to adjust how much each client should contribute to the global model update during each round. Our proposed mechanism can be applied to all the three FL algorithms and experiments show that it can substantially improve both their parsing performance and their convergence speed, despite being very simple.
In summary, our main contributions are:
• To the best of our knowledge, we are the first to study federated learning for semantic parsing, a promising paradigm for multiple institutions to collaboratively build natural language interfaces without data sharing, which is es-
pecially beneficial for institutions with little training data.
• We propose an evaluation setup to simulate a realistic heterogeneous FL setting where different participating institutions have very different data. We re-purpose eight single-domain text-to-SQL datasets as eight clients, which demonstrate high heterogeneity in terms of dataset sizes, language usage, database structures, and SQL complexity.
• We propose a novel re-weighting mechanism, which uses the training loss reduction of each client to adjust its contribution to the global model update during each round. Experiments show that our re-weighting mechanism can substantially improve the model performance of existing FL algorithms on average, and clients with smaller training data observe larger performance gains. We discuss the limitations of our work and encourage future work to further study this task.

heading: 2 Motivation and Task Formulation
text: Semantic parsing aims to translate natural language utterances into formal meaning representations and has numerous applications in building natural language interfaces that enable users to query data and invoke services without programming. As many institutions often lack data to develop neural semantic parsers by themselves, we propose a federated learning paradigm, where clients (i.e., “institutions”) collaboratively train a global semantic parsing model without sharing their data.
There are two realistic settings of FL: crosssilo setting and cross-device setting (Kairouz et al.,
2021; Lin et al., 2022). For the cross-silo setting, clients are large institutions, such as hospitals and companies, and the number of clients is limited in this setting. In general, they have large computational resources and storage to train and store a large model, and large communication costs between the server and clients are tolerated. For the cross-device setting, clients are small devices such as mobile phones and Raspberry Pis, thus there may exist a huge number of clients. They have limited computational resources and storage and only small communication costs between the server and clients are affordable. Here our FL for semantic parsing can be regarded as a cross-silo setting, where each client is a relatively large institution that hopes to build a natural language interface based on its user utterances and underlying data. Studying FL for semantic parsing under a cross-device setting could be interesting future work.

heading: 3 Evaluation Setup
text: As we are the first to study cross-silo FL for semantic parsing, there is no benchmark for this task. Thus we establish an evaluation setup by re-purposing eight single-domain text-to-SQL datasets (Finegan-Dollak et al., 2018) as eight “clients”, which demonstrate high heterogeneity in terms of dataset sizes, domains, language usage, database structures and SQL complexity. Table 1 shows their statistics.
Given a natural language question and the database schema, text-to-SQL parsing aims to generate a SQL query. Here the question is a sequence of tokens and the database schema consists of multiple tables with each table containing multiple columns. Figure 7 in Appendix shows an example of this task. We adopt T5-base as our backbone
model, which has been shown as an effective unified model for various semantic parsing tasks (Xie et al., 2022). Similarly as in previous work (Xie et al., 2022), we concatenate the question tokens with the serialized relational table schemas (table names and column names) as the model input and output a sequence of SQL tokens.
The heterogeneity of the eight clients is described in detail from the following perspectives.
Domain: The clients are from diverse domains. Some clients such as Scholar and Academic are from closer domains than others.
Dataset Size: The clients differ significantly in terms of dataset sizes. Here, we consider datasets with more than 1000 train examples as large-sized datasets, with 200∼1000 as medium-sized datasets, and with less than 200 as small-sized datasets. In our setup, we have 2 large-sized clients (Advising and ATIS), 3 medium-sized clients (Geoquery, Restaurants and Scholar), and 3 small-sized clients (Academic, IMDB and Yelp).
Diversity: “SQL pattern count” shows the number of SQL patterns in the full dataset. The patterns are abstracted from the SQL queries with specific table names, column names and variables anonymized. A larger value under this measure indicates greater diversity. In our benchmark, Advising, ATIS and Scholar have larger diversity than the other datasets.
Redundancy: “Questions per unique SQL query” counts how many natural language questions can be translated into the same SQL query (where variables are anonymized). A larger value indicates higher redundancy in the dataset. Intuitively, the higher the redundancy, the more easily a model can make correct predictions. In our benchmark, the redundancy for Advising and Restaurants
is higher than the other datasets. Complexity: “Unique tables per SQL query” (where variables in the SQL query are anonymized) represents how many unique tables are mentioned in one query. “SELECTs per query” counts how many SELECT clauses are included in one query. The larger these two measures, the more complex the dataset is and the more difficult for a model to make predictions. In our benchmark, Advising and ATIS are more complex.

heading: 4 FL for Semantic Parsing
text: In this section, we first introduce the background of FL, more specifically, its training objective, training procedure and three widely adopted FL algorithms. Then we describe the motivating insights and details of our proposed mechanism.

heading: 4.1 Background
text: Training Objective. Federated learning aims to optimize the following objective function:
min w F(w) := ∑N i=1
piLi(w) where Li(w) = Eb∼Di [fi(w, b)].
(1)
In Eqn. (1), Li(w) denotes the local training objective function of the client i and N denotes the number of clients. w ∈ Rd represents the parameters of the global model. b denotes each batch of data. The local training loss function fi(w, b) is often the same across all the clients, while Di denotes the distribution of the local client data, which is often different across the clients, capturing the heterogeneity. pi is defined as the training size proportion in Eqn. (2), where |Di| is the training size of client i.
pi = |Di| / ∑N
i=1 |Di| (2)
Training Procedure. Federated learning is an iterative process shown in Figure 2. The server initializes the global model, followed by multiple communication rounds between the server and clients. In each communication round, there are four steps between the server and clients. 1) In round t, the server sends the global model wt to all the clients. 2) After clients receive the global model wt as the initialization of the local model, they start to train it using their own data for multiple epochs and obtain the local model changes ∆wti during the local training stage. 3) The clients send their local model
changes to the server. 4) The server aggregates the local model changes ∆wti collected from different clients as Eqn. (3) shows, and then uses the t-th round’s global model wt and the aggregated local model changes ∆wt to update the global model. As Eqn. (4) shows, wt+1 is the global model after the update. Here, η denotes the server learning rate. The server will send the updated model wt+1 to the clients, then the (t+1)-th round starts.
The above procedure will repeat until the algorithm converges.
∆wt = ∑N
i=1 pi∆w
t i (3)
wt+1 = wt − η∆wt (4)
FL Algorithms. We explore three popular FL algorithms for our task:
Federated Averaging (FedAvg) (McMahan et al., 2017) uses stochastic gradient descent (SGD) as the local training optimizer to optimize the training procedure and uses the same learning rate and the same number of local training epochs for all the clients.
FedOPT (Reddi et al., 2020) is a generalized version of FedAvg. The algorithm is parameterized by two gradient-based optimizers: CLIENTOPT and SERVEROPT. CLIENTOPT is used to update the local models on the client side, while SERVEROPT treats the negative of aggregated local changes "−∆wt" as a pseudo-gradient and applies it to the global model on the server side. FedOPT allows powerful adaptive optimizers on both
server side and client side. FedProx (Li et al., 2020a) tries to tackle the statistical heterogeneity issue by adding an L2 regularization term, which constrains the local model to be closer to the local model initialization (i.e., the global model) during each round for stable training.
To summarize, for the local training stage, both FedAvg and FedOPT optimize the local training objective fi(w, b); for FedProx, it optimizes Eqn. (5), where µ is a hyperparameter and wt is the local model initialization (i.e., the global model) during the t-th round.
min w
hi(w, b, w t) := fi(w, b) +
µ 2 ∥w − wt∥2
(5)
For the cross-silo setting where all clients participate in training for each round, these three algorithms optimize Eqn. (1) during the FL process.

heading: 4.2 Our Proposed Re-weighting Mechanism
text: Motivating Insights. Heterogeneity, where the data distributions and dataset sizes on different clients are different, is recognized as one of the biggest challenges in FL, which usually leads to performance degradation for clients. Here, we uniquely observe the clients’ heterogeneity from the perspective of their training loss reduction. Take Restaurants and Yelp as two example clients. Figure 3 shows their training loss variation w.r.t. "Step". Here the "Step" is the number of iteration steps for each client during training. Adjacent high and low points in the figure correspond to one communication round. When the curve goes down, it means the client is in the local training stage. When the curve goes up, it means the server has updated the global model based on the aggregated local model changes from all clients and each client starts a new round of local training with the updated global model as the local model initialization. Since for different clients, the dataset sizes and the local training epochs are different, for the same communication round, the "Step" for different clients is different.
As we can see, after each round, the global model deviates from the optimization trajectory of each client. Thus the reduction of the training loss can signalize how far the global model is away from the client’s local optimum. As pi decides how much each client contributes to the global model update, we give larger weights to
those clients who have larger training loss reduction to make the global model update accommodate them better, thus mitigating potential performance degradation caused by high heterogeneity. Proposed Mechanism. Based on the above insights, we use the training loss reduction to adjust the weight of each client, so as to reschedule its contribution to the global model update. The final weight is formulated as Eqn. (6), where ∆Lti is the training loss reduction during the t-th round.
pti = |Di|∆Lti/ ∑N
i=1 |Di|∆Lti (6)
FedAvg, FedOPT, FedreProx with our proposed mechanism are summarized in Algorithm 1 in Appendix.

heading: 5 Experiments
text: Datasets. We re-purpose eight datasets: ATIS (Srinivasan Iyer and Zettlemoyer, 2017; Deborah A. Dahl and Shriber, 1994), GeoQuery (Srinivasan Iyer and Zettlemoyer, 2017; Zelle and Mooney, 1996), Restaurants (Tang and Mooney, 2000; Ana-Maria Popescu and Kautz, 2003; Giordani and Moschitti, 2012), Scholar (Srinivasan Iyer and Zettlemoyer, 2017), Academic (Li and Jagadish, 2014), Advising (Finegan-Dollak et al., 2018), Yelp and IMDB (Navid Yaghmazadeh and Dillig, 2017) as eight clients. These datasets have been standardized to the same SQL style by Finegan-Dollak et al. (2018). Their characteristics have been described in Section 3. We follow "question split" datasets preprocessed by FineganDollak et al. (2018) to split the train, dev and test data, which means we let the train, dev and test examples have different questions but the same SQL queries are allowed. For Advising, ATIS, GeoQuery and Scholar, we directly use the original question split as our split. For Restaurants, Academic, IMDB and Yelp, since the data sizes are
relatively small, the original question split uses 10 splits for cross validation without specifying train, dev and test examples. Given FL is costly as we need multiple GPUs to finish one experiment, we fix the train, dev and test set by randomly selecting 6 splits as the train set, 2 splits as the dev set and 2 splits as the test set. Evaluation Metrics. 1) Exact Match (EM): a prediction is deemed correct only if it is exactly the same as the ground truth (i.e., exact string match), which is widely used for text-to-SQL parsing (Finegan-Dollak et al., 2018). All the evaluations in our experiments consider the values generated in the SQL query. 2) MacroAvg: The arithmetic mean of EM across all clients, which treats each client equally. 3) MicroAvg: The total number of correct predictions on all the clients divided by the total test examples, which treats each test example equally. Learning Paradigm. We compare three learning paradigms: finetuning, centralized and FL. 1) Finetuning: we individually finetune our backbone model (T5-base) on the training data of each client. 2) Centralized: we merge the training data of all the clients and finetune our backbone model on the merged training data to obtain one model. 3) FL: we leverage eight clients and a server to learn a global model without sharing each client’s local data. By comparing individual finetuning and FL, we can show the benefit of FL for some clients, especially for small-sized clients. The centralized paradigm is less practical compared with the other two paradigms due to privacy considerations. However, it can serve as a useful reference to help validate how effective an FL algorithm is in fully exploiting heterogeneous data across multiple clients. Implementation Details. We implement the FL algorithms and T5-base model based on FedNLP (Lin et al., 2022), FedML (He et al., 2020) and UnifiedSKG (Xie et al., 2022). We use Adafactor (Shazeer and Stern, 2018) as the optimizer for finetuning and centralized paradigms, and as the client optimizer2 for FL paradigm, since it has been shown as the best optimizer to optimize the T5 model. More details are in Appendix A.1.
For the computing resources, we use 1 NVIDIA A6000 48GB GPU for finetuning, with batch size 8. We use 2 NVIDIA A6000 48GB GPUs for central-
2Note we use Adafactor as the local optimizer for FedAvg, so the FedAvg in our paper is slightly different from the original proposed FedAvg, which uses stochastic gradient descent(SGD) as the local optimizer.
ized training, with batch size 8. We use 5 NVIDIA A6000 48GB GPUs for all federated learning experiments. Specifically, one GPU is used as the server and the other four GPUs are used as 8 clients, with each GPU accommodating 2 clients. The batch size for clients GeoQuery, Restaurants, Scholar, Academic, IMDB and Yelp is 4, and for clients Advising and ATIS is 8.

heading: 6 Results and Analysis

heading: 6.1 Main Results
text: Centralized vs. Finetuning. As Table 2 shows, compared with the individual finetuning setting, the model performance under the centralized setting has been improved on all the datasets except Scholar. This means merging all the data to train a model, which increases the size and diversity of training data, can improve the model’s generalization ability and lead to improvement for most datasets. This observation also motivates us to leverage these datasets to study FL for semantic parsing, which is a more practical paradigm than the centralized one. Effectiveness of Lorar in FL. Applying our proposed Lorar mechanism can substantially improve the performance of all three FL algorithms overall. As Table 2 shows, for FedOPT, our proposed FedOPTlorar performs substantially better or similarly on all clients, except for a slight drop on GeoQuery and Scholar. Moreover, on the three smaller datasets: Academic, IMDB and Yelp, Lorar brings much larger performance gains. For FedAvg and FedProx, in addition to these three datasets, Lorar also brings substantial improvements on two medium-sized clients: Restaurants and Scholar. These observations validate the effectiveness of our proposed mechanism under different FL algorithms and across different clients.
We additionally analyze these three FL algorithms and their performance variation with and without using Lorar under different communication rounds. More details are included in Appendix A.2 and A.3. FL vs. Finetuning/Centralized. As Table 2 shows, the original FedOPT outperforms finetuning on GeoQuery and IMDB, which shows that FL can boost the model performance for some clients. In addition, although there is still a gap between existing FL algorithms (FedOPT, FedAvg, and FedProx) and the centralized setting, by equipping them with our proposed Lorar, we can reduce the
gap by 4-20 points (i.e., absolute difference under MacroAvg). It is worth noting that institutions are often reluctant or prohibited to share their data in practice, especially for SQL data that may directly reveal private database content. Therefore, the centralized paradigm is impractical. Nonetheless, it can serve as a useful reference to help validate how effective an FL algorithm is in fully exploiting heterogeneous data across multiple clients. The results show that our benchmark provides a challenging testbed for a realistic FL problem, and there is still a large room to further improve the FL algorithms.

heading: 6.2 Training Loss Analysis
text: To better understand how Lorar affects the training process in FL, we show the training loss variation for FedOPT and FedOPTlorar in Figure 4. For FedOPT, we can see for larger datasets such as Advising and ATIS, the training converges much faster and the global model is closer to the client’s local optimum within very few rounds. While for smaller datasets such as Academic, IMDB and Yelp, the
training loss oscillates widely, which means the global model converges slower for these clients (if at all). After applying Lorar, however, the training loss converges faster on almost all the clients, which means the global model can get close to the client’s local optimum more quickly and easily.

heading: 6.3 Alternative Weighting Mechanisms
text: As FedOPT performs best among all three FL baselines, we use it to compare Lorar with alternative weighting mechanisms. As Table 3 shows, Lorar, which considers both the training set size and the loss reduction in the weight, can achieve the best results. Comparing FedOPTlr (i.e., FedOPT with only loss reduction considered in the weight) and FedOPTlorar, we can see removing the training set size from the weight will lead to a large drop under MacroAvg and MicroAvg, which indicates that training set size is an important factor during the aggregation. This is intuitive since for those clients which have more training data, their local models tend to be more reliable and more general-
izable. We also compare with FedOPTequal where all clients are given the same weight. We can see that our FedOPTlorar yields superior performance. The conclusion can also be verified in Figure 6 in Appendix, where we show their performance variation under different communication rounds.

heading: 6.4 Impact from Dataset Heterogeneity
text: (1) The impact of diversity, redundancy and complexity: In Table 2 and 3, for Restaurants, the results of finetuning, centralized training, and varying weighting mechanisms of FedOPT are pretty close and all very high (close to 100%), which shows it is a relatively easy dataset for any learning paradigm and weighting mechanism. Looking at Table 1, Restaurants has the smallest “SQL pattern count” (i.e., lowest diversity), second largest “Questions per unique SQL query” (i.e., second highest redundancy), close to the smallest “Unique tables per query” and “SELECTs per query” (i.e., close to lowest complexity), which makes models easily learn from this dataset (Section 3). For other datasets, they have higher diversity, lower redundancy, or higher complexity, which makes models harder to make predictions and the performance is generally lower than Restaurants. (2) The impact of dataset size: Smaller datasets tend to have lower performance, as shown in Table 2, which means they are harder to learn in general due to lack of data; however, they can benefit more from our proposed FL paradigm.

heading: 7 Related Work
text: Text-to-SQL. Text-to-SQL problem which translates natural language questions to SQL queries has been studied for many years. There have been several single-database text-to-SQL datasets such as Geoquery (Srinivasan Iyer and Zettlemoyer, 2017) and ATIS (Srinivasan Iyer and Zettlemoyer, 2017), which map from natural language questions to SQL queries on a single database. Finegan-Dollak et al.,
2018 curate eight datasets to unify their SQL format. These datasets cover a variety of domains and have different characteristics of the tables and SQL, which provide us a foundation to study the heterogeneous FL for the text-to-SQL problem.
One line of work designs special models for the text-to-SQL task such as designing a relationaware self-attention mechanism for the Transformer model to better encode the relation of the column mappings (Wang et al., 2020a) or adding constraints to the decoder (Scholak et al., 2021) to generate valid SQL queries, while another line of work tries to directly finetune a pre-trained language model such as T5 (Xie et al., 2022; Raffel et al., 2020; Rajkumar et al., 2022). As directly finetuning T5 has shown great performance and allows us to use a unified model architecture for all clients and the server, we choose T5-base as the backbone model in our work. Heterogeneity in Federated Learning. Heterogeneity is one of the major challenges in federated learning. Existing work (McMahan et al., 2017; Reddi et al., 2020; Li et al., 2020a, 2021; Shoham et al., 2019; T Dinh et al., 2020; Li et al., 2022) shows that heterogeneity can cause performance degradation. Several methods have been proposed to address this issue. For instance, FedOPT (Reddi et al., 2020) uses powerful adaptive optimization methods for both the server and clients, while FedProx (Li et al., 2020a) (and pFedMe (T Dinh et al., 2020)) regularizes the local training procedure. However, based on our observations in Section 6.1, our mechanism significantly outperforms these methods. Other work that aims to address the heterogeneity issue in FL includes FedNova (Wang et al., 2020b) and Li et al., 2020b. Specifically, FedNova (Wang et al., 2020b) uses the local training update steps to normalize the server aggregation, and Li et al., 2020b proposes to optimize the power-scaled training objective. Compared to FedNova, we use a more direct indicator, training
loss reduction, to adjust the weight for each client during aggregation. Different from Li et al., 2020b, our proposed simple yet effective mechanism does not require modification of the local client optimization step or additional tuning of any related hyperparameter.

heading: 8 Conclusions
text: To the best of our knowledge, we are the first to study federated learning for semantic parsing. Specifically, we propose a realistic benchmark by re-purposing eight single-domain text-to-SQL datasets. Moreover, we propose a novel loss reduction adjusted re-weighting mechanism (Lorar) that is applicable to widely adopted FL algorithms. By applying Lorar to FedAvg, FedOPT and FedProx, we observe their performance can be improved substantially on average, and clients with smaller datasets enjoy larger performance gains.
Limitations
In this work, we address the heterogeneity challenge in the task of FL for semantic parsing, by leveraging the reduction of training loss signal. Our work is motivated from the FL training procedure perspective to adjust the contribution of each client during the global model aggregation stage, but how each client’s data contribute to the final global model is still unclear. As the data of different clients contain different information, what kind of information of each client is helpful and can be more directly linked and utilized to facilitate the FL training is worth more efforts in future work.
In addition, our proposed re-weighting mechanism is a universal technique for cross-silo FL. Thus generalizing our proposed re-weighting mechanism to a broader range of tasks beyond semantic parsing, and further studying under what kind of conditions, Lorar can make a huge difference for FL would be interesting future work to pursue.

heading: Acknowledgements
text: The authors would like to thank colleagues from the OSU NLP group and all anonymous reviewers for their thoughtful comments. This research was supported in part by NSF OAC 2112606, NSF IIS 1815674, NSF CAREER 1942980, and Ohio Supercomputer Center (Center, 1987). The work done at IBM research was sponsored by the Combat Capabilities Development Command Army Research Laboratory and was accomplished under
Cooperative Agreement Number W911NF-13-20045 (ARL Cyber Security CRA). The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Combat Capabilities Development Command Army Research Laboratory or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation here on. We thank Chaoyang He for his help during reproducing FedNLP. We thank Wei-Lun (Harry) Chao for valuable discussion.

heading: A Appendix
text: Algorithm 1: Input: local datasets Di, number of
communication rounds T , number of local epochs E, server learning rate η, client learning rate ηi
Output: the final global model wT 1 Server executes: 2 for t ∈ 0, 1, 2, ..., T do 3 Sample a set of clients Cta 4 for i ∈ Ct in parallel do 5 Send the global model wt to client i 6 ∆wti , |Di|∆Lti ← LocalTraining(i, wt) 7 ∆wt =
∑ i∈Ct p t i∆w t i
8 For FedOPT/FedAvg/FedProx: pi = |Di| / ∑ i∈Ct |Di|
9 For ours (Lorar): pti = |Di|∆Lti/ ∑ i∈Ct |Di|∆Lti
10 wt+1←wt − η∆wt
11 return wT 12 Client executes: 13 FedAvg/FedOPT: L(w; b) = ∑(x,y)∈b f(w;x; y) 14 FedProx: L(w; b) =∑ (x,y)∈b f(w;x; y) + µ 2∥w − wt∥2 15 LocalTraining(i, wt) 16 wti ← wt 17 for epoch k = 0, 1, 2, ..., E do 18 for each batch b = {x, y} of Di do 19 wti ← wti − ηi∇Lt,ki (wti ; b)
20 ∆wti ← wt − wti 21 ∆Lti ← maxLti −minLti 22 return ∆wti , |Di|∆Lti to the server
aWe use all clients in our experiments.
A.1 Implementation Details
We use T5-base (Raffel et al., 2020) as the model for text-to-SQL task in all three learning paradigms (finetuning, centralized and FL), as it has been shown as an effective unified model for various semantic parsing tasks in UnifiedSKG (Xie et al., 2022). For all three FL algorithms, we implement them based on FedNLP (Lin et al., 2022) and FedML (He et al., 2020). We use Adafactor (Shazeer and Stern, 2018) as the optimizer for
finetuning and centralized paradigms, and as the client optimizer3 for FL paradigm, since it has been shown as the best optimizer to optimize for the T5 model.
For the FL paradigm, we tune hyperparameters for FedOPT, FedAvg and FedProx as follows. For FedOPT, we test all the combinations of the server learning rate from {0.001, 0.01 0.1, 0.5, 1} and {w/ 0.9, w/o} server momentum. We found 1 as the server learning rate and 0.9 as the server momentum is the best hyperparameter combination. For FedProx, we vary µ from {0.0001, 0.001, 0.01, 0.1, 1} and use the dev set to choose the best model. We finally choose the best hyperparameter 0.0001 in our experiment. For all the federated learning paradigms, we set local training epochs as 6 for two large datasets: ATIS and Advising. We set the local training epoch as 12 for all the other six datasets. We let all the clients participate in each round and we train the entire process for 60 rounds (which lasts around 60 hours). And we test the global model performance on the merged dev set for every 5 communication rounds to choose the best model. We use the best global model to evaluate on all eight test sets to get the global model performance on each client.
For the finetuning paradigm, we finetune T5base on each dataset for a maximum of 200 epochs. We use the dev set of each client to choose the best model and then evaluate the model on each test set.
For the centralized paradigm, we merge all eight training sets and then finetune T5-base for a maximum of 200 epochs on the merged dataset to get one centralized model. We merge all eight dev sets and use the merged dev set to choose the best model. Then we evaluate the centralized model on each test set.
For all finetuning, centralized and federated learning paradigms, we set the input length as 1024 and the output length as 512. We try learning rate in {1e-5, 1e-4, 1e-3}. We finally choose 1e-4 for the centralized paradigm, and 1e-4 for Advising, ATIS, Geoquery and Yelp in the finetuning paradigm and FL paradigm. We use 1e-3 for Restaurants, Scholar, Academic and IMDB in the finetuning paradigm and FL paradigm.
For the computing resources, we use 1 NVIDIA A6000 48GB GPU for finetuning, with batch size 8.
3Note we use Adafactor as the local optimizer for FedAvg, so the FedAvg in our paper is slightly different from the original proposed FedAvg, which uses stochastic gradient descent(SGD) as the local optimizer.
We use 2 NVIDIA A6000 48GB GPUs for centralized training, with batch size 8. We use 5 NVIDIA A6000 48GB GPUs for all federated learning experiments. Specifically, one GPU is used as the server and the other four GPUs are used as 8 clients, with each GPU accommodating 2 clients. The batch size for clients GeoQuery, Restaurants, Scholar, Academic, IMDB and Yelp is 4, and for clients Advising and ATIS is 8.
A.2 Comparison of FL Baselines. We treat FedAvg, FedOPT and FedProx as our FL baselines. As Figure 5 shows, among FedAvg, FedOPT and FedProx, FedOPT performs the best, achieving the closest performance to the centralized paradigm and the fastest convergence speed. FedAvg and FedProx have similar performances, and both of them have a large gap with FedOPT. This indicates that the server’s adaptive optimizer which only exists in FedOPT plays an important role to improve the performance.
A.3 Performance Variation under Varying Communication Rounds.
In Figure 5, comparing the performance of FL baselines with ours, FedOPTlorar performs slightly better than FedOPT. We hypothesize the small gap between FedOPT and the centralized paradigm limits the room for Lorar to show a large gain over FedOPT. For FedAvg and FedProx, we can see that applying Lorar performs significantly better, which demonstrates the effectiveness of leveraging the loss reduction to adjust the weights.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
"Limitations" Section
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? "Abstract" Section and "Introduction" Section (1)
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? "Evaluation Setup" (Section 3)
3 B1. Did you cite the creators of artifacts you used? "Experiments" (Section 5)
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? "Evaluation Setup" (Section 3)
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. "Evaluation Setup" (Section 3)
C 3 Did you run computational experiments? "Implementation Details" (Section 5)
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? "Implementation Details" (Section 5 and Appendix)
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? "Implementation Details" (Section 5 and Appendix)
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? "Implementation Details" (Section 5 and Appendix), "Results" (Section 6)
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? "Implementation Details" (Section 5 and Appendix)
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: As the data of different clients contain different information, what kind of information of each client is helpful and can be more directly linked and utilized to facilitate the FL training is worth more efforts in future work. In addition, our proposed re-weighting mechanism is a universal technique for cross-silo FL. Thus generalizing our proposed re-weighting mechanism to a broader range of tasks beyond semantic parsing, and further studying under what kind of conditions, Lorar can make a huge difference for FL would be interesting future work to pursue