FedID: Federated Interactive Distillation for Large-Scale Pretraining Language Models
Authors: Xinge Ma
Jiangming Liu
Jin Wang
Xuejie Zhang
Venue: EMNLP 2023 Main (2023)
Keywords: decentralized learning
federated learning
federated distillation
pre-trained language model

Abstract: The growing concerns and regulations surrounding the protection of user data privacy have necessitated decentralized training paradigms. To this end, federated learning (FL) is widely studied in user-related natural language processing (NLP). However, it suffers from several critical limitations including extensive communication overhead, inability to handle heterogeneity, and vulnerability to white-box inference attacks. Federated distillation (FD) is proposed to alleviate these limitations, but its performance is faded by confirmation bias. To tackle this issue, we propose Federated Interactive Distillation (FedID), which utilizes a small amount of labeled data retained by the server to further rectify the local models during knowledge transfer. Additionally, based on the GLUE benchmark, we develop a benchmarking framework across multiple tasks with diverse data distributions to contribute to the research of FD in NLP community. Experiments show that our proposed FedID framework achieves the best results in homogeneous and heterogeneous federated scenarios. The code for this paper is available at: https://github.com/maxinge8698/FedID.

FedID: Federated Interactive Distillation for Large-Scale Pretraining Language Models
Xinge Ma, Jiangming Liu; Jin Wang" and Xuejie Zhang
School of Information Science and Engineering
Yunnan University
Kumming, China
Contact: : jiangmingggnni.. www.uu.uuuuu..mmuoo edu. en, wangjin@ynu. edu.cn cn
Abstract
Bussche 2017) have imposed strict requirements on preserving user data privacy, making impractical to aggregate such data to a centralized location for training. Federated learning (FL; Mcmahan et al. 2017) has emerged as a privacy-preserviee decen- tralized training paradigm, in which federation of clients is orchestrated by a central server to collabo- ratively train shared global model via aggregating the local models trained on their respective data, As a result, the private data of massive clients is effectively exploited in the form of model param- eter exchange to train a unified model for better performance than individually working.
The growing concerns and regulations rounding the protection of user data pri- vacy have necessitated decentralized training paradigms. To this end, federated learning (FL) is widely studied in user-related natural language processing (NLP). However, it suf- fers from several critical limitations including extensive communication overhead, inability handle heterogeneity, and vulnerability to white-box inference attacks. Federated distilla- tion (FD) proposed to alleviate these limita- tions, but its performance faded by confirma- tion bias. To tackle this issue, we propose Fed- erated Interactive Distillation (FedID), which utilizes small amount of labeled data re- tained by the server to further rectify the local models during knowledge transfer. Addition- ally, based on the GLUE benchmark, we de- velop a benchmarking framework across multi- ple tasks with diverse data distributions con- tribute to the research of FD in NLP commu- nity. Experiments show that our proposed Fe- dID framework achieves the best results in ho- mogeneous and heterogeneous federated sce- narios. The code for this paper is available at: https://gt thub. cwm/ max wwwaaaxinnne88/FFe1111
Previous work on federated NLP mainly targets solving either word-level language modeling ap- plications such as mobile keyboard suggestion (Ji et al., 2019) and recommendation (Lin et al., 2020), or biomedical named entity recognition (Liu and Miller, 2020; Ge al., 2020; Sui et al., 2020). More recently, Lin et (2022) provide a research- oriented benchmarking framework for advancing FL in NLP. However, these federated NLP frame- works are limited to identical architectures across the server and clients, making it impossible for clients to design their models independently accord- ing to their inconsistent system resources and non- independent and identically distributed (non-IID) data. Also, the frequent model parameter exchange entails expensive communication costs. These ob- stacles significantly hinder the applicability and scalability FL for large-scale PLMs.
1 Introduction
The remarkable success of natural language pro- cessing (NLP) is highly dependent on large-scale pre-trained language models (PLMs; Devlin et al. 2019; Liu al. 2019; Yang al. 2019; Clark et al. 2020). To fully realize the potential of PLMs, they are typically trained using large amounts of combined data that collected from multiple dis- tributed user devices (u.k.a,, clients) and transmit- ted to a single data center (a.k.a,, server). With the growing concerns about privacy protection, data regulations such as the Personal Data Protection Act (PDPA; Chik 2013) and the General Data Pro- tection Regulation (GDPR; Voigt and von dem
Instead, federated distillation (FD) eliminates the need to share model parameters by transfer- ring knowledge from the clients to the server us- ing an unlabeled public proxy dataset (Jeong et al., 2018 8; Li and Wang, 2019; Chang et al., 2019; Gong et al., 2022; Itahara et al., 2021; Hu al., 2021), thereby allowing collaboration between heteroge- neous models with less communication costs. How- ever, FD suffers from confirmation bias (Arazo et al., 2020; Pham et al., 2021) induced by incor-
"Corresponding authors.

---

Server
Labeled Samples
Labels
Public Samples
Feedbackk
@eeverDistillttinn
Agggregation
V Validation Loss
Central Model
Ensemble Predictions
Broadccss
Broadcust
Client
Clicnt 1
Local Model
isttllDistillaoin
2 ocalProdictio
Public Samples
Public Samples
Local Samples
Labels
Figure I: The framework of FedID.
porating incorrect or even biased predictions unlabeled data for knowledge transfer, where the central model will tend to degenerate.
mogeneous and heterogeneous settings.
We present novel Federated Interactive Dis- tillation framework to mitigate the problem of misleading privileged knowledge caused by confirmation bias in conventional FD.
To tackle this challenge, we propose Federated Interactive Distillation (FedlD), where a small handful of labeled data is retained in the server, aiming provide feedback to the local models to debias the predictions.
We provide unified benchmarking frame- work across multiple NL. tasks with diverse data distributions to contribute to the research of FD in NLP community.
In addition, previous studies on FD tend de- sign diffcrent partitioning strategies on different datasets, and only small-scale models are taken into consideration, which makes it difficult to eval- uate and compare various FD approaches scaled to the NLP domain in systematic and fair man- For this reason, based the General Lan- guage Understanding Evaluation (GLUE) bench- mark (Wang et al., 2019), we create a unified bench- marking framework across multiple tasks with di- verse data distributions to simulate a variety of federated scenarios for evaluating the effectiveness of these methods on the decentralized training of large-scale PLMs, advancing the research of FD in NLP. Empirical experiments show that pro- posed FedID achieves the best results in homoge- neous and heterogeneous federated scenarios.
2
Related Work
2.1 Federated Learning
FL has gained significant interest and attention in the NLP field due to its potential for collaborative training distributed data sources while preserv- ing data privacy (Liu et al., 2021). Recent efforts have made preliminary explorations for the appli- cation of parameter averaging- FL (e.g., Fe- dAvg (Memahan et al., 2017)) in the context of NLP (Tian et al., 2022: Dong et al., 2022: Zhang et al., 2022: Lin et al., 2022). Despite some suc- cess, several system system-orient oriented challenges have to be faced to make FL widely available in NLP, includ- ing extensive communication overhead, inability to handle heterogeneity, and vulnerability to white- box inference attacks.
The contributions of this paper are summarized as follows:
To the best of our knowledge, we are the first to investigate the application of FD to decen- tralized learning of lcage-scale PLMs in ho-
Several variants of FL. have emerged to attempt to alleviate these issues. FedDF (Lin et al.) builds prototypical models with the same structure as the

---

client models on the server side enable model heterogeneity, and allows server-side ensemble dis- tillation on unlabeled data from other domains to enhance model aggregation. FedED (Sui et al., 2020) reduces uplink communication costs by up- loading the predictions of the local models instead of the parameters train the central model, but still requires broadcasting the parameters of the central model over the downlink. Accordingly, these solu- tions still rely on exchanging model parameters and therefore are unable to completely address these limitations.
subject to confirmation bias caused by trans- ferring knowledge over unlabeled data, which greatly limits their performance.
Preliminaries
3.1 Problem Definition
Consider a federated training environment with clients, where the k-th client holds labeled private dataset D = {(x y } drawn from the same or distinct distribution, along with a homogeneous or heterogeneous local model f parameterized by @ The goal is to train central model parame- terized by 0 on the server, but without direct access to these private data.
2.2 Federated Distillation
FD is a new algorithmic paradigm for FL with fun- damentally different communication properties by exchanging the knowledge obtained during the lo- cal training in the form of model outputs rather than model parameters. This shared knowledge can be an aggregated statistic of model outputs local private data (Jeong al., 2018) ble of local model outputs computed on a publicly available proxy dataset (Li and Wang, 2019; Chang et al., 2019; Gong et al., 2022; Itahara et al., 2021; Hu et al., 2021). Existing efforts on FD fall into two main categories:
3.2 Federated Learning for NLP
In a general FL framework, the training process is divided into T communication rounds through a server-client paradigm, where all clients share the same model architecture coordinated by a central server. Specifically, at the beginning of federated training, the server initializes the global model pa- rameters 0n- At each communication round f, the training is proceeded follows:
ensem-
Broadcast A subset of the client population C {1,2, K} is sampled participate in training, where |G,] E K, and æ the sampling rate. Then the server distributes the current global model parameters 0,-1 to the participating clients.
The server does not hold any model and is only used as an aggregator FedMD (Li and Wang, 2019) adopts a labeled public dataset for transfer learning among clients to seek fast improvement across all participants. Cronus (Chang et al., 2019) combines the lo- cal private dataset and the pseudo-labeled pub- lic dataset jointly for local training, where the pseudo-labels are ensembled with more robust aggregation rules.
Local training Each participating client k e C, uses the received parameters to initialize its local model,
1
(1)
The server holds central model that acts the target for collaborative train- ing FedKD (Gong al., 2022) adopts pewacy-prseerving ensemble strategy on cross-domain unlabeled data for one-way and one-shot distillation of the central model. In addition to server-side distillation, DS-FL (Ita- hara et al., 2021) also performs client-side dis- tillation using the ensemble predictions on the unlabeled public dataset. Instcad of transfer- ring ensemble of predictions, MHAT (Hu et al., 2021) achieves information aggregation by directly using predictions from multiple clients to train the central model simultanc- ously. However, these methods are generally
and updates it several epochs with its own private data D
0 + - -L--V/LCoule LCw 8
(2)
where 7 is the learning rate of the central model, and Lee denotes the loss function, which usually a categorical cross-entropy for classification tasks.
Upload The updated local model parameters of are sent back to the server.
Aggregation The server collects and aggre- gates the parameters from clients to obtain the

---

global model parameters for the next round,
4 Federated Interactive Distillation
D
(3) In existing FD approaches, the central model is only allowed to passively mimic the local models by one-way knowledge transfer, leading to confir- mation bias that heavily fades the superiority of FD. Instead of directly transmitting the entire pub- lic dataset and its predictions between the server and clients, the proposed FedID slices the unla- beled public dataset into multiple smaller batches for training, and handles only a small batch of data and predictions in each communication, which al- lows for an interaction between the central model and local models during the knowledge transfer process, while significantly reducing the load of single communication. After each server distilla- tion, the central model is allowed to feedback its performance on small amount of labeled data held by the server back to each client to adapt its local model accordingly for rectifying its confirmation bias. The overall framework of FedID is presented in Figure 1.
00
0
[D]
AEC
where [D&] and [D] Σ
[DR]
are the num-
kEC,
ber of local data held by the k-th client and all participating clients, respectively.
3.3 Federated Distillation for NLP
general FD framework, an unlabeled public [20]
dataset D [=: j;=1 is hosted by the server and
transmitted to all clients for knowledge transfer be- fore the federated training starts. At each communi- cation round 1, the training process summarized as the following steps:
Local training Each participating client trains its local model 6 on its own private data D* for several epochs,
0 - 1 0
(4)
where oft is the learning rate the k-th local model.
4.1
Server Interactive Distillation
Local prediction Each participating client computes its local predictions the entire public proxy dataset Do using its updated lo- cal model 0/.
The server samples a batch of unlabeled public data x" from D and distributes them to each participat- ing client for local prediction,
yf - f'(xx0)))
(9)
v = f 0!).
(5)
The predictions from clients are uploaded to the server and aggregated with the same strategy as Eq. (6),
Upload Participating clients upload their lo- cal predictions to the server.
D D
Aggregation The predictions from clients are collected and aggregated by the server as ensemble predictions,
(10)
kCC.
together with the batch input xo, which are adopted to train the central model for knowledge transfer,
D
YE
Yk
(6)
0g + 0, LVLCLy f(xx;(xx))) (11)
Server distillation The ensemble predictions are treated as teacher knowledge to train the central model for several epochs,
The updated central model 0, is then evaluated batch of data (xxal va)) sampled from the
labeled dataset Denl {xpaa .y.a) [Deal] the server,
Jez held by
0g + 1 VV 1)). (7)
The server broadcasts the ensem- ble predictions to participating clients.
Broadcast
LCElyvel ( ((xval;e)) A Lon(yval f(xtral, 0 VLCE( Σ [D [ ((xx)))))) kEC,
(12)
Local distillation Each participating client distills its local model using the received en- semble predictions on the entire public proxy dataset.
In addition the ensemble predictions yt: the above-computed validation loss also broadcast together to each participating client as feedback.
# + ""V
(8)

---

Algorithm 1: Federated Interactive Distillation (FedID)
Input: labeled private datasets (D*) unlabeled public dataset DO; handful of labeled dataset {
central model 0: communication rounds T D held by the server; local models {Ø Jk=
Output: decentrally trained 0
Each client initializes the local model f
2 Server initializes the central model 0g
3 for each communication round & = .... do
m <- max(s. K, 1)
C + randomly sample a subset of m clients from K clients
for each client C in parallel do Update the local model parameters Of via Eq. (4) end
for each mini-batch of unlabeled data x Do do
Server distributes mini-batch of unlaheled data x2 all participants C₁
10
11
for each client Ce parallel do Compute local predictions yf on x") via Eq. (9)
12
Upload local predictions y to the server
13
14
end
15
Server aggregates local predictions to create the ensemble predictions yr via Eq. (10)
16
Server updates the central model parameters 9g via Eq. (11)
Server samples mini-batch of labeled data (xcral vall) ~ Draa
Server computes the validation loss (y ( (xxaa, 0z)) via Eq. (12)
18
Server broadcasts the validation loss (yr(y" f(x B.)) and ensemble predictions, y: all participants C,
19
20
for each client & C in parallel do
Update the local model parameters 0f via Eqq (15)
21
22
end
23
end
end
return 0r
4.2
Client Interactive Distillation
knowledge learned by local models during local training on their respective private data to the cen- tral model, while the server-to-client interaction attempts rectify confirmation bias by allowing the local models learn from the central model's ss feedback. The detailed procedures are summarized in Algorithms 1.
For each participating client & C.. the gradients the ensemble predictions yı are computed to learn knowledge from other clients for alleviating data heterogeneity,
* Satstal
V CoE(s,,,
(13)
Also, the feedback gradients from the server to the client computed from the validation loss,
Experiments
5.1 Datasets
Fjeedback = V Cccly"al f(x val 0.)), (14)
Considering that simulating data distributions with varying heterogeneity requires sampling by labels, exclude the regression task STS-B (Cer et al.. 2017) from the GLUE benchmark (Wang al., 2019) because it lacks available labels for sam- pling, and take the remaining eight classification tasks for evaluation, including WNLI (Levesque et al., 2012), RTE (Dagan et al., 2005: Bar-Haim
and are added to further rectify its local model,
0 5 ( + Fooodbackk)
(15)
In this way, FedID establishes interactive dis- tillation between the server and clients, where the clieverto-server interaction aims to transfer the

---

WNLI
RTE
MRPC
CoLA
SST-2
QNLI
QQP
MNLI
Method
(0.6k)
(2.5k)
(3.7k)
(8.6k)
(67k)
(105k)
(364k)
(393k)
Acc
Acc
F1/Acc
Mec
Acc
Acc
F1/Acc
Ace m/mm
Centralized
56.3
61.4
85.9/79.2
52.4
92.4
90.2
86.3/89.7
82.8/83.5
a=100
FedAvg
49.6
56.7
84.5/78.0
51.0
91.8
89.8
85.5/88.2
82.0/82.7
FedDF
50.2
56.9
84.6/77.8
51.4
92.0
89.6
85.7/88.5
82.2/83.0
FedED
47.9
55.4
82.2/75.3
50.5
91.1
88.9
84.8/87.9
81.4/81.9
FedKD)
45.4
53.6
80.3/74.0
48.3
89.2
86.1
82.6/86.0
79.8/80.1
DS-FL
50.6
56.4
84.0/78.4
51.7
90.7
89.1
84.9/88.2
82.6/83.4
MHAT
50.0
56.5
84.2/78.6
51.7
91.0
89.3
84.3/88.1
82.0/83.0
FedID
51.1
57.0
84.9/79.0
52.0
91.6
89.9
85.6/88.5
82.3/83.2
a=1
FedAvg
48.3
55.1
82.9/77.2
48.9
91.0
88.7
84.2/87.6
81.44/1.8
FedDF
50.2
56.4
83.2/77.3
49.8
91.2
89.0
84.5/88.0
81.8/82.1
FedED
46.8
55.0
82.0/76.4
47.8
90.7
88.1
83.1/87.5
80.8/81.3
FedKD
44.4
52.6
80.4/74.8
46.4
88.9
86.5
82.4/86.6
78.8/79.4
DS-FI.
50.1
56.7
82.8/76.7
49.4
90.6
88.5
83.8/87.8
81.2/81.8
MHAT
50.2
56.7
82.6/76.8
49.2
90.7
88.8
84.1/87.4
81.6/82.0
FedID
50.9
56.7
83.2/77.3
49.6
90.9
88.8
84.7/88.1
81.9/82.3
Table 1: Experiment results of the homogeneous setting on the GLUE dev sets.
et al., 2006; Giampiccolo et al., 2007; Bentivogli al., 2009), MRPC (Dolan and Brockett, 2005), CoLA (Warstadt al., 2019), SST-2 (Socher et al.. 2013), QNLI (Rajpurkar et al., 2016), QQP!, and MNLI (Williams et al., 2018). See Appendix for more details about GLUE.
tion (Hsu et al., 2019), where p is prior class dis- tribution over N classes, and is concentration parameter that controls the degree of data hetero- geneity among clients. Typically, when - co, clients tend to be assigned to the identical data dis- tribution, and conversely, when a - 0, clients are more likely to hold examples from only one ran- dom class. In our experiments, we set to 100 and to generate IID and non-llD data, respectively.
For each task, the original development set is employed evaluate the performance of the cen- tral and local models, while the original training set is divided into private and public datasets at a ratio of 1:1, which are used for client training and knowledge transfer between the server and clients, respectively. Particularly, for the resulting public dataset, we further sample 10% of it as the labeled dataset reserved for the server, and the as the unlabeled public dataset after rounding off labels. Furthermore, to create disjoint client training data from the private dataset, the training instances of each client are drawn independently with class labels following categorical distribution N classes parameterized by vector q (g: 2 i e N]. and ||q/] = 1). Mcanwhile, to simu- late varying data distributions for clients, we fur- ther draw Dir(ap) from a Dirichlet distribu-
5.2 Settings
Homogeneous setting For a homogeneous fed- erated scenario, the model architectures of clients limited to be the be compatible with FL methods for comparison, we adopt BERT-base (Devlin al., 2019) as the central model since FL cannot usually be applied to larger PLMs due to communication bottlenecks.
that of the server. To
Heterogeneous setting For a heterogeneous federated scenario, the central model initialized with BERT-base, while each local model selected from BERT-base, BERT-large, RoBERTa base (Liu et al., 2019), or RoBERTa-large"
"If not specified, a used by default. IIf not specified, heterogencous setting is used by default.
1https: //oorr/////oraaataa www.aattt.qurr.cooomm

---

WNLI
RTE
MRPC
CoLA
SST-2
QNLI
QQP
MNLI
Method
(0.6k)
(2.5k)
(3.7k)
(8.6k)
(67k)
(105k)
(364k)
(393k)
Acc
Acc
F1/Acc
Mec
Acc
Acc
F1/Acc
Acc m/mm
a=100
FedKD
54.1
60.5
85.2/80.5
53.6
90.8
87.7
84.2/87.2
81.9/82.3
DS-FL
56.8
63.4
87.7/82.4
55.4
92.1
89.9
86.5/89.4
83.6/84.0
MHAT
56.9
63.2
87.7182.5
55.7
92.0
90.6
87.0/89.5
84.0/84.2
FedID
58.2
64.6
88.5/83.6
56.5
92.7
91.4
88.1/91.2
84.6/84.6
a=1
FedKD
52.7
58.4
84.3/80.0
50.0
88.9
87.1
84.2/87.5
81.2/81.6
DS-FL.
55.6
60.3
86.6/81.6
53.9
90.9
89.8
86.4/89.1
83.0/83.5
MHAT
55.9
60.8
86.6/81.5
53.8
91.2
90.3
86.6/89.3
83.4/83.7
FedID
57.0
61.5
87.7/82.2
54.9
91.8
91.0
87.6/90.5
84.2/84.2
Table 2: Experiment results of the heterogeneous setting the GLUE dev sets.
5.3
Implementation Details
change for better privacy protection. However, this performance gap is gradually alleviated as the train- ing data increases. In addition, the performances of FL and FD models are significantly degener- ated when encountering non-lID data. Also, when sufficient public data is made available, the perfor- mance of FD models be comparable that of FL models, accompanied by lower communication costs.
We adopt the AdamW optimizer (Loshchilov and Hutter, 2019) with initial learning rate of 26-5 to update the model parameters. For single-sentence or sentence-pait input to the model, the maximum sequence length is set to 128, and the batch size set to 32. For hyperparameters in federated train- ing, the number of epochs for local training, local distillation, and server distillation is set to 3, 3, and 3, respectively, the number of clients K' is set to 10, the fraction of client sampling is set to 1, and the number of communication rounds T is set to 10.
Heterogeneous setting Table 2 shows the per- formances across models in heterogeneous setting. The proposed FedID outperforms other baselines, demonstrating the superiority of tackling the con- finmation bias. In particular, FedID exhibits strong robustness when only small amount of training data is available, there is not enough private data to adequately train the local models and thus the confirmation bias becomes more pronounced.
5.4 Baselines
We compare FedID with FL algorithms including FedAvg (Memahan et al., 2017), FedDF (Lin et al.), and FedED (Sui et al., 2020), well FD al- gorithms including FedKD (Gong et al., 2022), MAHT (Hu et al., 2021), and DS-FL (Itahara et al., 2021). We also provide the models with centralized training (denoted as Centralized) that have access to all private data held by the clients as an upper bound on model performance.*
Cross-domain setting We also use the origi- nal training sets of IMDB (Maas et al., 201 and PAWS (Zhang et al., 2019) as unlabeled public data for SST-2 and QQP, respectively, to construct cross- domain knowledge transfer environments, where the confirmation bias is more likely to occur. The experimental results the dev sets of SST-2 and QQP are shown in Table 3, where the greater per- formance gap between FedID and other baselines further confirms our claim.
5.5 Results
Homogeneous setting Table 1 shows the per- formances across models in homogeneous setting. Without considering data privacy, centralized mod- els always exhibit the best performance, while decentralized models sacrifice performance ex-
5.6 Ablation Study
'FedED and MHAT require a labeled public dataset in the original paper. For a fair comparison, version using the unlabcled public dataset is provided in our implementation.
We remove the feedback gradient and the knowl- edge transfer gradient from Eq. (15), respectively,

---

SST-2 - IMDB
QQP PAWS
Method
Formulation
Method
Acc
F1/Ace
FedAvg
(10) x (K + 1)) T
FedKD
81.9
80.0/83.5
FedDF
(10) x (K + + X)
DS-FL
83.3
82.3/85.4
FedED
(10) x [
MHAT
83.9
82.8/85.6
FedKD
FedID
86.4
84.1/87.1
DS-FL
([Do) (K)+ XT
MHAT
([29) x (K +1) x
Table Results of models in the cross-domain setting.
FedID
([29) (K+
RTE
SST-2
QQP
Table 5: Formulations of communication costs.
Method
Ace
Acc
F1/Aec
FedID
61.5
91.8
87.6/90.5
public dataset. To further investigate the effect of different proportions of the public dataset per- formance, we keep the size of the private dataset constant while conducting experiments using 10%, 20%, 40%, 80%, and 100% of the public dataset, respectively. The results in Figure show that the performance of the central model improves to some extent as the size the public dataset increases, where FedID still exhibits superior performance and robustness.
w/o feedbback
60.1
90.7
86.5/89.4
w/o Fdistall
60.6
91.1
86.9/89.8
Table 4: Results of model ablations on RTE, SST-2, and QQP.
to conduct ablation experiments the small dataset RTE, the medium dataset SST-2, and the large dataset QQP. As shown in Table 4, without the feedback gradient or the knowledge transfer gradient, the performances models get worse, where the feedback gradient contributes more.
5.9 Effect of Labeled Dataset Size
To investigate the effect of the size of the labeled dataset retained by the server on performance, we experiment with 10%, 20%, 40%, 80%, and 100% of the labeled dataset, respectively. For FedID, the labeled data is used to rectify the confirmation bias in the client models' predictions, while for other FD methods, the labeled data is added to the train- ing of the central model. The results in Figure 3 show that FedID is least sensitive to the size of the labeled dataset since this data is not used to directly participate in the training of the central model. Moreover, although other FD methods use the labeled data directly for additional training of the central model, there is still no significant per- formance improvement observed because the pro- portion of the labeled data is far lower than that the unlabeled data, and thus its supervision on the central model is limited. Our solution makes better use of the small amount of labeled data by leverag- ing it rectify confirmation bias in the predictions from unlabeled data.
5.7 Communication Cost
Communication costs between the server and client models across baselines are presented in Table 5. The communication costs FedAvg, FedDF, and FedED are much higher than those of FedKD, DS-FL, MHAT, and FedID as they entail exten- sive communication share the model parameters. FedKD exhibits the lowest communication costs the clients' predictions the need to send back to the clients and only one round of communication is executed, while DS-FL and MHAT need to broadcast the ensemble pre- dictions from the server back to each client. Simi- larly, FedID is required to transmit ensemble pre- dictions and the validation loss in batches to clients feedback, but the communication costs for the validation loss negligible compared that of ensemble predictions. As a result, the communica- tion costs of FedID remain in line with DS-FL and MHAT, but the communication between the server and clients is more frequent.
aggregated without
5.10 Effect of Number of Clients
The number of clients usually imposes a signifi- cant impact on performance, as the entire training dataset is partitioned and distributed to multiple clients. To investigate this, we increase the num- ber of clients from 5 to 10 and 20 while keeping
5.8 Effect of Unlabeled Public Dataset Size
our experimental setup, we partition the orig- inal training dataset into a private dataset and a

---

FedKD
93
reakD
DS-FL
MIИT
92
NHAT
TedlD
Vecorrmnee
86
82-
20
40
60
100
10
20
Rabio of Public Dataset (6)
Number of Elients
Figure 2: Performance of the central model SST-2 with different sizes of public dataset.
Figure 4: Performance of the central model on SST-2 with different numbers of clients.
entially private local predictions for stronger pri- vacy guarantee, enhancing the resilience of FedID against malicious server or clients.
90
Performeno
Limitations
There two main limitations to work com- pared previous efforts: 1) We assume that small amount of labeled data is retained in the server. However, this situation may be common in real life. For instance, an institution possesses only a small amount of training data, which not enough to train well-performing model, thus it may want to resort to collaborative training with other insti- tutions with the help of FD on large amount of unlabeled public data. However, directly transfer- ring knowledge on the unlabeled data may not yield satisfactory performance, while the small amount of training data retained by the institution can be used as labeled data by the proposed FedID to max- imize the performance. In addition, our approach is more suitable for the case where one client in the federation acts the server; 2) Compared with other FD approaches, beled public dataset into multiple smaller batches for training, thus entailing more frequent commu- nication between the server and clients. However, the increase in communication frequency may be tolerable considering the similar communication costs and the fact that transmitting smaller pack- ets avoids potential network congestion when the public dataset is too large.
FedED
87
FeelD
Figure 3: Performance of the central model on SST-2
with different sizes of labeled dataset.
the total amount of private data for all clients con- stant, which results in corresponding change in the quantity of local private data that can be allo- cated each client. The results depicted in Fig- ure 4 show a decrease in the performance of the central model as the number of clients increases, which can be attributed to the fact that the reduc- tion local private data makes it challenging to achieve adequate local training. However, FedID) shows better robustness in response to this change due to the mitigation of confirmation bias.
solution slices the unla-
6 Conclusions
This study explores the application of FD to de- centralized training of large-scale PLMs in homo- geneous and heterogeneous settings, and further presents interactive FD scheme to mitigate the confirmation bias caused by transferring knowl- edge on an unlabeled public dataset. Moreover, benchmarking framework across multiple tasks with diverse data distributions developed to con- tribute to the research of FD in NLP community. Future work will executed to aggregate differ-
Ethics Statement
This study aims explore alternative decentral- ized training paradigm for large-scale PLMs, and the proposed method does pose ethical issues or po- tential biases. All models, baselines, and datasets

---

used in this work are publicly available and widely used.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguisties: Human Language Technolo- gies (NAACL-HHLT 2019), pages 4171 1-71-4186.
Acknowledgements
This work was supported by the National Natural Science Foundation of China (NSFC) under Grant Nos.61966038 and 62266051, and the Postgradu- ate Research and Innovation Foundation of Yunnan University under Grant No. 1-233333365. The au- thors would like thank the anonymous reviewers for their constructive comments.
William Dolan and Chris Brockett. 2005. Automati- cally constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP 2005), pages 9-16.
Chenhe Dong, Yuexiang Xie, Bolin Ding, Ying Shen, and Yaliang Li. 2022. Collaborating heteroge- neous natural language processing tasks via feder- ated learning- arXiv preprini 2IX.v:221..05789.
References
Eric Arazo, Diego Ortego, Paul Albert, Noel E. O'Connor, and Kevin McGuinness. 2020. Pseudo- labeling and confirmation bias deep semi- supervised learning. In Proceedings of the Interna- tional Joint Conference on Neural Networks (IJCNN 2020).
Suyu Ge, Fangzhao Wu, Chuhan Wu, Tao Qi. Yongfeng Huang, and Xing Xie. 2020. Fed- ner: Privacy-preserving medical named entity recog- nition with federated learning. arXiv preprint
NVV:20033009888
Danilo Giampiccolo, Bernardo Magnini. Ido Dagan, and Bill Dolan. 2007. The third pascal recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pages 1-9.
Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. 2006. The second pascal recognising tex- tual entailment challenge. In Proceedings of the Sec- ond PASCAL Challenges Workshop on Recognising Textual Entailment.
Xuan Gong, Abhishek Sharma, Srikrishna Karanam, Ziyan Wu, Terrence Chen, David Doermann, and Arun Innanje. 2022. Preserving privacy in feder- ated learning with ensemble cross-domain knowl- edge distillation. In Proceedings of the 36th AAAI Conference on Artificial Intelligence (AAAI 2022). pages 11891-11888 1899.
Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. 2009. The fifth pascal recognizing textual entailment challenge. In Proceedings of the Third Text Analysis Confer- ence.
Daniel Cer, Mona Diab, Eneko Agirre, lfiig Lopez- Gazpio, and Lucia Specia. 2017. Semeval-2017 task 1: Semantic textual similarity multilingual and cross- lingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evalua- tion (SemEval 2017), pages 1-14.
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. 2019. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.065
Li Hu, Hongyang Yan, Lang Li. Zijie Pan, Xi- aozhang Liu, and Zulong Zhang. 2021. Mhat: An efficient hnterl-hetroogenous aggregation training scheme for federated learning. Information Sci- ences, 560:493-503.
Hongyan Chang, Virat Shejwalkar, Reza Shokri, and Amir Houmansadr. 2019. Cronus: Ro- bust and heterogeneous collaborative learning with black-box knowledge transfer. arXiv preprint arXiv:19/2.11279.
Sohei Itahara, Takayuki Nishio. Yusuke Koda, Masahiro Morikura, and Koji Yamamoto. 2021. Distillation- semi-supervised federated learn- ing for cntpuunicatiooeffciiieteo collaborative train- ing with non-iid private data. IEEE Transactions on Mobile Computing, 22(1):191-205,
Warren B. Chik. 2013. The singapore personal data protection act and assessment of future trends in data privacy reform. Computer Law and Security Review, (955)555-5575
Kevin Clark, Minh-Thang Luong, Quoe V.Le, and Christopher D.Manning. 2020. Electra: Pre-training text encoders discriminators rather than genera- tors. In Proceedings of the 8th International Confer- ence on Learning Representations (ICLR 2020).
Funjeong Jeong. Seungeun Oh, Hyesung Kim, Ji- hong Park, Mchdi Bennis, and Seong-Lyun Kim. 2018. Communication--fficieett on-device ma- chine learning: Federated distillation and augmen- tation under non-iid private data. arXiv preprint arXiv:1811.11479
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment challenge. In Machine Learning Challenges Work- shop, pages 177-190.
Shaoxiong Ji, Shirui Pan, Guodong Long, Xue Li, Jing Jiang, and Huang. 2019. Learning private neural language modeling with attentive aggregation. In

---

Proceedings of the International Joint Conference on Neural Networks (IJCNN 2019).
Hieu Pham, Zihang Dai, Qizhe Xie, and Quoc V. Le. 2021. Meta pseudo labels. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition (CVPR 2021), pages 11557- 11568.
Hector J. Levesque, Ernest Davis, and Leora Morgen- stern. 2012. The winograd schema challenge. In Proceedings of the Thirteenth International Confer- ence on Principles of Knowledge Representation and Reasoning (KR 2012), pages 552-561.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirieal Methods in Nat- ural Language Processing (EMNLP 2016), pages 2383 2392.
Daliang Li and Junpu Wang. 2019. Fedmd: Heteroge- nous federated learning via model distillation. arXiv preprint 1910003910
Bill Yuchen Lin, Chaoyang He, Zihang Zeng, Hulin Wang, Yufen Huang, Christophe Dupuy, Rahul Gupta, Mahdi Soltanolkotabi. Xiang Ren, and Salman Avestimehr. 2022. Fednlp: Benchmark- ing federated learning methods for natural language processing tasks. In Findings of the Association for Compulational Linguisties: NAACL 2022. pages 157-175.
Richard Socher, Alex Perelygin, Jean Y. Wu, Jason Chuang. Christopher D. Manning, Andrew Y. Ng, and Christopher Potis. 2013. Recursive deep mod- els for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP 2013), pages 1631-1642.
Guanyu Lin, Liang Feng. Pan Weike, and Ming Zhong. 2020. Fedrec: Federated recommendation with ex- plicit feedback. IEEE Intelligent Systems, 36(5):21- 30.
Dianbo Sui, Yubo Chen, Jun Zhao, Yantao Jia, Yuantao Xie, and Weijian Sun. 2020. Feded: Federated learn- ing via ensemble distillation for medical relation ex- traction. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP 2020), pages 21 218822128.
Tao Lin, Linging Kong. Sebastian U. Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. In Advances in Neural Infor- mation Processing Systems (NeurIPS 2020), pages 2351-2363.
Yuanyishu Tian, Yao Wan, Lingjuan Lyu, Dezhong Yao, Hai Jin, and Lichao Sun. 2022. Fedbert: When federated learning meets pre-training, ACM Transactions on Intelligent Systems and Technology. 13(4):1-26.
Dianbo Liu and Tim Miller 2020. Federated pretrain- ing and fine tuning of bert using clinical notes from multiple silos. arXiv preprint arXiv:2002.0855
Paul Voigt and Axel von dem Bussche. 2017. The eu general data protection regulation (gdpr). A Prac- tical Guide, 1st Ed., Cham: Springer International Publishing.
Ming Liu, Stella Ho. Mengqi Wang, Longxiang Gao, Yuan Jin, and He Zhang. 2021. Federated learning meets natural language processing: A survey. arXiv preprint arXiv: 207.122307..
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. Glue: A multi-task task benchmark and analysis platform for natural language understanding. In Proceed- ings of the 7th International Conference on Learn- ing Representations (ICLR 2019).
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint 19ti::9997.1660
Alex Warstadt, Amanpreet Singh, and Samuel R. Bow- man. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled
weight decay regularization. In Proceedings of the 7th International Conference on Learning Represen- tations (ICLR 2019).
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Pouts. 2011. Learning word vectors for sentiment analy- sis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics (ACL 2011), pages 142-150.
Adina Williams, Nikita Nangia, and Samuel R. Bow- man. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies (NAACL-HLT 2018), pages 1112-1122.
H Brendan Memahan, Eider Moore. Daniel Ramage. Seth Hampson, and Blaise Aguera y Areas. 2017, Communication-effiicenn learning of deep networks from decentralized data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS 2017), pages 1273-1283.
Zhilin Yang. Zihang Dai, Yiming Yang. Jaime Car- bonell, Ruslan Salakhutdinov, and Quoc Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. Advances Neural formation Processing Systems (NeurIPS 2019).

---

Yuan Zhang, Jason Baldridge, and Luheng He. 2019. Paws: Paraphrase adversaries from word scrambling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- lional Linguisticx: Human Language Technologies (NAACI-HHLT 2019), pages 1298-1300
Dataset
#Train
#Dev
#Test
WNLI
635
71
146
RTE
2490
277
3000
MRPC
3668
408
1725
STS-B
5749
1500
1379
Zhuo Zhang, Xiangjing Hu, Lizhen Qu, Qifan Wang. and Zenglin Xu. 2022. Federated model decompo- sition with private vocabulary for text classification. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP 2022), pages 6413 1-4-55...
CoLA
8551
1043
1063
SST-2
67349
872
1821
QNLI
104743
5463
5463
QQP 363846
40430
390965
MNLI-m 392702
9815
9796
MNLI-mm
9832
9847
A Details of GLUE Benchmark
GLUE (Wang et al., 2019) is a benchmarking framework designed to assess the performance and generalization capability of NLP models especially large-scale PLMs across nine NLP tasks. The de- scriptions of each task presented follows:
Table 6: Statistics of the GLUE benchmark.
model to determine whether a given English sentence is grammatically correct, with the evaluation metric of the Matthews correlation.
WNLI The Winograd Natural Language In- ference (Levesque et al., 2012) is a sentence- pair binary classification task that requires the model to determine whether two sentences in given sentence-pair entailment relations, with the evaluation metric of accuracy.
SST-2 The Stanford Sentiment Tree- bank (Socher et al., 2013) a single-sentenee binary classification task that requires the model to determine whether a given movie review is positive or negative in sentiment, with the evaluation metric of accuracy.
RTE The Recognizing Textual Entail- ment (Dagan al., 2005; Bar-Haim et al., 2006; Giampiccolo al., 2007; Bentivogli et al., 2009) is sentence-pair binary clas- sification task, which requires the model to determine whether two sentences in a given sentence pair are entailment relations, with the evaluation metric of accuracy.
QNLI The Question Natural Language Infer- ence (Rajpurkar et al., 2016) is sentence-pan binary classification task. Given question and context, the model is required to deter- mine whether the context contains the answer the question, with the evaluation metric of accuracy.
QQP The Quora Question Pairs is sentence- pair binary classification task. Given pair of questions, the model required determine whether the two sentences are semantically equivalent, with evaluation metrics of accu- racy and Fi-score.
MRPC The Microsoft Research Paraphrase Corpus (Dolan and Brockett, 2005) is a sentence-par pair binary classification task that requires the model to determine whether two sentences in given sentence pair seman- tically equivalent, with evaluation metrics of accuracy and Ficccoe.
The Multi-genre Natural Language Inference (Williams et al., 2018) is sentence- pair three-way classification task. Given a premise and a hypothesis, the model is re- quired to determine whether the hypothesis is an entailment, contradiction, or neutral with respect to the premise. The task is divided into matched and mismatched versions, with evaluation metrics of matched accuracy and mismatched accuracy, respectively.
MNLI
STS-B The Semantic Textual Similarity Benchmark (Cer et al., 2017) is sentence- pair regression task that requires the model to evaluate how similar two sentences in given sentence-pair are by floating score range from 0 to 5, with evaluation metries of Pearson and Spearman correlations.
CoLAA The Corpus of Linguistic Acceptabil- ity (Warstadt et al., 2019) is a single-sentence binary classification task that requires the
The statistics of these tasks are presented in Ta-
ble


Reviews:
[{"review_id": "m8V8O9eD9m", "reviewer": "Reviewer_kBJp", "summary": "", "questions": "There is no report on hyperparameter settings and code, how to ensure the reproducibility of the paper？", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We highly appreciate your insightful comments and suggestions. Each comment has significantly improved the quality of the paper. \n\n**Reasons To Reject 1:** The homogeneous and heterogeneous problems in FD of PLM do not seem to be a contribution, so the contribution seems to be a bit weak.\n\n**Response:** Recent federated work in NLP only considers the decentralized training of homogeneous PLMs, but does not explore how the heterogeneous PLMs perform decentralized training. Therefore, we mainly focus on investigating FD in homogeneous and heterogeneous large-scale PLMs. Thanks again for your suggestion about the data heterogeneity on different modalities, our future work will explore heterogeneous federated learning in multi-modality settings.\n\n**Reasons To Reject 2:** Although many published federated learning papers lack theories, if there are some more important analyzes such as errors, it will make the papers more solid.\n\n**Response:** We will consolidate this paper with more perspectives on empirical risk minimization, convergence analysis, and generalization bounds.\n\n**Reasons To Reject 3:** Lack of any reproducible content, including code, experiment/hyperparameter settings.\n\n**Response:** In addition to the implementation details provided in Section 5.3, due to the page limitation, the codes and more experimental setups are illustrated in the \"Supplementary Materials\". If there are some spaces left, we will present more experimental settings in the final version. In addition, we will also make the codes publicly available.\n\n**Questions For The Authors 1:** There is no report on hyperparameter setting and code, how to ensure the reproducibility of the paper?\n\n**Response:** Please refer to **Reasons To Reject 3**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. It is a good idea to solve the homogeneous and heterogeneous in federated distillation through some labeled data of the server\n2. The author has done sufficient experiments to verify the effectiveness of the method", "reasons_to_reject": "1. The homogeneous and heterogeneous of PLM and server are the homogeneous and heterogeneous faced by other models. It seems that there is not much difference. As the author said, when the server model is PLM, the data can be reasonably divided into public data on the server and private data on the client, but it can also be used in other FDs in visual scenes, recommendation scenes, and audio scenes. Therefore, the first exploration of the server model is that the homogeneous and heterogeneous problems in FD of PLM do not seem to be a contribution, so the contribution seems to be a bit weak.\n2. Although many published federated learning papers lack theories, if there are some more important analyzes such as errors, it will make the papers more solid\n3. Lack of any reproducible content, including code, experiment/hyperparameter settings,", "questions_to_authors": "There is no report on hyperparameter settings and code, how to ensure the reproducibility of the paper？", "paper_topic_and_main_contributions": "The authors investigated FD in homogeneous and heterogeneous large-scale pre-trained language models and proposed a novel Federated Interactive Distillation to mitigate the problem of misleading privileged knowledge caused by confirmation bias in conventional FD.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4r60k5CkoC", "reviewer": "Reviewer_aUk1", "summary": "", "questions": "Given the interactive nature of FedID, it leads to a higher communication overhead - so please elaborate on the below statement: \n\"but the communication cost of validation loss is negligible compared to that of ensemble predictions.\"\nIt seems that the authors are focusing primarily on the transmitted data size, and not the communication rounds. Please elaborate.\n\nOn what basis were the 8 GLUE classification tasks selected?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly value your perceptive remarks and recommendations, which contribute a lot to improving the quality of the paper. \n\n**Reasons To Reject 1:** FedID is more of an incremental contribution over existing Federated Distillation (FD) approaches, with a different distribution mechanism. So the strength of paper is primarily on the applied part.\n\n**Response:** In existing FD approaches, the central model can only passively mimic the local models, which results in the confirmation bias that heavily fades the superiority of FD. Different from the existing approaches, the proposed FedID establishes a link between the central model and the local models by introducing an interaction mechanism. This interaction allows the central model to provide timely performance feedback to adjust the local models for progressively correcting confirmation bias, thereby improving performance. We will highlight this contribution in the introduction section.\n\n**Reasons To Reject 2:** For Federated Learning contributions, it is always good to assess how resilient it is to malicious clients/server. So I would encourage the authors to add this discussion.\n\n**Response:** In the current version, we focused mainly on how to improve the performance of FD and therefore neglected to protect against malicious attacks from the server or clients. Thanks to your suggested references, we identify that FedID can be well integrated with the Private Aggregation of Teacher Ensemble (PATE) to compensate for this deficiency by aggregating differentially private local predictions for a stronger privacy guarantee. We will perform additional experiments about resilience to malicious attacks to demonstrate this improvement. Thanks again for your recommendation.\n\n**Questions For The Authors 1:** Given the interactive nature of FedID, it leads to a higher communication overhead – so please elaborate on the below statement: \"but the communication cost of validation loss is negligible compared to that of ensemble predictions\". It seems that the authors are focusing primarily on the transmitted data size, and not the communication rounds. Please elaborate.\n\n**Response:** We will reformulate the communication cost in Section 5.7 by considering the transmitted data size and the communication rounds together. As a result, the communication cost of FedID will higher due to more frequent communication between the server and clients. However, as we discussed in the limitations, the increase in communication frequency may be tolerable due to the fact that FedID transmits smaller packets avoiding potential network congestion if the public dataset is too large. This is another merit.\n\n**Questions For The Authors 2:** On what basis were the 8 GLUE classification tasks selected?\n\n**Response:** It is well known that GLUE contains 8 classification tasks and 1 regression task. Considering we use the Dirichlet distribution to simulate data distribution with different heterogeneity, which samples by label, thus we selected all classification tasks and excluded the regression task because it lacks available labels for sampling.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The FedID approach is generic and can potentially be applied to train non-NLP models as well in a Federated Learning setting. In this paper, the authors specifically apply it to PLMs and show their efficiency and performance benefits for NLP tasks - making it relevant for EMNLP.\n\nThe experimental results are comprehensive providing interesting insights around public dataset distribution between clients and communication aspects. The heterogenous setting is of course more interesting from a 'confirmation bias' perspective, where the authors show FedID's robustness when only a small amount of training data is available that are insufficient to train the local models.", "reasons_to_reject": "FedID is more of an incremental contribution over existing Federated Distillation (FD) approaches, with a different distribution mechanism. So the strength of paper is primarily on the applied part.\n\nFor Federated Learning contributions, it is always good to assess how resilient it is to malicious clients/server. So I would encourage the authors to add this discussion.", "questions_to_authors": "Given the interactive nature of FedID, it leads to a higher communication overhead - so please elaborate on the below statement: \n\"but the communication cost of validation loss is negligible compared to that of ensemble predictions.\"\nIt seems that the authors are focusing primarily on the transmitted data size, and not the communication rounds. Please elaborate.\n\nOn what basis were the 8 GLUE classification tasks selected?", "paper_topic_and_main_contributions": "The authors propose a Federated Learning variant - called Federated Interactive Distillation (FedID) - for Pre-trained Language Models (PLMs). They then validate its performance with extensive experimental results on GLUE tasks for both homogeneous and heterogeneous model settings.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Please consider:\nN. Papernot, M. Abadi, U. Erlingsson, I. Goodfellow, and K. Talwar. Semisupervised Knowledge Transfer for Deep Learning from Private Training Data. arXiv, abs/1610.05755, 2017.\nin the context of FD where the authors show how multiple (teacher) nodes trained on sensitive data, can further train a (student) model based on differentially private aggregated outputs of the teacher nodes."}}, {"review_id": "MExaeE6LHA", "reviewer": "Reviewer_VAtG", "summary": "", "questions": "1. Are there any design intuitions behind the interaction procedure? Mentioning the intuition in the introduction could help readers better understand the paper.\n2. With various types of tasks in GLUE being evaluated, are there any insights in the influence of task type in whether interactive distillation is useful?\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply value your insightful comments and suggestions. Each comment has significantly enhanced the quality of the paper. \n\n**Reasons To Reject 1:** Lack case-level comparison to analyze/justify in detail why the proposed method alleviates confirmation bias.\n\n**Response:** We will add a case study section to verify that the proposed method alleviates confirmation bias by calculating whether local model predictions are more accurate after each interaction than when there is no interaction.\n\n**Reasons To Reject 2:** Lack of comparison with alternative designs (e.g., for server-to-client interaction, can we also use the proxy dataset instead of the gradient?)\n\n**Response:** We will rich the ablation study in Section 5.6, to examine the superiority of using the feedback gradient compared to the proxy dataset in mitigating confirmation bias.\n\n**Reasons To Reject 3:** Lack real-world use case analysis for the proposed training scenario (in the limitation section the author explains a constructed scenario which is reasonable, but still it would be informative to find concrete use case)\n\n**Response:** The proposed FedID can be applied to many real-world cases. For example, an accurate medical text classification system requires the proposed training scenario, to quickly predict which category of disease a patient belongs to based on his/her symptoms for assigning the appropriate departments and specialists in a timely and convenient manner. Specifically, a hospital usually has only a small amount of training data, which is not enough to train a well-performing model, thus it may want to resort to the private data and models from other hospitals with the help of FD on a large amount of unlabeled public medical data. The small amount of training data retained by the hospital can be used as labeled data by the proposed FedID to maximize the performance. We will elaborate on this use case analysis in the limitation section.\n\n**Questions For The Authors 1:** Are there any design intuitions behind the interaction procedure? Mentioning the intuition in the introduction could help readers better understand the paper.\n\n**Response:** Yes. In existing FD approaches, the central model only can passively mimic the local models, which results in the confirmation bias that heavily fades the superiority of FD. Different from the existing approaches, the proposed FedID establishes a link between the central model and the local models by introducing an interaction mechanism. This interaction allows the central model to provide timely performance feedback to adjust the local models for progressively correcting confirmation bias, thereby improving performance. We will also add this intuition to the introduction to assist readers in understanding our work.\n\n**Questions For The Authors 2:** With various types of tasks in GLUE being evaluated, are there any insights in the influence of task type in whether interactive distillation is useful?\n\n**Response:** Yes, Tables 1 and 2 show that interactive distillation has subtle discrepancies across tasks. Overall, for tasks with more data, the effect of interaction distillation will be smaller because the client has enough data for local training, in which the problem of confirmation bias will be less obvious. For tasks with more labels, interactive distillation is more useful because more labels mean that confirmation bias is more likely to occur. We will also add these findings to Section 5.5 to provide more interpretation for the experimental results.\n\n**Typos Grammar Style And Presentation Improvements 1:** The introduction does not provide enough context about FD to understand the general contribution. The authors may consider adding a summarized version of section 2.2 to the introduction to help readers better understand what is being investigated in this paper.\n\n**Response:** We will add more details in the introduction to provide readers with more preliminaries about federal distillation as well as its recent efforts and limitations.\n\n**Typos Grammar Style And Presentation Improvements 2:** Table 4: feedback should be subscripted.\n\n**Response:** We have rechecked the paper and corrected these typos.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Useful privacy-preserving training method with good performance\n2. Reasonable and effective algorithm design\n3. Comprehensive experiment to show the effectiveness of the method\n", "reasons_to_reject": "1. Lack case-level comparison to analyze/justify in detail why the proposed method alleviates confirmation bias\n2. Lack of comparison with alternative designs (e.g. for server-to-client interaction, can we also use the proxy dataset instead of the gradient?)\n3. Lack real-world use case analysis for the proposed training scenario (in the limitation section the author explains a constructed scenario which is reasonable, but still it would be informative to find concrete use case)\n", "questions_to_authors": "1. Are there any design intuitions behind the interaction procedure? Mentioning the intuition in the introduction could help readers better understand the paper.\n2. With various types of tasks in GLUE being evaluated, are there any insights in the influence of task type in whether interactive distillation is useful?\n", "paper_topic_and_main_contributions": "The paper proposes a new training algorithm in the federated distillation setting. The proposed approach uses a public proxy dataset for collecting client knowledge and reflects the gradient of the small labeled dataset to mitigate the problem of confirmation bias. Experiments on various GLUE tasks show that the proposed method is effective and outperform many baselines while being robust.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The introduction does not provide enough context about FD to understand the general contribution. The authors may consider adding a summarized version of section 2.2 to the introduction to help readers better understand what is being investigated in this paper.\nTable 4: feedback should be subscripted.\n"}}]

Author rebuttal:

Meta-review: Thanks reviewers so much your efforts in providing comprehensive reviews and comments to improve the paper.

Thanks authors for providing actionable rebuttals and facilitating discussions.

Summary: The authors conducted research on Federated Distillation (FD) in large-scale pre-trained language models, both homogeneous and heterogeneous. They introduced Federated Interactive Distillation (FedID) to address confirmation bias issues in FD. FedID is a Federated Learning variant for Pre-trained Language Models (PLMs) and was tested extensively on GLUE tasks in various model settings. Their approach includes a new training algorithm that uses a public proxy dataset to collect client knowledge and adjusts the gradient of the small labeled dataset to combat confirmation bias. Experimental results demonstrated the effectiveness and robustness of their proposed method, which outperformed several baseline approaches in different GLUE tasks.

Average Soundness: (3+4+5)/3 = 4
Average Excitement: (3+4+4)/3 = 3.7
Reproducibility: (3+4+4)/3 = 3.7

Summary of Pros:
+ Solve the homogeneous and heterogeneous in federated distillation through some labeled data of the server
+ Comprehensive experiments providing interesting insights around public dataset distribution between clients and communication aspects
+ Relevant for EMNLP.
+ Useful privacy-preserving training method with good performance
+ Reasonable and effective algorithm design

Summary of Cons:
+ Lack theories
+ Lack of reproducible content
+ Contribution is incremental
+ Lack of discussion on resilience
+ Need more comparisons such as case-level analysis, alternative designs, real-world use case analysis

The authors have clarified and promised the changes to tackle the cons. The changes are feasible and can be timely ready for the final version.
Decision: Accept-Main