Exploit Unlabeled Data on the Server! Federated Learning via Uncertainty-aware Ensemble Distillation and Self-Supervision
Authors: Jae-Min Park
Won-Jun Jang
Tae-Hyun Oh
Si-Hyeon Lee
Venue: Submitted to ICLR 2023 (2023)
Keywords: Federated Learning
Knowledge Distillation
Ensemble Distillation
Self-supervised Learning
Uncertainty

Abstract: Federated Learning (FL) is a distributed machine learning paradigm that involves the cooperation of multiple clients to train a server model. In practice, it is hard to assume that each client possesses large-scale data or many clients are always available to participate in FL for the same round, which may lead to data deficiency. This deficiency degrades the entire learning process. To resolve this challenge, we propose a Federated learning with entropy-weighted ensemble Distillation and Self-supervised learning (FedDS). FedDS reliably deals with situations where not only the amount of data per client but also the number of clients is scarce. This advantage is achieved by leveraging the prevalent unlabeled data in the server. We demonstrate the effectiveness of FedDS on classification tasks for CIFAR-10/100 and PathMNIST. In CIFAR-10, our method shows the improvement over FedAVG by 12.54% in data deficient regime, and by 17.16% and 23.56% in more challenging scenarios of noisy label or Byzantine client cases, respectively.

Under review as a conference paper at ICLR 2023
EXPLOIT UNLABELED DATA ON THE SERVER! FEDERATED LEARNING VIA UNCERTAINTY-AWAR ENSEMBLE DISTILLATION AND SELF-SUPERVISION
Anonymous authors Paper under double-blind review
ABSTRACT
Federated Learning (FL) is distributed machine learning paradigm that involves the cooperation of multiple clients to train a server model. In practice, it is hard to assume that each client possesses large-scale data or many clients are always available to participate in FL for the same round, which may lead to data deficiency. This deficiency degrades the entire learning process. To resolve this challenge, we propose a Federated learning with entropy- weighted ensemble Distillation and Self-supervised learning (FedDS). FedDS reliably deals with situations where not only the amount of data per client but also the number of clients is scarce. This advantage is achieved by leveraging the prevalent unlabeled data in the server. We demonstrate the effectiveness of FedDS on classification tasks for CIFAR-10/100 and PathMNIST. In CIFAR-10, our method shows the improvement over FedAVG by 12.54% in data deficient regime, and by 17.16% and 23.56% in more challenging scenarios of noisy label or Byzantine client cases, respectively.
INTRODUCTION
Federated Learning (FL) is distributed machine learning paradigm that involves the cooperation of multiple clients to train server model (McMahan et al., 2017). In FL. a server model is trained as follows: 1) the server distributes the current server model clients, 2) each client inde- pendently trains each model downloaded from the server with their available local data and sends back the resul- tant model to the server, 3) the server updates the server model with the collected repeat the steps. By collecting updated model parameters at the server instead of raw client data, FL can mitigate the personal information leakage.
87.5
85.0
Test accuracy
89.5
12.54%
60.0
77.5
models, and 4)
75.0
FedAVG
72.5
FodDS
12500
10000
7500
5000
2500
Data amounts each client
In some FL. scenarios, such as developing a medical di- agnosis algorithm. is often the case where the number of clients (participating hospitals) and the size of labeled datasets in each client (the number of relevant patients and those labels at each hospital) are deficient. Such a lack of participating clients and labeled datasets the clients leads to performance degradation for the standard FL method, e.g., FedAVG (McMahan et al., 2017) (refer to Fig. 1). The deficiency may also destabilize the learning process which increases label noise sensitivity of FL meth- ods. Even in such scenarios, unlabeled data is abundant easy to collect in practice, which may help to mitigate the data deficiency and label noise vulnerability of FL algorithms.
Figure 1: method, FedAVG (MeMahan al., 2017). rapidly drops as the amount of available data at each client decreases. Our FedDS miti- gates the effect of data deficiency by exploit- ing unlabeled data the server. The accu- racy is measured the 50th communication round on CIFAR- 10 classification task. This result follows the same setting as the main experiment in Fig. 4 except the data amounts for each client.
The accuracy of the standard FL
In this paper, we propose a robust FL algorithm by utilizing additional unlabeled data on the server. The key idea of our method is to leverage unlabeled data to mitigate lack of data as well to reliably

---

Under review as a conference paper at ICLR 2023
aggregate the client models into the single server model by unsupervised ensemble knowledge distillation. We postulate that the aforementioned degradation of the accuracy mainly stems from unreliable clients participated in the server model update. To mitigate the influence of unreliable clients, we measure the entropy of each client's 's output to assess the uncertainty of each model, and use them to weigh for each client model. We found that this simple entropy measure is sufficiently well-calibrated to train the server model better. Thereby, we suppress unreliable clients' contribution when aggregating them. Furthermore, our proposed entropy-weighted ensemble distillation (EED) is performed jointly with another self-supervised learning (SSL) loss with unlabeled data on the server. With our setting, while additionally imposing a SSL loss is simple and demands negligible overheads, it is found to be crucial and have several benefits, such as faster convergence and reducing the influence of unreliable nodes.
Our major contributions We propose Federated learning with entropy-weighte ensemble Distillation and Self-supervised learning (FedDS), a method for reliably updating the server model by utilizing the server's unlabeled data in an unsupervised way. We demonstrate through experiments that our FedDS outperforms several strong baselines in the classification tasks for CIFAR-10/ and PathMNIST, especially in data deficient regime, and is also robust to various tough scenarios with unreliable clients.
2 RELATED WORK
Since the emergence of FL paradigm (McMahan et al., 2017), there have been many follow-up studies to tackle various challenges in FL: enhancing communication efficiency (Alistarh et al., 2017; Suresh et al., 2017; Bernstein et al., 2018; Tang et al., 2018; Wu et al., 2018; Hamer et al., 2020; Rothchild et al., 2020; Reisizadeh et al., 2020; Haddadpour et al., 2021; Qiao et al., 2021; Konečný et al., 2;16; Syyeon-Woo et al., 2022; Jeong et al., 2018), stabilizing convergence and solving various issues arising from heterogeneity in client's data (Li et al., 2020; Karimireddy et al., 2020; Acar et al., 2021; Reddi et al., 2021; Yuan & Ma, 2020) and in client's 's model structure (Diao et al., 2021b), protecting clients from privacy attacks (Ammad- -d------Dii et al., 2019; Gong et al., 2021), and effectively aggregating unreliable client models containing malicious clients (Chang et al., 2019). In particular, the use of extra data, in addition to local data available at each client, has been shown to be effective to deal with some of the aforementioned challenges in FL (Chang al., 2019; Lin et al., 2020; Li & Wang, 2019). In this section, we focus on briefly reviewing two main research streams utilizing additional data in FL, which are closely related to our work, i.e., utilizing unlabeled data on the server. For comprehensive review of FL, please refer to Kairouz et al. (2019)
Knowledge distillation in federated learning Knowledge distillation (KD) is used to transfer knowledge of teacher model to student model (Xie et al., 2020; Zhou et al., 2021; Radosavovic et al., 2018; Kim et al., 2021). The teacher model used to provide pseudo labels to either labeled or unlabeled data, and the student model learns to mimic the teacher's behavior by using these pseudo-labeled data as supervision.
To adapt KD for Fl., data be pseudo labeled should be presented the point of aggregation (the server); thereby, we can extract knowledge that will be transferred to the aggregated model (the server model). However, if the data be distilled is provided from clients to the server, FL loses the privacy and security property which is one of the big advantages of FL. To address this, there are studies exploiting the assumptions that transferring features and logits from client's data is allowed (He et al., 2020; Gong et al., 2021) or assumptions that additional datasets exist (Gong et al., 2021; Chang et al., 2019; Lin al., 2020; Li & Wang, 2019; Shi al., 2021). The latter assumes that an extra public dataset is shared across all clients. Then, KD is performed at the server by collecting the logit values for the public dataset from the clients, instead of model parameters. This approach can achieve higher communication efficiency and privacy protection. Our work relaxes the assumption by possessing extra unlabeled data only in the server, rather than sharing across all clients.
As similar setting to ours, Lin et al. (2020) proposes a FL method, called FedDF. In FedDF, the server obtains pseudo labels for the unlabeled data with the collected client models, and performs KD with those pseudo labels. They show that their approach is robust to non-i.i.d. data, achieving a higher accuracy than FedAVG (McMahan et al., 2017). We further take into account confidence of each client and core knowledge of the server by measuring uncertainty of client predictions and applying
2

---

Under review as a conference paper at ICLR 2023
self-supervised learning (SSL), respectively. Thereby, our proposed FedDS enhances both reliability and robustness of the learned model against the data deficiency and label noise vulnerability.
Semi-supervised federated learning Unlabeled data is much more prevalent than labeled ones. Semi-supervised learning approaches in FL (Shi et al., 2021; Jeong et al., 2021; Zhang et al., 2021; Diao et al., 2021a; Lin et al., 2021) have been studied to leverage these prevalent unlabeled data. All of these studies consider scenarios where each client has few labeled data but instead has a large amount of unlabeled data, and exploits the unlabeled data with unsupervised learning losses. Unlike these preceding studies, our work considers the scenario where unlabeled data is leveraged on the server side while clients have limited labeled data, which is more practical setring.
3 METHOD
3.1 PROBLEM SETTING
We assume of C' clients with client index e 21, C} and single server. Each client with index c has its own labeled dataset D. = {zt, i-) where (Iz, ys) X' x y for each zy is input data instance and ys is its corresponding label in the one-hot vector form. Client data distributions can be i..dd (each client has the same data distribution), or non-i.i.d. (each client has different data distribution with others). In our setting, the central server has an unlabeled dataset D, ( where H; E X for each i. The training process is iterative. The server model is learned by repeating the client update and the server update for each communication round E 21, T).
Client update At round t, each client e re- ceives the server model f from the previous round t- - and initializes its client model with it. Then, each client e trains its client model in supervised way with each client's own labeled dataset Dei e.g., with the cross-entropy loss each client e trains its model by minimizing the empirical loss ft = arg min, (f) for E. epochs, where Lo( == wD (1)) y)]. Then, all clients transfer their models {f)c the server.
Labeled dataset
Modal parameter communication
Server
Client
,>1)
Client
Unlabeled dataset
Client C
Problem setting. Each client has labeled dataset, whereas central server has unlabeled dataset. server model is learned through iterative communi- cations of model parameters between the clients and the server.
Figure 2:
Server update In the server update phase, at round 4, the server aggregates the client models {fae into a single server model f.. Distinctively, we aggregate the client models by leveraging unlabeled server dataset D,. Designing such model aggregation method is our goal of this work. After we establish the server model fú, the model is distributed to all the clients.
3.2 FEDERATED LEARNING VIA UNCERTAINTY-AWAN ENSEMBLE DISTILLATION AND SELF-SSUPERVSSON
We focus on the client model aggregation step in the server update phase. In the server update phase, the server model starts either from the previous model from the parameter average of client models. Then, our FedDS (A (Alg. 1) aggregates client models' knowledge via an ensemble distillation that aggregates client models while taking into account uncertainty of each client model per sample as prediction quality, called entropy-weighen weighted ensemble distillation (EED). We also propose to exploit unlabeled server dataset by supervuper learning (SSL), so that the server model can further leam generic knowledge about the task. Both our proposed methods exploit unlabeled dataset. This stabilizes and better poses the server model as better initial points clients every round. We simultaneously conducts EED and SSL with the unlabeled server data.
pseudo label lp for each unlabeled data is generated by normalizing the entropy-weighted sum of predictions of each client model as:
Entropy-weighted ensemble distillation (EED)
4 = E (e))ep (+
(x())xpp( (HH)pp((
3

---

Under review as a conference paper at ICLR 2023
Algorithm 1: FedDS on C' clients for T rounds. Experimentally, when N: N,, setting BAVG false gives higher test accuracy (refer Table 2 and Table 3). fo o g denotes the composition functions f and g. (Notations)
D.: an unlabeled datasct in the server. De: labeled dataset in the e-th client.
S Sh. ): the server model at round I, where Jay denotes feature extracter part of the server model. f.. classifier part of the server model, and fü, self-superrised learning (SSL) classifier of the server model, which is only used to calculate the SSL loss.
f2: the e-th client model at round t.
E,: the number of server training epochs for each round. q: weight hyperaarameter for the SSL loss.
B BAvG: a boolean hyperparameter that determines the server's 's initial model each round.
Input :D.. De for == 1,... Randomly initialize fi....fe. 12 for round t .....
for client c = 1... C' in parallel do fe ClientUpdate (f==² fay De)
end
Bava true then
f. + c
fc
/+ start from average of client models +/
c=1
else
fi + pt-1
/N start from previous server model x/
end
for epach L.... E. do
for each input data x E D. Ip +- CreatePssudoolabeefff Leg + CrossEntropy (S:. o Say (x))
fc,2)
LSSL GetSelfSupervisedL.oss/ fa, z2y,
Optimize f minmmize Lce 7L8st
/* using any optimizer x/
end end
end
return
where H(p) = = log P; for probability vector p denotes the entropy of p. & the tempera- ture hyperparameter, and ||-I|ı the -nom.. The entropy H (x)( measures the degree of uncer- tainty about the class prediction (x) made by client for input x round the larger the
Softmax value for tree class
Entopy scishting
Mas predicton weggting
entropy is, the larger the uncertainty is. Then,
the pseudo label is used compute the cross-
0.2
3.0
Weight
entropy loss LoB with the server model's pre-
diction (ft, (z)) as done in the standard knowledge distillation (Hinton et al., 2015).
Figure 3: In CIFAR- 10 classification task, the relation- ship between the assigned weight and the initial soft- max value the true class for each client is illustrated. The blue curve based our entropy- based weighting method, and the green curve based on max pre- diction weighting method using the maximum softmax value for comparison. The weight exp [-kIl (f)))) with k=5 for entropy weighting and max f((x) for max class prediction weighting are normalized so that the weights for all the clients for given unlabeled data sum to 1. The range of weight (0. 1] is quantized into 50 bins and the weight of each client for each data æ is mapped to the nearest bin. For each bin, the softmax values the true classes are averaged.
The entropy-based based weighting mitigates the in- fluence of unreliable clients by suppressing the uncertain predictions. Our preliminary exper- iment in Fig. shows that this entropy-bor -based weighting indeed tends to assign higher weights to more correct predictions, i.c., the predictions for which softmax values on the true classes are higher. In Fig. 3, we compared with another weighting method based maximum softmax values of clients's prediction, which does not exhibit such tendency.
4

---

Under review as a conference paper at ICLR 2023
Table 1: Comparison between FedDS and other baseline algorithms.
FedAVG
FedDF
FedD (ours)
FedAVG-S
FedDS (ours)
Ensemble distillation
Uncertainty awareness
Self-supervised loss
Self-supervised learning The EED is applied jointly with self-supervised learning (SSL) to further leverage the benefits of available unlabeled data. An arbitrary SSL method can be applied, which helps to learn generic feature representation (Chen & He, 2021; Chen et al., 2020; Gidaris et al., 2018). This may act as stabilizing the FL learning procedure aggregation, which motivates us deploy. For image data, we use the image rotation prediction (Gidaris et al., 2018) as the self-supervised pretext task, i.e., rotating imput images and predicting those rotated angles, which is known to be simple and effective without cumbersome tuning, We use it due to its simplicity. but SSL in our approach is not limited to that method. We append the SSL head fú, on top of the server's feature extractor part fay, i.e. (f% o f The SSL loss is applied to the output of the SSL head f.. Our final loss for aggregating client models into the server model is defined as LCE + yCssL in multi-task manner.
EXPERIMENT RESULTS
4.1 EXPERIMENT SETUP
Dataset and model Our FedDS is evaluated on CIFAR-10/100 and PathMNIST datasets with the ResNet-18 18 (He et al., 2016). CIFAR-I0 (resp. CIFAR-100) has 10 classes (resp. 100 classes) containing 6,000 images (resp. 600 images) per class. For each of CIFAR- 10/100 datasets, the training and test sets contain 50,000 and 10,000 images, respectively. PathMNIST (Yang al., 2021) is a medical image dataset consisting of 9 classes of Colon Pathology images. The training and test sets contain 89,996 and 7,180 images, respectively.
Dataset split In our experimental setting, we mainly assume 4 clients unless otherwise mentioned. The total client training dataset is sampled in such way that it has the same class distribution as the entire training dataset, and it is partitioned by the number of clients and allocated to each client. Each client's 's class distribution follows the Dirichlet distribution Dir(a) in Lin et al. (2020; 202 I); Yurochkin et al. (2019); Wang et al. (2020); Hsu et al. (2019). The hyperparameter a indicates the homogeneity of the distribution: the higher is, the closer i.i.d. client data setting. We use a = 100, 000 and a = 0.5 for the i.i.d. and non-i.i.d. client data settings, respectively. Some portion of training dataset not sampled for client training datasets is used in the server without labels to mimic an unlabeled dataset.
Baseline algorithms Our baseline algorithms are FedProx, FedDF (Lin et al., 2020), and variants of FedAVG and FedDS. Since FedAVG (McMahan al., 2017) and does not leverage unlabeled data, we devise a method by combining FedAVG and SSL (called FedAVG-S). After the server model is updated by model averaging in FedAVG, we apply the SSL loss (the rotation loss (Gidaris et al., 2018)) to further update the server model with the unlabeled dataset on the server. We confirm that this indeed provides a strong baseline the presence of unlabeled data on the server (refer to Sec. 4.2). In addition, to the effect of the SSL task in our FedDS, consider a variant of our FedDS (called FedD), where the SSL part is suppressed by setting the SSL loss weight y 0. For FedD and FedDS, set BAYG = false unless otherwise mentioned.
The main characteristics of the baselines and FedDS are summarized in Table 1. In summary, both FedDF and FedD are based ensemble distillation, where FedD incorporates entropy-based weights to take into account confidence of each client. On the other hand, FedAVG-S and FedDS both apply SSL, while FedDS takes the ensemble distillation jointly with SSL.
5

---

Under review as a conference paper at ICLR 2023
FedDS
FedD
FedDF
FedAVG-S
FedAVG
FedProx
CIFAR-I0 i.i.d.
CIFAR-10 non-i.i.d.
Test accuracy
Test accuracy
70
70
60
60
Communication round
Communication round
CIFAR-100 i.id.
CIFAR-100 non-i..d.
50
50
Test accuracy
Test accuracy
40
40
30
30
20
20
Communication round
Communication round
Figure 4: Test accuracy of FedDS and baseline algorithms according communication rounds CIFAR 10/ 100 classification tasks with i.i.d. and non-i.id. client data settings. Each client 2,500 labeled and the server 40,000 unlabeled data. Each experiment was repeated times by changing random seeds. Each line and shaded region represent the mean and standard deviation of the repeated experiments, respectively.
4.2 RESULTS WITH NORMAL DATA CONDITION
this subsection, we present the accuracy results for CIFAR-10/100 and PathMNIST classification tasks with i.i.d. and non-i.i.d. client data settings, called normal data condition. For CIFAR- 100100 tasks, Fig. 4 shows the test accuracy FedDS and baselines according to communication rounds. It shows that the proposed FedDS consistently outperforms all the basclines starting from the early communication rounds. The effectiveness of our algorithm according to the number of clients and the size of client data is analyzed based Tables 2 and Table 4 shows that our FedDS still outperforms the baselines even when unlabeled data on the server is relatively small. Finally, the effectiveness of our algorithm on motivating medical diagnosis applications is verified via the PathMNIST classification task in Table 5.
Table 2: Test accuracy of FedDS and basclines the 50th communication round CIFAR- classification
task for various numbers of clients with the same sized and i.i.d. client data setting. Each client has 1,000 labeled data and the server has 10,000 unlabeled data. For B BAVG = false, the performance gain due to EED is evaluated by the difference between the test accuracy of FedD and that of FedAVG, the additional gain due SSL. is calculated by the difference between the accuracy of FedDS and that of FedD. The performance difference of FedDS by taking client model average calculated by the difference between the accuracy FedDS with Bave = true and that with BAVe = false.
Total number of
FedD (ours)
FedDS (ours)
Clients
FedAVG
client data
BAVG false
BAYG false
BAVG true
2,000
50.42
60.26 (+9.84)
66.28 (+6.02)
62.01 (-4.27)
4,000
59.43
67.36 (+7.93)
72.02 (+4.66)
69.40 (-2.62)
8
8,000
66.59
72.78 (+6.19)
75.43 (+2.65)
75.09 (-0.34)
20
20,000
76.53
76.15 (-0.38)
78.11 1 (+1.96)
81.12 (+3.01)

---

Under review as a conference paper at ICLR 2023
Table 3: Test accuracy of FedDS and baselines at the 50th communication round CIFAR-10 classification task for fixed number of clients and various sizes of client datasets with ii.d. client data setting. The server has 10,000 unlabeled data. The clients are trained with the same number of gradient step updates. For BavG = false. the performance gain due to EED is evaluated by the difference between the test accuracy of FedD and that of FedAVG, and the additional gain due to SSL is calculated by the difference between the test accuracy of FedDS and that of FedD. The performance difference of FedDS by taking client model average calculated by the difference between the test accuracy FedDS with BAVG = Irue and that with BAv = false.
Total number of
FedD (ours)
FedDS (ours)
Clients
FedAVG
client data
BAVG
false
BAVG = false
BAVG true
4
2,000
50.63
57.43 (+6.80)
66.42 (+8.99)
61.38(-5.04)
4
4,000
59.43
67.36 (+7.93)
72.02 (+4.66)
69.40(-2.62)
4
10,000
74.46
76.82 (+2.36)
78.78 (+1.96)
78.32(-0.46)
Comparison of two ensemble-distillaiinn methods We confirm the effectiveness of our uncertainty aware ensemble-distilltion by comparing two ensembleddistilaatio methods, i.e., our FedD and FedDF (Lin al.. 2020). For all the settings in Fig. 4, our FedD outperforms FedDF. The main differences of our FedD from FedDF are twofold; measuring the confidence of each client's 's prediction based on the entropy of softmax values and choosing the initial point of the ensemble distillation as the previous server model. For the initial point of the ensemble distillation, our FedD with BAvG = false starts ensemble distillation directly from the server model at the previous round while FedDF starts ensemble distillation from the parameter average of transferred client models at each round. Our empirical study in Tables 2 and 3 shows that using the averaged client model as a new initialization at each round is detrimental when the total number of client data is small.
Effect of EED and SSL In Fig. 4, our FedD shows comparable or better performance to the other baselines for all the settings, indicating that the mntihhy-weighted ensemble distillation is as effective as applying self-supervision. i.e., FedAVG-S, or even better. From FedAVG-S results, it is interesting to see that upplying SSL is mostly effective, but its effect is diminishing in some cases. FedAVG-S achieves the highest accuracy among the baselines in the CIFAR- 10 classification task. However, it achieves similar accuracy with FedProx in the CIFAR- classification task. We conjecture that this is because the method of averaging the client model parameters in FedAVG results in unstable initialization for SSL when the total number of client data is small.
By jointly applying SSL with FedD, corresponding to our FedDS, we can see considerable perfor- mance improvements. In particular, our FedDS is effective for the CIFAR-100 classification task, where the data per each client is quite small, i.e., 25 images per class average.
Table 4: Test accuracy of FedDS and baselines at the 50th communication round on CIFAR-10 10 classification task with various sizes of unlabeled server dataset with i..dd client data setting. Each of four clients has 2.500 labeled data. The numbers of unlabeled server data for each algorithm are represented in the second row of the table.
FedAVG
FedDF
FedDS
No. unlabeled server data
0
40,000
10,000
20,000
30,000
40,000
Test accuracy
72.87
77.37
80.22
82.46
83.48
84.15
Experiments with various numbers of clients and sizes of client data To examine the effective- ness FedDS with various dataset-split setups, we evaluate the performance gain of our FedDS compared to the baselines for various numbers of clients (Table 2) and for various sizes of dataset in each client (Table 3). For our FedDS with BAYG = false, Table shows that both the gain due to EED (in blue color) and the additional gain from SSL (in red color) increase the number of clients decreases. For small number of clients, the influence of unreliable clients becomes higher. Thus, the EED which mitigates the influence of unreliable clients, and the SSL which provides the generic
7

---

Under review as a conference paper at ICLR 2023
Table 5: Test accuracy of FedDS and baseline algorithms measured at the 50th communication round MNth classification task with i..d. client data setting. Each elient has 2,500 labeled data and the server has 40,000 unlabeled data, Each experiment was repeated times by changing the random seed and the mean values are taken.
Client data distribution i..d. non-i.i.d
FedAVG
FedDF
FedAVG-S
FedDS (ours)
87.59
88.34
89.66
90.51 90.05
86.80
87.06
88.93
knowledge about the task become more effective. Similarly, Table shows that our FedDS with BAVG false is more effective the clients' data becomes more deficient. Particularly, we can see that SSL becomes the dominant factor of performance gain as the client's 's dataset gets smaller.
Tables and 3 also show the performance difference of FedDS by setting BavG true instead of BAVG = false (represented in teal colors). It shows that starting from the server model at the previous round (BAvG false) performs better when the total number of client data is relatively small. This can be caused by the fact that averaging client models with limited client data unstabilizes the initial point, because small fraction of outlier undesirable contamination in client parameters can lead to notable biases in resultant parameter aggregation.
Small unlabeled dataset at the server is still useful To see the effect of the size of the unlabeled server dataset on the performance gain of FedDS, FedDS was evaluated for the CIFAR-10 classifica- tion task by changing the size of server's unlabeled dataset. Table shows that FedDS is still useful when the number of unlabeled data at the server is comparable with that of total labeled data the clients. Although the test accuracy of FedDS decreases as the number of unlabeled data at the server decreases, FedDS using 10,000 unlabeled data still outperforms FedDF using 40,000 unlabeled data.
Results on medical data evaluate the performance of the proposed FedDS and other baselines for PathMNIST classification task with i.id and non-i.i.d. client data settings. Table 5 confirm the effectiveness of our method.
To exemplify a medical diagnosis scenario with client-data deficiency, we
4.3 RESULTS WITH UNRELIABLE CLIENT-SIDE DATA CONDITION
Table 6: Test accuracy FedDS and baseline algorithms measured at the 50th communication round on CIFAR-10 10 classification task under various scenarios with unreliable clients. Each of four clients has 2,500 labeled data and the server has 40,000 unlabeled data. Noise-1 (resp. Noise-20) represents noisy-label scenario where 1% (resp. 20%) of each client's labels is mislabeled. Byzantine represents Byzantine attack scenario with malicious client whose labels totally mislahcled. The performance degradation in i.i.d. setting and in non-i.i.d. setting is evaluated by the difference between the test accuracy under normal data condition (No noise) and that under the corresponding noise scenario. Each experiment was repeated times by changing the random seed and the mean values are taken.
Client data
FL without SSL
FL with SSL
Noise type
distribution
FedAVG
FedDF
FedD (ours)
FedAVG-S
FedDS (ours)
i..dd
71.53
76.90
79.64
80.24
84.07
No noise
non-i.id.
70.29
74.69
77.12
77.88
81.99
i.dd
65.54
-5,99)
72.04
-4.86)
75.05 ( -4.59)
75.24 ( -5.00)
81.14 -2,93)
Noise-1 1
non-i.i.d.
63.74
-6.55)
69.32
-5.37)
71.85 ( -5.27)
72.73 ( (5.15)
78.14
-3.85)
iid.
50.78 (-20.75)
55.31 (-21.59)
59.88 (-19.76)
59.06 (-21.18)
67.94 (-16.13)
Noise-20
non-i.idd
45.04 (-25.25)
47.95 (-26.74)
52.37 (-24.75)
55.11 (-22.77)
60.30 (-21.69)
iid.
(-14.79)
65.92 (-10.98)
75.27 ( -4.37)
66.12 (-14.12)
80.30 -3.77)
Byzantine
non-i.id.
56.32 (-13.97)
58.88 (-15.81)
70.20 ( -6.92)
64.16 (-13.72)
80.27 -1.72)
To see the robustness of incertainty-aware ensemble distillation and self-supervision, we consider two types of scenarios with unreliable clients. In particular, we consider two noisy-label scenarios with 1%
8

---

Under review as a conference paper at ICLR 2023
and 20% label noises at each client (denoted as Noise-1 and Noise-20, respectively), and a Byzantine attack scenario with a malicious client whose labels are totally mislabeled (denoted as Byzantine). We present the test accuracy of FedDS and baselines for Noise-1, Noise-20, and Byzantine scenarios in Table 6. Let us focus on the i.i.d. client data setting as the tendency for the non-i.i.d. case is almost the same. While our FedD's performance degradation for Noise-1 1 and Noise-20 scenarios is slightly smaller than FedDF, it is remarkably smaller for the Byzantine case. This shows that the server model effectively rejects the Byzantine client's prediction using our entropy-basee weighting (refer to Fig. 3). Our FedDS shows the most robust performance, i.e. performance degradations for Noise- 1, Noise-20, and Byzantine senarios are the much smaller than other baselines.
CONCLUSION
In this work, we propose FedDS, which exploits the unlabeled data on the server side by entropy- weighted ensemble distillation (EED) and self-supervised learning (SSL). By using both techniques jointly, we demonstrate that FedDS outperforms the competing algorithms when the number of clients small and data in the clients is deficient. In particular, when clients are unreliable, both techniques effectively suppress unreliable clients and reduce accuracy degradation. It is more effective as the number of participating clients are smaller, because, in such regimes, even introducing a single unreliable client would have much higher impact than other cases. Also, SSL effectively imposes the common knowledge for generic tasks to the server model. Thus, it stabilizes the overall learning round and helps the server model to overcome the deficiency of client data, especially when mislabeled samples are prevalent across clients. Our experiments show that EED and SSL lead to their own improvement and jointly using them still preserves notable improvement, which may imply that they play different roles that are both necessary for FL.
Limitations
The main limitation is that FedDS requires additional server resources including computation and additional unlabeled data. FedDS would be more fruitful on the situation when the central server has large unlabeled dataset and sufficient computation power.
Further directions To apply FedDS to the situation when there is little unlabeled data on the server, some synthetic data would be useful. Lin al. (2020) shows that the ensemble distillation using synthetic data from GAN gives decent pseudo label.
FedDS would also be applicable the clients with heterogeneous architectures. For each round, one can update clients regardless of heterogeneous architectures by distilling the server model's knowledge using the client's 's data, instead of directly substituting the server model's parameter each client model. FedDS would be applicable to tasks other than computer vision, like natural language. One can choose suitable SSL task, like next word prediction, and use it as a SSL loss for FedDS.
Reproducibility Our algorithms are described in detail at Sec. 3.2 and Sec. A.1.1. Implementation details needed to reproduce our results are described in See, 4.1 including hyperparameters and data distribution. Our source code will be publicized upon acceptance.
REFERENCES
Durmus Alp Emre Acar, Yue Zhao, Ramon Matas, Matthew Mattina. Paul Whatmough. and Venkatesh Saligrama. Federated learning based on dynamic regularization. In International Conference on Learning Representations (ICLR), 2021.
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna:
A next-generation hyperparameter optimization framework. In Proceedings of the 25rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2019. 14, 17, 18
Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. Qsgd: Communication- efficient sgd via gradient quantization and encoding. Advances in Neural Information Processing Systems (NeurIPS), 2017.
Muhammad Ammad-Ud-
Elena Ivannikova, Suleiman A Khan, Were Oyomno, Qiang Fu, Kuan Eeik Tan, and Adrian Flanagan. Federated collaborative filtering for personalized recommendation system. arXiv preprint arXiv:1901 190109988, 2019. 2

---

Under review as a conference paper at ICLR 2023
Shai Ben-David John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory learning from different domains. Machine learning, 79(1):151- 2010. 18
Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Animashree Anandkumar. signSGD: Compressed optimisation for non-convex problems. In International Conference Machine Learning (ICML), 2018. 2
Hongyan Chang, Virat Shejwalkar, Reza Shokri, and Amir Houmansadr. Cronus: Robust and heteroge- neous collaborative learning with black-box knowledge transfer. arXiv preprint arXiv: 12. 1279, 2019. 2
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML), 2020. 5
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 5
Patryk Chrabaszez, llya Loshchilov, and Frank Hutter. A downsampled variant of imagenet as an alternative to the cifar datasets. arXiv preprint arXiv:1707.08819, 2017. 16
Enmao Diao, Jie Ding, and Vahid Tarokh. Semifl: Communication efficient semi-supervised federated learning with unlabeled clients. arXiv preprint arXiv:2106.01432: 2021a. 3
Enmao Diao, Jie Ding, and Vahid Tarokh. HeteroFL: Computation and communication efficient feder- ated learning for heterogeneous clients. In International Conference on Learning Representations (ICLR), 2021b. 2
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. International Conference on Learning Representations (ICLR), 2018. 5, 13, 14, 17
Xuan Gong, Abhishek Sharma, Srikrishna Karanam, Ziyan Wu, Terrence Chen, David Doermann, and Arun Innanje. Ensemble attention distillation for sewacy-preserving federated learning. In IEEE International Conference on Computer Vision (ICCV), 2021.
Farzin Haddadpour, Mohammad Mahdi Kamani, Aryan Mokhtari, and Mehrdad Mahdavi. Federated learning with compression: Unified analysis and sharp guarantees. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2021. 2
Jenny Hamer, Mehryar Mohri, and Ananda Theertha Suresh. Fedboost: A communication-efficiennn algorithm for federated learning. In International Conference on Machine Learning (ICML), 2020.
Chaoyang He, Murali Annavaram, and Salman Avestimehr. Group knowledge transfer: Federated learning of large cnns at the edge. Advances Neural Information Processing Systems (NeurIPS), 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2016. 5, 19
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, al. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.025: 2015.
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non- con-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06333 2019.
Nam Hyeon- Hyeon-Woo, Moon Ye-Bin and Tae- fae-Hyun Oh. Fedpara: Low-rank hadamard product for communication-effiee federated learning. International Conference on Learning Representations (ICLR), 2022.
10

---

Under review as a conference paper at ICLR 2023
Eunjeong Jeong. Seungeun Oh, Hyesung Kim, Jihong Park, Mehdi Bennis, and Seong- Kim. Communication effiiceent on-device machine learning: Federated distillation and augmentation under non-iid -id private data. arXiv preprint arXiv:
2018. 2
Wonyong Jeong, Jaehong Yoon, Eunho Yang, and Sung Ju Hwang. Federated semi-supervised learning with inter-client consistency & disjoint learning. In International Conference Learning Representations (ICIRR,, 2021.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Keith Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. arXiv preprint arXiv: (9tp:::912209799 2019. 2
Sai Praneeth Karimireddy, Saryen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning (ICML), 2020. 2
Kyungyul Kim, Byeongmoon Ji, Doyoung Yoon, and Sangheum Hwang. Self-knowledge distillation with progressive refinement of targets. IEEE International Conference on Computer Vision (ICCV), 2021.
Jakub Konečný, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:/610.05492, 2016. 2
Alex Krizhevsky. Learning Multiple Layers of Features from Tiny Images. 2009. 19
Alex Krizhevsky, Σlya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances Neural Information Processing Systems (NeurIPS), 2012.
15, 16, 19
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. 15, 19
Daliang Li and Junpu Wang. Fedmd: Heterogenous federated learning via model distillation. In Advances in Neural Information Processing Systems Workshops (NeurIPSW), 2019. 2
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks.
Dhillon, D. Papail- iopoulos, and V. Sze (eds.), Proceedings of Machine Learning and Systems, volume 2, pp. 429 450, 2020. URL https: httttp:://procceedingg www.eeeeeiiinnmm.ss..m....ee www.sysssrrr...oeeooooo..eoooeoe orggpppper//22///// /0t///////////////// 38af8613 34dd50 www.ab/bdddffffeeoooeeeeeeeeeeee 1et00011fff3330 wwwffaee30dddddeeeee http://4422eaaaae..ppp..
Haowen Lin, Jian Lou, Li Xiong, and Cyrus Shahabi. Semifed: Semi-supervised federated learning with consistency and pseudo-labeling. arXiv preprint arXiv:2108.09// 2021. 3, 5
Tao Lin, Lingjing Kong, Sebastian U. Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020. 2, 5, 7, 9, 17,
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Arcas. Communication-efficin learning of deep networks from decentralized data. International Conference Artificial Intelligence and Statistics (AISTATS), 2017. 1, 1, 5, 13
Zhefeng Qiao, Xianghao Yu, Jun Zhang, and Khaled Letaief. Communication-effiiiee federated learning with dual-side low-rank compression. arXiv preprint arXiv:2104.1244 124111 2021. 2
Ilija Radosavovic, Piotr Dollár, Ross Girshick, Georgia Gkioxari, and Kaiming He. Data distilla- tion: Towards suprrvsee learning. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2018. 2
Sashank J. Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and Hugh Brendan McMahan. Adaptive federated optimization. In International Conference Learning Representations (ICLR), 2021. 2

---

Under review as a conference paper at ICLR 2023
Amirhossein Reisizadeh, Aryan Mokhtari. Hamed Hassani, Ali Jadbabaie, and Ramtin Pedarsani. Fedpaq: A communication-effiiii federated learning method with periodic averaging and quan- tization. In International Conference Artificial Intelligence and Statistics (AISTATS), 2020.
Daniel Rothchild, Ashwinee Panda, Enayat Ullah, Nikita Ivkin, Ion Stoica, Vladimir Braverman, Joseph Gonzalez, and Raman Arora. Fetchsgd: Communication-efiiiii federated learning with sketching. In International Conference on Machine Learning (ICML), 2020. 2
Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithms. Cambridge university press, 2014.
Haizhou Shi, Youcai Zhang, Zijin Shen, Siliang Tang, Yaqian Li, Yandong Guo, and Yueting Zhuang. Towards communication--ffiiieee and privacy-preserving federated representation learning. arXiv preprint arXiv:2109. /4611, 1, 2021. 2,3 3
Ananda Theertha Suresh, X Yu Felix, Sanjiv Kumar, and H Brendan McMahan. Distributed mean estimation with limited communication. In International Conference on Machine Learning (ICML), 2017.
Hanlin Tang, Shaoduo Gan, Ce Zhang, Tong Zhang, and Ji Liu. Communication compression for decentralized training. S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Cesa-Biande Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems (NeurIPS), 2018.
Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Fed- crated learning with matched averaging. In International Conference on Learning Representations
(ICLR), 2020.
Jiaxiang Wu, Weidong Huang, Junzhou Huang, and Tong Zhang. Error compensated quantized sgd and its applications to large-scale distributed optimization. In International Conference on Machine Learning (ICML), 2018. 2
Qizhe Xie, Minh- Thang Luong, Eduard Hovy, and Quoc V. Le. Self-training with noisy student improves imagenet classification. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni. Medmnist v2: A large-scale lightweight benchmark for 2d and 3d biomedical image classification. arXiv preprini arXiv:2 10. 14795, 2021. 5, 13, 19
Honglin Yuan and Tengyu Ma. Federated accelerated stochastic gradient descent. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, and Yasaman Khazaeni. Bayesian nomparametric federated learning of neural networks. In International Conference on Machine Learning (ICML), 2019. 5
Zhengming Zhang, Yaoqing Yang. Zhewei Yao, Yujun Yan, Joseph E Gonzalez, Kannan Ramchan- dran, and Michael W Mahoney. Improving semi-supervised federated learning by reducing the gradient diversity of models. In 2021 IEEE International Conference on Big Data (Big Data), 2021. 3
Helong Zhou, Liangchen Song. Jiajie Chen, Ye Zhou, Guoli Junsong Yuan, and Qian Zhang. Rethinking soft labels for knowledge distillation: A bias-variang tradeoff perspective. In Interna- tional Conference on Learning Representations (ICLR), 2021.
12

---

Under review as a conference paper at ICLR 2023
A
APPENDIX
The appendix consists of a detailed description of the proposed FedDS algorithm (Sec. A.1.1), additional experimental results (Sec. A.2), and dataset and model information (Sec. A.4).
A.1 ALGORITHMS AND IMPLEMENTATION DETAILS
A.1.1
ALGORITHM DESCRIPTION
@ Client Update
Server Update
91
Ent.-weighted
Self-Super. Learning
Lce
LSsL
Client Model Transfer
SSL classifier
4
Server Model Distribution
param zhare
.Yc)
Clients dataset
Client model
True label
4
Pseudo label
Server dataset
Server madel
SSL classifier
9
Prediction
Figure Illustration of the proposed FedDS algorithm. In each round, the model is updated by simultancously conducting entropy- entropy-weighted ensemble distillation (EED) and self-supervised learning (SSL) with the unlabeled server data.
Fig. 5 illustrates the operation of the proposed FedDS algorithm. Alg. 2, Alg. 3, and Alg. 4 correspond to ClientUpdate, CreatPseudoLabel. GetSelfSSperviieelL.ss algorithms used in the FedDS algorithm (Alg. I), respectively.
Alg. updates the client model as in FedAVG (McMahan al., 2017). It trains the distributed server model using the client's own data and returns the trained model.
Alg. creates the pseudo label for wetropy-weighted ensemble distillation (EED). gets the client models from Alg. 2 and creates a pseudo label for each unlabeled data of the server as the entropy- weighted average of the client models' predictions.
Alg. 4 illustrates an example of self-supervised learning (SSL) algorithm, which we used in the experiments. In general, GerSelfSupervisedlloss algorithm returns SSL loss Lsst. for each unlabeled data x of the server from feature extractor fe, and SSL classifier fz, of the server model. In this example, we assume that the input data is an image, and use the predicting image rotation task (Gidaris et al., 2018) as the self-supervised pretext task, i.e. rotating the input image and predicting its rotated angle.
A.1.2 IMPLEMENTATION DETAILS AND TRAINING
For each round, each client model is trained for 30 epochs using its own dataset unless otherwise men- tioned. We use the Adam optimizer with the client settings of learning rate 0.001, B (0.9, 0.999), weight decay 0, and batch size 64 for CIFAR- 10/100 and 128 for PathMNIST In the server update phase in each round, the server model is trained for 15 epochs for CIFAR- 10/1-000/100 and one epoch for PathMNIST using its own unlabeled dataset. The unlabeled dataset is pseudo-labeled by the gathered client models in the proposed entropy-weighted ensemble way. We set the entropy weight temperature & = 5 obtained from our empirical tests. We also use the Adam optimizer with the server settings of learning rate 0.00038 for CIFAR-10/100 and 0.001 for PathMNIST, 3 (0.9, 0.999), weight decay 0, and batch size 64 for CIFAR-10/10 10/100 and 128 for PathMNIST. The learning rate for
'The hyperparameters for PathMNIST in experiments chosen following those of Yang al. (2021).
13

---

Under review as a conference paper at ICLR 2023
Algorithm 2: ClientUpdate (same as the FedAVG).
DD is client e's labeled dataset. J',, and fax are the feature extractor and the classifier parts of the server model at round I, respectively. fe is the client c's model at round L. Ee is the number of client training epochs for each round.
Input: :uput:fa. fx 1 pt-1 De
fc fx. {(
for E L.... E. do
for inpur data and its label (æ, y) E De do LCE + CrossEntropy(jt (z)
Optimize fa to minimize Lce
/+ using any optimizer */
end end
return fc
Algorithm 3: CreatePscudoLabel.
ft is the client c's model at round f. x is an unlabeled server data. -pp)=-> Ps log for a
probability vector denotes the entropy of p, is the temperature hyperparameter for entropy weight, and || 112 denotes the Ln-norm.
Input: f ........ Sc, x
return Ee f((()e -he (J((o)
Dg f5(x)e ((zz)--bH(A( -H(((())
Algorithm 4: GerSelfSupervisedl.oom Loss (rotation loss (Gidaris et al., 2018))
An example of SSI. algorithm, which is used for FedDS in all the experiments in this paper. f", and fr, are the feature extractor and the SSL classifier parts of the server model at round t, respectively. In this example, the output of fá, is the prediction of rotation angle. For input image rota(z) rotates r by d degrees.
Input: f o fay:
LSSL+ 1 0
for = 0, do
LSSI. <- LSSL+ CrossEntropy(i, f2, Sú, (roteo: (z)))
end
return LSSL.
(IFAR-10/100 is found by a thyerpaaraeete search tool, Optuna (Akiba et al., 2019). The balance parameter for the SSL. loss is y 300. The underlying reason why such large 7 300 was applied is because the classification loss is implemented instead of averaging. If we take into account such an accumulation effect, the equivalent balance parameter was in fact 4.68 = 300/64 (64 is the server's batch size). For the chosen 7, the proportion of the rotation loss is about twice that of the classification loss the 50th round for the i.i.d. setting in Fig. 4
way that all the batch losses add up,
A.2 ADDITIONAL EXPERIMENTAL RESULTS
A.2.1 SELF-SUPERVISION MORE EFFECTIVE THAN FEW LABELED DATA
To see the effect a few labeled data on the server, we change 10% of server's 's dataset to be labeled data. The server model performs ensemble distillation with this labeled data by adding the supervised cross-entropp loss. For each batch pseudo-labeled data, labeled data is appended into the batch and is trained together.
14

---

Under review as a conference paper at ICLR 2023
Table 7: Test accuracy of FedDS and baseline algorithms measured at the 50th communication round CIFAR-10 classification task when the server data is unlabeled and when 10% of server data labeled and the remaining server data unlabeled.
Server labeled data
Client data distribution
FedDF
FedD (ours)
FedDS (ours)
i.d.
76.90
79.64
84.07
No labeled
non-i.id.
74.69
77.12
81.99
i.... non-ii.
79.41
81.31
84.80
10% labeled
77.36
80.36
83.62
Heterogeneous data setting
Client: CIFAR- 100, Server: TinylmageNet
Client: CIFAR-100, Server: ImageNet
05.0
65.1
07.5
07.0
02.5
02.3
PojAVG
65.0
45.0
10
20
10
20
Communication round
Communication round
Figure 6: Test accuracy of FedDS, FedDF and FedAVG for CIFAR-100 classification task with 4 clients i.id. client data settings. Each client has 12,500 labeled CIFAR- 1-00 data and the server has 100,000 Tiny Net unlabeled data (left) and 130,000 ImageNet (100 classes) unlabeled data (right).
Table 7 shows the effect of partial labels on the server. FedDS without labeled data outperforms both FedD and FedDF with labeled data. This shows that the self- self-supervision on large unlabeled data could give much valuable information than relatively little labeled data on our setting, More interestingly, the performance gain by using labels in FedDS is marginal. This may provide hint at possibility that the type of information from self-supervision is similar to and already covers the type of information the model can extract from labeled data.
A.2.2 HETEROGENBOUS DATA SETTING
To see the effectiveness of our method under heterogeneous data settings, FedDS was evaluated with other baselines Fig. by using CIFAR- 100 dataset for clients' labeled dataset and TinyImageNet (Le & Yang) and ImageNet(Krizhewe) experiment, whole CIFAR-100 training dataset and the whole TinylmageNe///maeeNee training datasets were used for client training and for server training, respectively, and the test accuracy was evaluated for CIFAR- 100 test dataset. Fig. 6 shows that FedDS outperforms both FedDF and FedAVG in such heterogencous data settings.
et al., 2012) datasets for server's unlabeled dataset. In this
A.2.3 EXPERIMENT WITH A LARGE NUMBER OF CLIENTS
While our FedDS mainly targets FL. scenarios where the number of clients or the labeled data of clients is insufficient, let us investigate the effectiveness of our method in a normal FL situation with a large number clients. For CIFAR- classification task with 100 clients, Fig. 7 shows that FedDS and FedDF using ensemble distillation consistently outperform FedAVG from early communication rounds. The difference between FedDF and FedAVG shows that situations where there are many clients, the effect of distillation decreases the communication round increases. In contrast, the consistent gap between FedDS and FedDF shows that entropy- based weighting and SSL are effective when there are many clients.
15

---

Under review as a conference paper at ICLR 2023
CIFAR-10 i..dd for 100 clients
Test accuracy
FedDS
65
FedDF
FedAVG
10
20
30
40
50
Communication round
Figure 7: Test accuracy of FedDS, FedDF and FedAVG for CIFAR-I0 classification task with 100 clients L.d. client data settings. Each client 400 labeled and server 10,000 unlabeled data.
ImageNet i.i.d.
40.0
35.0
Test accuracy
32.5
25.0
FedUS
22.5
Fed13
10
50
Communication round
Figure 8: Test accuracy of FedDS, FedDF and FedAVG for ImageNet classification task with 4 clients on i.i.d. client data settings. Each client has 6,500 labeled data and the server has 104,000 unlabeled data.
A.2.4 EXPERIMENT WITH A LARGER DATASET
Fig. shows that our FedDS outperforms FedDF and FedAVG for ImageNet classification task (Krizhevsky et al., 2012), which has 100 classes, 130,000 images for training and 5,000 images for test. For computational feasibility, we down-sampled to image resolution 32 (Chrabaszez et al., 2017).
A.2.5 EXPERIMENT WITH DIFFERENT CLASS DISTRIBUTIONS THE CLIENTS AND AT THE SERVER
our main experiments in Sec, 4, we assumed that the class distribution of the whole client dataset and that of the server's unlabeled dataset are the same. To investigate the effectiveness of our method when those class distributions are different, we evaluated FedDS with other baselines in Fig. 9 for CIFAR-10 classification task with i..dd. class distribution for clients and Dir(0.5) distribution for the server. We can see that our FedDS outperforms the baselines in such scenarios with different class distributions the clients and at the server.
16

---

Under review as a conference paper at ICLR 2023
Nonidentical class distribution
80
Test accuracy
20
FedDS
FedDF
FedAVG
10
20
30
40
50
Communication round
Figure 9: Test accuracy of FedDS, FedDF and FedAVG for CIFAR-I0 classification task with 4 clients i.i.d. client data settings. Each client has 2,500 labeled data the server 20,000 unlabeled data. Server data follows Dirichlet(a) distribution for a 0.5.
Table 8: Test accuracy of FedDS at the 50th communication round on CIFAR-10 classification task with different methods of combining BED and SSL. In addition to the proposed method of simultaneously conducting EED and SSL (2nd row), sequential approach is considered where SSL starts after EED (3rd row). In this experiment, the setting Sec. 4.1 is assumed and the predicting image rotation task (Gidaris al., 2018) used the SSL task.
SSL timing
FedDS
Simultaneous
84.07
Sequential
83,88
A.2.6 ABLATION STUDIES
Effect of combining methods for EED and SSL In the proposed FedDS algorithm (Alg. 1), the server model is updated by applying the entropy-weighhte ensemble distillation (EED) and the self-supervised learning (SSL) simultaneously. Another possible approach is to start SSL training after the end of EED training. Table shows that the performance gap between the two methods of combining EED and SSL is marginal, indicating that the timing SSL is not an important factor.
Effect of temperature Fig. 10 shows the optimized temperature values & for FedDS from the hyperparameter search tool Optuna (Akiba et al., 2019). For the i.i.d. and the non-i.i.d data settings, k around 0.1 and k around 8 achieve near-optimal performances, respectively, while the performance remains almost the same over a relatively wide range of k. In the non-i..d data setting, client's data deficiency is much severe than the i.i.d. case for certain classes, which necessitates stronger effect of weighti- that filters out immature predictions.
A.3 THEORETICAL RESULT
We extend the generalization error bound presented in FedDF (Lin et al., 2020) from uniform averaging logits to weighted-ensemble version corresponding to our FedD on Theorem 1. Our new analysis may hint the following important remarks: 1) the bound of FedDF (uniform averaging logits) special case of our bound, and 2) our entropy-weighted method (FedD) has the same upper bound as that of FedDF; thus, there is no reason that FedDF performs better than our FedD. The latter also supported by empirical experiments showing that our FedD outperforms FedDF, e.g., in Fig. and Table 6.
17

---

Under review as a conference paper at ICLR 2023
i.idd.
non-i.i.d
87
Test accuracy
Test accuracy
83
86
85
81
84
10
10"1
10"
10
10F
10 10
10"
10
10F
k
k
Figure 10: Test accuracy of FedDS on CIFAR- 10 classification task for near-optimal values of entropy- temperature l scarched from the hyperparameter search tool Optuna (Akiba et al., 2019).
Theorem (informal) We denote the global data distribution D, the e-th local distribution De- its empirical distribution De with size n, and the global empirical distribution D U.D The hypothesis h & H. learned on De is denoted by h.b.- Given the same assumptions with Lin et al. (2020), for every h E H and any 8 E (0, 1). with the probability - 5, we have
ENND C Σ wolo) (asc))) (o) (
LE ENDD [ (hb(2))) + Em m) + dduan(Des (D:, DD + d
(1)
where we(+) denotes entropy weight for each clients with E. we(+) and we()) O, : measures the distribution discrepancy between two distributions, m(6, n)
2
4+ (tglo((r((22)) Xe = inf (x) (h(x))] + EEEDD [wo()) and TH is the growth h
6v'2n
function bounded by polynomial of the VCdim of
Proof sketch. Given our entropy-weighted ensemble model E. wc(x) hDg (z)), we define its generalization error as ERDD c EE [_we(x) wc(x) 1b (x)) Then, we have
EgED [e (E.we(t)) " (. (1)) S EOND
(c(x)) (ND_ (x))] (By Jensen's inequality)
EAEDD
C.. (x))
(C. = wxxx)
E. FRNDD
[c (1Ds (2)) + 2 dHaH D) do
(By the domain gap bound (Ben-David al., 2010) with prob.)
E.ED [c' (1D. (x)) + m(6, n) +
2 (H D) + d
(By the uniform convergence (Shalev-Shwarz, & Ben-David, 2014) with prob.)
16 ]
E. [=)] E BaND. [c (n... (x))
+.
(by Hölder's inequality and H. m(6, n) dHaH (Da, D) + Ad)
2
E.E C (n... (x)) E.
(by (x) E [0, 1] and setting p = and 1)
E.E [c (n)()))) + E.He
(by the definition of ERM, L(h,, (x)) < L(hb(a)))
E BnD C (hb[z))] + E.H. (by D = U.D.)
=
18

---

Under review as a conference paper at ICLR 2023
Thus, have
E [c (E.u((t) nD, (z)) - <Bnb [c ((.())))]+mm(00..) (, D) (2)
with probability at least which concludes the proof.
A.4 DATASET AND MODEL
For classification tasks, dataset (Krizhevsky et al., 2012) (100 classes), TinylmageNet (Le & Yang) dataset are used CIFAR-
datasets (Krizhevsky, 2009)(MIT licence), ImageNet
10 (resp. CIFAR-100) has 10 classes (resp. 100 classes) containing 6000 images (resp. 600 images) per class. ImageNet (100 classes) has 100 classes containing 1,300 (resp. 50) images per class for training (resp. test). TinylmageNet has 200 classes containing 500, 50, 50 images per class for training, validation, test, respectively. We used only validation data for evaluate the test accuracy. PathMNIST (Yang et al., 2021) medical image dataset consisting of classes of Colon Pathology images. The training and test sets contain 89, 99,996 and 7,180 images, respectively. The proposed FedDS and baseline algorithms are implemented on ResNet- 18 (He et al., 2016) by changing the last layer to adjust to the number of classes, using the open source https: / / g thub. com/ com/kuang kuangl iu/ httppyyttrrhheee far. .it.....
19


Reviews:
[{"review_id": "T5sEIdtXS4", "reviewer": "Reviewer_CynN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes FedDS to improve the performance of the global model with distillation-based model aggregation. The key idea is to suppress the contributions of unreliable clients when performing multi-teacher ensemble distillation. Compared to FedDF, the authors added two additional components, the entropy-weighted pseudo label and the self-supervised learning (SSL) loss. The former calculates the uncertainty of each teacher's prediction score for weighting a teacher’s output. The latter can be an arbitrary SSL method. The experimental results verify the effectiveness of the FedDS to some extent.", "summary_of_the_review": "The paper has a good motivation. However, the novelty (FedDF + entropy-weighted + SSL loss) is limited. The experiments are unconvincing that in most experiments there are only four clients and the dataset is small.", "strength_and_weaknesses": "Strengths:\n1. The motivation is good. It is intuitive to suppress the contribution of unreliable clients.\n2. The overall paper is well organized and easy to follow.\n\nWeaknesses:\n1. The novelty of the proposed method is limited. The overall method is FedDF + entropy-weighted + SSL loss. For ‘entropy-weighted’, the authors should explain more why entropy can be used to measure the uncertainty of classification. For ‘SSL loss’, in learning with unlabeled data, SSL loss is commonly used and can be added in any distillation-based model aggregation method. These two components added to FedDF are incremental.\n2. The experiments are unconvincing. In most experimental settings, there are only four clients. Unless in the cross-silo setting, there should be at least 50 clients in the cross-device setting. \n3. The proposed method needs to be tested on a larger dataset, the conclusion on CIFAR is not convincing. \n4. The results shown in Figure 3 seems meaningless. The relationship between ‘softmax value for true class’ and ‘entropy-weight’ is inherent (because the former is calculated using the latter). How does this result relate to the motivation or performance of FedDS? \n5. It is mentioned in 4.1 EXPERIMENT SETUP that ‘the total client training dataset is sampled in such a way that it has the same class distribution as the entire training dataset’, which means that the server dataset has the same distribution as those of all the clients. Under such a setting, it is not surprising that the scheme based on weighted distillation is effective, because the difference between the dataset used in distillation and the client datasets is too small.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NckmeVdKBhI", "reviewer": "Reviewer_qQ7S", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, authors explore how to utilize unlabeled data on server to improve upon current Federated Learning methods. Authors propose (1) using entropy-weighted ensemble of client models to create pseudo labels for distillation training of global model (2) directly using SSL method with unlabeled data to help to train global model. Authors show compared with previous baselines such as FedAVG, FedDF and FedAVG-S, their proposed method shows significant increase in final accuracy. Also, authors show that their method demonstrates more robust performance with unreliable client-side data conditions. Finally, authors show small unlabeled datasets on server is still useful.  ", "summary_of_the_review": "I recommend weak rejection mainly due to lack of overall novelty and some questions on the experiment settings. ", "strength_and_weaknesses": "Strength:\n\n1: The paper is well motivated. It is true that in realistic FL tasks, we might be able to collect a lot of unlabeled data on central server. Thus, it is a very important question on how we could utilize such unlabeled data.\n\n2: The proposed method is simple but seems to be effective in the presented experiments. It is interesting that it is shown that small unlabeled dataset will still be useful. \n\nWeakness:\n\n1: The proposed method might not be novel enough. As stated by authors in related work section, both distillation training and SSL method have already been applied to federated learning. Authors only propose additional entropy weighted ensembles for distillation, which also might not be new in ensemble learning. \n\n2: In experiment sections, if I understand correctly, authors only use 4 clients in general (and upto 20 clients in some experiments). This might not be enough number of clients to be a meaningful federated learning experiment (which should have at least 100 clients). \n\n3: Authors should also include many other FL baselines: (1) authors should include algorithms that focus on solving non-iid FL, such as FedProx and Scaffold, in order to show the effectiveness of using additional unlabeled data (2) author should also compare with many other FL+SSL algorithms (some of which are already mentioned by authors in related work section).\n\n4: Lack of theoratical analysis of the proposed method. But I understand this is minor point for the emperical paper.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0MSTRdT5Ca6", "reviewer": "Reviewer_fQQY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a knowledge distillation based federated learning method by utilizing unlabeled data on the server. A global model, including a feature extractor and classifier, is trained to approximate entropy-weighted pseudo-label on the server. Further, a self-supervised learning framework is applied on top to learn generic feature representation better. \nThe method is suitable when the amount of data per client or the number of clients is scarce. Empirically, it beats strong baselines such as FedAVG and FedDF. ", "summary_of_the_review": "This paper proposed FedDS, which is a continuation and extension of FedDF. The empirical results are well presented. More comparisons to related SOTA results would be better though. Some theoretical support for the convergence of the proposed method is lacking. ", "strength_and_weaknesses": "Pros:\n1. The usage of entropy to denote prediction uncertainty of each client\n2. Ablation comparison of different components of the framework (EED and SSL) is well done\n\nCons:\n1. The entropy-induced uncertainty is at the client level, but could it be that a local model performs well on part of the unlabelled data only?  Can this pseudo label be calculated on a datapoint level?\n2. The choice of $\\gamma$ (balance parameter for SSL loss) is not well justified\n3. The claim when $N_c \\leq N_s$, settig $B_{\\text{AVG}}$ = false would give higher test accuracy is not well justified: when $N_c =N_s$, how could $B_{\\text{AVG}}$ affect test accuracy? Would be nice to show it empirically.\n4. How the train-test split is done on each client? Do they have the same distribution? This needs to be detailed about.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject