Towards scalable and non-IID robust Hierarchical Federated Learning via Label-driven Knowledge Aggregator
Authors: Duong Minh Nguyen
Viet Quoc Pham
Hoang Thai Dinh
Diep Nguyen
Long Tran-Thanh
Won-Joo Hwang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Federated Learning
Knowledge Distillation
non-IID

Abstract: In real-world applications, Federated Learning (FL) meets two challenges: (1) scalability, especially when applied to massive IoT networks, and (2) how to be robust against an environment with heterogeneous data. Realizing the first problem, we aim to design a novel FL framework named Full-stack FL (F2L). More specifically, F2L utilizes a hierarchical network architecture, making extending the FL network accessible without reconstructing the whole network system. Moreover, leveraging the advantages of hierarchical network design, we propose a new label-driven knowledge distillation (LKD) technique at the global server to address the second problem. As opposed to current knowledge distillation techniques, LKD is capable of training a student model, which consists of good knowledge from all teachers' models. Therefore, our proposed algorithm can effectively extract the knowledge of the regions' data distribution (i.e., the regional aggregated models) to reduce the divergence between clients' models when operating under the FL system with non-independent identically distributed data. Extensive experiment results reveal that: (i) our F2L method can significantly improve the overall FL efficiency in all global distillations, and (ii) F2L rapidly achieves convergence as global distillation stages occur instead of increasing on each communication cycle.

Under review as a conference paper at ICLR 2023
TOWARDS SCALABLE AND NON-IID ROBUST HIERARCHICAL FEDERATED LEARNING VIA LABEL-DRIVEN KNOWLEDGE AGGREGATOR
Anonymous authors Paper under double-blind review
ABSTRACT
In real-world applications, Federated Learning (FL) meets two challenges: (1) scalability, especially when applied to massive IoT networks, and (2) how to be robust against environment with heterogeneous data. Realizing the first problem, we aim to design novel FL framework named Full-stack FL (F2L). More specifically, F2L utilizes hierarchical network architecture, making extending the FL network accessible without reconstructing the whole network system. Moreover, leveraging the advantages of hierarchical network design, we propose new label- driven knowledge distillation (LKD) technique at the global server to address the second problem. As opposed to current knowledge distillation techniques, LKD is capable of training a student model, which consists of good knowledge from all teachers' models. Therefore, our proposed algorithm can effectively extract the knowledge of the regions' data distribution (i.e.. the regional aggregated models) to reduce the divergence between clients' models when operating under the FL system with non-independent identically distributed data. Extensive experiment results reveal that: (i) our F2L method can significantly improve the overall FL efficiency in all global distillations, and (ii) F2L rapidly achieves convergence global distillation stages occur instead of increasing on each communication cycle.
INTRODUCTION
Recently, Federated Learning (FL) is known as a novel distributed learning methodology for en- hancing communication efficiency and ensuring privacy in traditional centralized one McMahan et al. (2177). However, the most challenge of this method for client models is non-independent and identically distributed (non-III) data, which leads divergence into unknown directions. Inspired by this, various works on handling non-lID were proposed in Li et al. (2020); Acar et al. (202 1); Dinh ct al. (202 a Karimireddy et al. (2020); Wang et al. (2020); Zhu et al. (202 I); Nguyen et al. (2022b). However, these works mainly rely arbitrary configurations without thoroughly understanding the models' behaviors, yielding low-efficiency results. Aiming to fulfil this gap. in this work, we propose hierarchical FI. framework using information theory by taking deeper observation of the model's behaviors, and this framework can be realized for various FL systems with heterogeneous data. In addition, our proposed framework can trigger the FL system be more scalable, controllable. and accessible through hierarchical architecture. Historically, anytime a new segment (i.c., new group of clients) is integrated into the FL network, the entire network must be retrained from the beginning. Nevertheless, with the assistance of LKD, the knowledge is progressively transferred during the training process without information loss owing to the empirical gradients towards the newly participated clients" dataset.
The main contributions of the paper are summarized as follows. (1) We show that conventional FLs performance is unstable in heterogeneous environments due to non-III and unbalanced data by carefully analyzing the basics of Stochastic Gradient Descent (SGD). (2) We propose a new multi-teacher distillation model, Label-Driven Knowledge Distillation (LKD), where teachers can only share the most certain of their knowledge. In this way, the student model can absorb the most meaningful information from each teacher. (3) To trigger the scalability and robustness against non-lII data in FL, we propose a new hierarchical FL framework. subbed Full-stack Federated Learning (F2L.. Moreover, guarantee the computation cost at the global server, F21. architecture

---

Under review as a conference paper at ICLR 2023
integrates both techniques: LKD and FedAvg aggregators at the global server. To this end, our framework can do robust training by LKD when the FL process is divergent (i.e,, at the start of the training process). When the training starts to achieve stable convergence, FedAvg is utilized to reduce the server's computational cost while retaining the FL performance. (4) We theoretically investigate our LKKD) technique to make brief comparison in terms of performance with the conventional Multi-teacher knowledge distillation (MTKD), and in-theory show that our new technique always achieves better performance than MTKD. (5) We validate the practicability of the proposed LKD and F2L via various experiments based on different datasets and network settings. To show the efficiency of F2L in dealing with non-IID and unbalanced data, we provide a performance comparison and the results show that the proposed F2L architecture outperforms the existing FL methods. Especially, our approach achieves comparable accuracy when compared with FedAvg (MeMahan al. (2017)) and higher 7 - 20% in non-IlI settings.
2
RELATED WORK
2.1 FEDERATED LEARNING ON NON-IID DATA
To narrow the effects of divergence weights, some recent studies focused on gradient regularization aspects Li et al. (2020): : Acar et al. (2021); Dinh et al. 10; Karimireddy et al.) (2020); Wang ct. (2020); Zhu et al. (2021) Nguyen et (2022b). By using the same conceptual regularization. the authors in Li et al. (200)); Acar et al. (2021), and Dinh et al. (2021a) introduced the FedProx, FedDyne, and FedU, respectively, where FedProx and FedDyne focused on pulling clients' models back to the nearest aggregation model while FedU's attempted to pull distributed clients together. To direct the updated routing of the client model close to the ideal server route, the authors in Karimireddy et al. (2020) proposed SCAFFOLD by adding control variate to the model updates. Meanwhile, to prevent the aggregated model from following highly biased models, the authors in Wang al. (2020) rolled out FedNova by adding gradient scaling terms to the model update function. Similar to Dinh et al. (2021a), the authors in Nguyen et al. (2022b) launched the WALF by applying Wasserstein metric reduce the distances between local and global data distributions. However, all these methods are limited in providing technical characteristics. For example, Wang et al. (2020) demonstrated that FedProx and FedDyne are ineffective in many cases when using pullback to the globally aggregated model. Meanwhile, FedU and WAFL have the same limitation on making a huge communication burden. Aside from that, FedU also faces very complex and non-convex optimization problem.
Regarding the aspect of knowledge distillation for FL, only the work in Zhu et 2222 proposed a new generative model of local users as an alternative data augmentation technique for FL. However, the majority drawback of this model is that the training process at the server demands a huge data collection from all users, leading to ineffective communication.
Motivated by this, we propose a new FL architecture that is expected to be more elegant, easier to implement, and much more straightforward. Unlike Dinh et al. (2021a); Acar et al. (202T); )); Karimireddy et al. (2020), we utilize the knowledge from clients' models to extract good knowledge for the aggregation model instead of using model parameters to reduce the physical distance between distributed models. Following that, our proposed framework can flexibly handle weight distance and probability distance in efficient way, i.e., I|p" (y = c) p(y c)]l (please refer to Appendix B
2.2 MULTI-TRACCER KNOWLEDGE DISTILLATION
MTKD is an improved version of KD (which is presented in Appendix A.2 . in which multiple teachers work cooperatively to build a student model. As shown in Zhu et al. (018), every MTKD technique solves the following problem formulation:
P1: : min CKL TJ = Σ Σ p(lX (,,TT T) log p(] TT r=1
R
x w
here, r E {R) are the teachers' indices. By minimizing P1, the student 3 can attain knowledge from all teachers. However, when using MTKD, there are some problems in extracting the knowledge distillation from multiple teachers. In particular, the process of distilling knowledge in MTKD is typi- cally empirical without understanding the teacher's knowledge (i.e., aggregating all KI. divergences
2

---

Under review as a conference paper at ICLR 2023
Globd cnaeen
Resiooal Estente Neteak
Nivicl
reedel
Deapeel
Doo
Made paaendaa
Regical Tenqpemaeed sclres
Seft
Aggregatee
regeeal nall
Tasd: diver Disttaton
Dispee
Figure 1: The architecture of our F21. framework.
between each teacher and the student). Therefore, MTKD is unable to exploit teachers' detailed predictions for the KD (e.g.. Liu et al. (2020c), Asif et al. (2019), Zhu et al. (2018), Fukuda et al. (2017). Tran al. (2020)) Another version of MTKD, KTMDs can only apply for better teachers to distill knowledge (e.g-. [Shen et al. (2019), Zhu & Wang (022)) Zhang et al. (022)) [onn et al. (202T). ). For example, as provided in (Shen et al. 2019 eq. 6), the student only selects the best teacher to operate the knowledge distillation. Visually, this technique is the same the way of selecting a teacher among set of teachers to carry out a single teacher distillation. Therefore, the student's performance is always bounded by the best teacher's performance. Another popular research direction in MTKD is to leverage the advantage of the gap between teachers' hidden class features. However, owing the lack of explanatory knowledge in teachers' hidden layers, the method in ZZuu & Wang (2021) cannot obtain better student performance when compared to their teachers. Generally, current MTKD techniques cannot extract good knowledge from different customer models, leading to weight divergence in FL.
FULL-STACK FEDERATED LEARNING
3.1
THE F2L FRAMEWORK
The main objective of our work to design a hierarchical FI. framework, in which a global server manages set of distinct regional servers. Utilizing hierarchical FL, our proposed algorithm can achieve computation and computation efficiency. The reason is that Hierarchical FL makes the clients to train sectionally before making the global aggregation Liu et al. (2020a); : Briggs et al. (2020). Consequently, FL inherits every advantage from mobile edge intelligence concept over traditional non-hierarchical networks (e.g., communication efficiency, scalability, controlability) Pham al. (2020): Luong et al.] (0166): Lim et al. (2220). At the end of each knowledge-sharinn episode, the regions (which are supervised by regional servers) cooperate and share their knowledge (each region functions as distinguished FL system, with a set amount of communication rounds per episode).
In each episode, each region randomly selects group of clients from the network to carry out the aggregation process (c.g., FedAvg, FedProx); therefore, each region functions as a small-scale FL network. As a result, there are always biases in label-driven performance by applying random sampling on users per episode (see Appendix F.. Given the random sampling technique applied to the regional client set, the regions always have different regional data distributions. Consequently, various label-driven performances of different teachers might be achieved.
At the global server, our goal is to extract good performance from regional teachers while maintaining the salient features (c.g., understanding of the regional data distributions) of all regions. As result, we can capture useful knowledge from diverse regions in each episode using our proposed innovative knowledge distillation technique (which is briefly demonstrated in Section 3.2). We train the model on the standard dataset on the central server to extract knowledge from multplee teachers into the global student model. The preset data pool on the server S is used verify the model-class reliability and generate pseudo labels.
The system model is illustrated in Fig. 1 and thoroughly described in Appendix
C
The pseudo
3

---

Under review as a conference paper at ICLR 2023
algorithm for F2L is demonstrated in Algorithm 1 When the FL process suffers from client- drift Karimireddy et al. (2020) (i.e., the distribution of label-driven accuracies of different regions have large variance), the F2L framework applies LKD to reduce the class-wise performance gaps between regions (i.e., the regions with better performance on a particular class share their knowledge to regions with low performance). As result, the FL. network achieves significantly faster convergence when utilizing LKD (which is intensively explained in Section (.2) ) for the global aggregation process. When the generalization gap between regions is considerably reduced (i.e.,
max, 0 min, 8i1l S <), our F2L network becomes vanilla FL to reduce computation and communication costs. To this end, our method can achieve computation efficiency while showing robustness to the non-IID data in the network. Additionally, whenever new set of clients are added into the network and makes positive contributions to the FL system (e.g., max, _ - min, /851 2 e where || max, || a corresponding to the new region's 's performance) the LKD aggregator can be switched back to improve the FL system's performance over again.
3.2 LABEL-DRIVEN KNOWLEDGE DISTILLATION
To extract knowledge from multiple teachers to the global student model, we train the model on the standard dataset on the central server. The preset data pool on the server S is used to verify the model-class reliability and generate pscudo labels. In our work, the MTKD process executes two main tasks: (1) extracting the teachers' knowledge and (2) maintaining the students' previous performance.
To comprehend the LKD method, we first revisit the conventional MTKD, where the probabilistic output calculated by model on 24, the prediction label is p(] w, w,T.c) and its relation is: [p w, T), if argmax [p(1]a,, w. T)] c, otherwise.
01| w, T.c)
0,
On the one hand, we aim to transfer knowledge from different regional models to the global one. Inspired by Hinton et al. (2015), we use the Kullhack-Leibler (KI.) divergence between each regional teacher and the global logits as method to estimate the difference between two models' performance. The relationship is expressed as follows:
p (H,;, (T,c) x log 1' (1 w", .c.)) ( wa T.))²
CKL Σ
(3)
1=
where the number of samples of the fixed dataset S on the server. (X YTE is the dataset which is pseudo labeled and aligned by regional model r and [g] [ig(e) [c]) represents the set of data with size of S% labeled by the model r as c. Although the same preset dataset utilized on every teacher model, the different pseudo labeling judgments from different teachers lead to the different dataset tuples. The process of identifying S is demonstrated
Algorithm 3 Because the regional
models label on the same dataset S, we have Σ S = S for all regional models. D (p") (D²) is the e label-driven KL divergence between model and model g.
On the other hand, we aim to guarantee that the updated global model does not forget the crucial characteristics of the old global model. Hence, measure the divergence between the old and the updated model, we introduce the following equation:
19[122, w old: T.c)
p0 (1)21, Sid T,c) x log
(4)
pa (1 wh.. T,c)²
where Wold is the old parameters set of the global model which distilled in the last episode of F2L. More details about the label-driven knowledge distillation discussed in Appendix G
To compare the performance between LKD and MTKD, we consider the following assumption and lemmas:
Lemma Given of is the c-label driven predicting accuracy on model r. Let of Pr. model's 's variance and mean, respectively. The optimal value of variance and mean on stu- dent model (i)
or.e be the
yields otkD.g.
Σx oF ER e ooo and pixD.a.c
E ER= et Hr...
4

---

Under review as a conference paper at ICLR 2023
Proof:
The proof is provided in Appendix
Assumption Without loss of generality, we consider distinct regional models whose accuracy satisfy the following prerequisites ofe hh.c .... R. and |11, Juel |12,c Pe S ...
|PR,e- - Fel (Re denoted as an empirical global mean of the dataset on class c).
Lemma 2 Given the of models with variance satisfy of. I o3.0 S ... S RR the models' accuracy have the following relationship of 2 72 2. T
c
Proof. The proof can be found Appendix
K
Theorem
Let o o be the class-wise variance of the student model, and o gT be the class- wise variance of the model of teacher v, respectively. We always have the student variance using LKD technique always lower than that using MTKD:
LKDDgg.c S +2 MIKD.g.e'
+2
(5)
Proof: For the complete proof see Appendix
H
Theorem Let gix the empirical c-class-wise mean of the student model, and PATTD be of the model of teacher r, respectively. We always have the student
the empirical c-class-wise empirical mean using LKD technique always closer to the empirical global dataset's 's class-wise mean (De) than that using MTKD:
|LiKD.g.c Pel S [UMIKD.g.C /el.-
(6)
Given Theorems and 2 we can prove that our proposed LKD technique can consistently achieve better performance than that of the conventional MTKD technique. Moreover, by choosing the appropriate LKD allocation weights, we can further improve the LKD performance over MTKD. Due to space limitation, we defer the proof to Appendix
3.3 CLASS RELIABILITY SCORING
The main idea of class reliability variables 38. & LKD is weigh the critical intensity of the specific model. Therefore, we leverage the attention design from Vaswani et al. (2017) to improve the performance analysis of teachers' label-driven.
For regional models with disequilibrium non-lID data, the teachers only teach the predictions relying upon their specialization. The prediction's reliability can be estimated by leveraging the validation dataset on the server and using the function under the curve (AUC) as follows:
g Er=t expl
where Aiic denotes the AUC function classifier of the regional model Since AUC provides certainty that specific classifier can work on a label over the rest, we use the surrogate softmax function to weigh the co-reliability among the same labeling classifiers different teacher models. For simplicity, we denote & as the AUC on each labeling classifier:
exp(S (w))
8 fo (w,) exp(JAUC (C..TTT)))
(
exp(fAAC
In the model update class reliability, instead of calculating the co-reliability between teachers, equation [ compares the performance of the previous and current global models. Moreover, we introduce a temperatured value for the class reliability scoring function, denoted as Tu, By applying a large temperatured value, the class reliability variable sets 35, and 8 make a higher priority on the better performance (i.e., the label-driven performance on class c from teacher r. e.g., f in equation equation [ or class from old model Wold in equation equation ()). By this way, we preserve the useful knowledge which is likely ignored the new distillation episode. The more detailed descriptions of class reliability scoring are demonstrated in Algorithm
5

---

Under review as a conference paper at ICLR 2023
3.4 JOINT MULTI-TEACHER DISTILLATION FOR F2L
We obtain the overall loss function for online distillation training by the proposed F2L:
R
L1L = A1 Σ
KL + X2 + Xa AgLCEE
(9)
r=1
where \1, A2, A are the scaling coefficients of the three terms in the joint loss function. The first and second terms imply the joint LKD from the regional teacher models and the updating correction step, respectively. Moreover, to ensure that knowledge the student receives from teachers is accurate and can be predicted accurately in practice, the real data. Thus, we also compute the "standard" loss between the student and the ground- -rutth labels of the train dataset. This part is also known as the hard estimator, which different from the aforementioned soft-distillator. The hard loss equation is as follows:
need to validate the quality of the student model on
Cec H,,,H w
,T,,,,,))) ,T)
Σ y1 ( ,
(10)
Algorithm F2L framework
Require: Initialize clients' weights, global aggregation round, number of regions R, arbitrary c. while not converge do for all regions r E (1,2,... ,..,,,P R) do
for all user in regions do Apply FedAvg on regions r. end for
Send regional model w" to the global server. end for
if reach global aggregation round then
max, B min, 3 2 where 3 = {3, 18,,...,,87 H from Algorithm then then
Apply LKD as described in Algorithm
else
w L/R w
end if end if end while
We use the temperature coefficient T to calculate the class probability for this hard loss. The overall training algorithm for LKD is illustrated in Algorithm [ In terms value selection for scaling coefficients, the old global model can be considered as additional regional teacher's model in the same manner, in theory. Therefore, X2 should be chosen as:
\1 if update distillation in equation 4 is considered, otherwise,
(11)
0.
where is the number of regions decided by our hierarchical FL settings. With respect As, the value is always set as:
R+1 X1, if update distillation in equation 4 is considered,
A
(12)
A otherwise.
3.5 DISCUSSIONS ON THE EXTENT PROTECTING PRIVACY
In its simplest version, our proposed F2L framework, like the majority of existing FL approaches, necessitates the exchange of models between the server and each client, which may result in privacy leakage due to, for example, memorization present in the models. Several existing protection methods can be added to our system in order to safeguard clients against enemies. These include adding differential privacy /eyer et al. (2017) to client models and executing hierarchical and decentralized model fusion by synchronizing locally inferred logits, for example on random public data, as in work Chang et al. (019)) We reserve further research this topie for the future.

---

Under review as a conference paper at ICLR 2023
Table 1: The top-1 test accuracy of different baselines different data settings. The indicates the non-IID degree of the dataset (the lower value of a means that the data is more heterogeneous).
Dataset
FedAvg
FedGen
FedProx
Fed-
F2L.
FedAvg
FedGin FedProx
Fed- Distill
F2L.
Distill (Ours)
(Ours)
Dirichlet (a
1)
Dirichlet (a
0.1)
EMNIST
71.66
78.70
70.77
75.56
81.14
59.10
68.24
58.88
46.03
68.31
CIFAR-10
60.48
59.21
63.72
62.36
71.22
47.07
47.08
47.05
45.67
55.22
CIFAR-100
36.17
40.26
36.3
34.88
50.33
21.31
28.96
20.43
16.15
31.07
CINIC-10
65.23
71.61
65.15
67.77
74.85
47.55
52.35
48.2
47.1
57.12
CelebA
70.82
75.43
71.07
68.59
81.65
63.58
70.14
66.33
62.91
74.14
EXPERIMENTAL EVALUATION
4.1 COMPARISON WITH FL METHODS
We run the baselines (see Section B and compare with our F2L. Then, we evaluate the comparisons under different non-IID ratio. More precisely, we generate the IID data and non-IID data with two different Dirichlet balance ratio: shown in Table the F2L can outperform the four baselines with significant increase in accuracy. The reason for this phenomenon is that the LKD technique selectively draws the good features from regional models build global model. Hence, the global model predicts the better result on each different class and the entire accuracy of the global model then increases tremendously. The significant impact when applying LKD to distill different teachers to one student shown in Table 2
{1, 10). The comparison results are presented in Table I As
4.2 COMPUTATION EFFICIENCY F2L
To evaluate the computation efficiency of our proposed F2L process. we compare our F2L process with 3 benchmarks: (i) F2L-noFedAvg (aggregator only consists of LKD), (ii) vanilla FL (FL with flatten architecture and FedAvg as an aggregator), and (iii) flatten LKD (FL with flatten architecture based with LKD as alternate aggregator). Fig. [(a) shows that the F21. system can achieve convergence as good as the F2L-noFedAvg. The reason that: after several communication rounds, the distributional distance between regions is reduced thanks to the LKD technique. Hence. the efficiency the LKD technique on the data decreased. As a consequence, the LKD technique shows significant robustness FedAvg aggregator. In the non-hierarchical settings, the flatten LKD and FedAvg reveal under-performed compared to the proposed hierarchical settings. We assume that the underperformance above comes from the data shortage of clients' training models. To be more detailed, the clients' dataset are considerably smaller than that of the "regional dataset".. Thus, the regional models contain more information than the clients' models. We believe that: in the LKD technique, teachers' models require a certain amount of knowledge to properly train a good student (i.e., the global model). Given the convergence rate from Fig. 2(a) and the computation cost the
MM
120
(a)
(b)
Figure 2: Performance benchmarks of F2L under different settings. Fig. 2(a) reveals the convergence. Fig. ( shows the computational cost, and Fig. [(c] demonstrats the F2L convergence when new set of clients are added into the FL system (i.e., at communication round 100).
7

---

Under review as a conference paper at ICLR 2023
Table 2: Top-1 accuracy of F21. datasets MNIST, EMNIST, CIFAR-100, CINIC-10 and CelebA.
The data's ' heterogeneity is set at a = 0.1 on 110, MNIST, CINIC-10 and CelebA. We use
EMNIST "unbalanced" evaluate in this test. The "before update" and "after update" denote the teacher models' accuracies before and after the global distillation, respectively.
MNIST
EMNIST
CIFAR-100
CINIC-10
Celeb-A
before
after
before
after
before
after
before
after
before
after
update
update
update
update
update
update
update
update
update
update
Teacher
61.02
95.19
73.27
84.09
20.11
35.41
43.8
46.59
62.37
67.98
Teacher 92.49
98. 22
78.80
83.62
18.82
31.2
42.15
46
63.79
72.33
Teacher 81.60
97.63
80.5
84.10
22.40
34.93
40.02
42.15
64.05
69.44
G-student 98.71
84.11
37.68
47.65
70.12
server on Fig. 2(b) we can see that, by using the adaptive switch between LKD and FedAvg in F2L, achieve significant computational efficiency at the aggregation server. Note that F2L can substantially increase performance and computation efficiency compared with non-hierarchical architecture.
4.3 SCALABILITY
This section evaluates the F2L scalability. To do so, we inject a group of clients with non-llI data into our FL system after 100 rounds (when the convergence becomes stable). Note that the FL system has never trained these data. The detailed configurations of our experimental assessments can be found in Appendix As it can be seen from Fig. 2(c) when a new group of clients are added to the FL system, the vanilla FL shows significant drop in terms of convergence. The reason is because of the distribution gap between the global model's knowledge and knowledge of the clients' data. Whenever new data with unlearned distribution is added to stable model, the model will make considerable gradient steps towards the new data distribution. Thus, the FedAvg takes considerable learning steps become stable again. In contrast, in F2L system, the learning from newly injected regions does not directly affect the learning of the whole FL system. Instead, the knowledge from the new domains is selectively chosen via the LKD approach. Thus, the LKD process does not suffer from information loss when new clients with non-lID data are added to the FL system.
4.4 LKD ANALYSIS
In this section, we evaluate the LKD under various settings to justify the capability of LKD to educate the good student from the normal teachers. Our evaluations are as follows.
Can student outperform teachers? To verify the efficiency of LKD with respect to enhancing student performance, we first evaluate F2L on MNIST, EMNIST, CIFAR-100. CINIC-10, CelebA dataset. The regions are randomly sampled from the massive FL network. In this way, we only evaluate the performance of L.D random teachers. Table shows top-1 accuracy the regional teacher and student models. The results reveal that LKD can significantly increase the global model performance compared with that of the regional models. Moreover, the newly distilled model can work well under each regional non-IID data after applying the model update.
To make a better visualization for the LKD's performance, we reveal the result of LKD on EMNIST dataset in terms of confusion matrix as in Fig. 3 As it can be seen from the figure, the true predictions represented by a diagonals of the matrices. A LKD performance is assumed to be well predicted when the value on diagonals is high (i.e., the diagonals' colors is darker), and the off-diagonals is low (i.e., the off-diagonals' colors are lighter). As we can see from the four figures, there are a significant reduce in the off-diagonals' darkness in the student performance (i.e., Fig. 3(d) comparing to the results in other teachers (i.c., Figures 33a) ((b) and 3(c)). Therefore, we can conclude that our proposed MTKD techniques can surpass the teachers' performance as mentioned in Section
Teachers can really educate student? We evaluate LKD under different soft-loss coefficients X1 (the scaling value X is set to 0). Thus,
d while the hard-loss factor is set at λ3 we can justify whether the robust performance of LKD comes from the joint distillation from teachers just the exploitation of data-on-server training. We evaluate LKD on six scaling values
8

---

Under review as a conference paper at ICLR 2023
(a)
(b)
(c)
Figure 3: The illustrative results of LKD on EMNIST dataset. Confusion matrices show the effective- ness of joint distillation on regional models. Figures (a), (b), and (c) are the confusion matrix before distillation of teacher's predictions in region 1, 2, and 3, respectively (see Appendix [E). Fig. (d) is the confusion matrix of predictions after distillation of student. The matrix diagonal demonstrates the true-predicted label of the model.
A1 = {0, 0.001, 0.01, 0.1, 1}. We evaluate on three dataset, including EMNIST, CIFAR-10 10, and CIFAR 100, and summarize the results in Tables and in Appendices. We can from the three tables that the LKD cap off with Aς = 0.01. Especially, when A3 = (which means the LKD) acts vanilla cross-entropy optimizer), the model accuracy reduces notably. This means that the LKD only uses hard-loss as a backing force to aid the distillation. Thus, our LKD is appropriate and technically implemented.
Required training sample size for joint distillation. To justify the ability of LKD under a shortage of training data, we evaluate LKD with six different Mlta-on-servee settings: {1, 1/2, 1/4, 1/6, 1/8, 1 1/10}, where is the sample ratio when compared with the original data-on- server demonstrated in Table 4 As we can see from the implementation results in three Tables 9 and TO in Appendices, the F2I. is demonstrated to perform well under a relatively small data-on- server. To be more specific, we only need the data-on-server to be 4 times lower than the average data-on-client to achieve robust performance compared with the vanilla FedAvg. However, suggest using the data-on-server be larger than the data from distributed clients to earn the highest performance for LKD. Moreover, due to the ability to work under unlabeled data, the data-on-server does not need to all labeled. We only need a small amount of labeled data to aid the hard-loss optimizer. Thus, the distillation data the server be updated from distributed clients gradually.
BROADER IMPACT AND LIMITATION
Due to the hierarchical framework of our proposed F2L, each sub-region acts like an independent FL process. Therefore, our F2L integrable with other current methods, which means that we can apply varying FL techniques (e.g.. FedProx, FedDyne, FedNova, NCFyen all (2022a) into distinct sub-regions to enhance the overall F2L framework. Therefore, architecture search (e.g., which FL technique is suitable for distinct sub FL region) for the entire hierarchical network essential for our proposed framework, which is the potential research for the future work. Moreover, the hierarchical framework remains unearthed. Therefore, a potentially huge amount of research directions is expected to be investigated (e.g., resource allocation Nguyen et al. (2022c); (Saputra et al. (2022 202T) Dinh al. (202Tb), and task offloading hierarchical Fi. Yang et al. (2021). However, our LKD technique still lacks of understanding about the teachers' models (e.g.. how classification boundaries on each layer impact on the entire teachers' performance). By investigating in explainable AI, along with layer-wise performance, we can enhance the LKD, along with reducing the unlabeled data requirements for the distillation process in the future work.
5 CONCLUSION
In this research, we have proposed an FL technique that enables knowledge distillation to extract the- the-good-feattee feature-atur--nly from clients to the global model. Our model is capable of tackling the FL's heterogeneity efficiently. Moreover, experimental evaluations have revealed that our F2L model outperforms all of the state-of-the-an the-art FL baselines in recent years.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Entropy. Relative Entropy and Mutual Information, chapter 2, pp. 12-49. 2005. ISBN
9780471200611.
Durmus Alp Emre Acar, Yue Zhao, Ramon Matas, Matthew Mattina, Paul Whatmough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. In International Conference Learning Representations, Dec. 2021.
Umar Asif, Jianbin Tang, and Stefan Harrer. Ensemble knowledge distillation for learning improved and efficient networks. arXiv preprint arXiv:1909.080%: 2019.
Sergey Bobkov, Mokshay Madiman, and Liyao Wang. Fractional generalizations of young and bruna-minkowski inequalities. Jun. 2011.
Christopher Briggs, Zhong Fan, and Peter Andras. Federated learning with hierarchical clustering of local updates to improve training on non-iid data. In 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1-9, 2020.
Hongyan Chang. Virat Shejwalkar, Reza Shokri, and Amir Houmansadr. Cronus: Robust and heteroge- neous collaborative learning with black-box knowledge transfer. arXiv preprint arXiv: 1279, 2019.
Hong- You Chen and Wei-Lun Chao. Fedbe: Making bayesian model ensemble applicable to federated learning. arXiv preprint MiXi:2009.01994, 2020.
Gregory Cohen, Saced Afshar, Jonathan Tapson, and André van Schaik. EMNIST: an extension of MNIST to handwritten letters, Mar. 2017.
Luke N. Darlow, Elliot J. Crowley, Antreas Antoniou, and Amos J. Storkey. Cinic-10 10 is not imagenet or cifar-10. Oct. 2018. URL Uttpp:sts://axxi..coomm
abs /aas/188 11100.110000 03505
W DelPozzo, C P L Berry, A Ghosh, T S F Haines, L P Singer, and A Vecchio. Dirichlet pro- cess Gaussian-mixtur model: An application localizing coalescing binary neutron stars with gravitational-wav observations. Monthly Notices of the Royal Astronomical Society, 479(1): 601-614, 06 2018. ISSN 0035-8711.
Canh Dinh, Tung T Vu, Nguyen H Tran, Minh N Dao, and Hongyu Zhang. FedU: A uni- fied framework for federated multi-task learning with laplacian regularization. arXiv preprint arXiv:2102.07148, 2021a.
Thinh Quang Dinh, Diep N. Nguyen, Dinh Thai Hoang, Pham Tran Vu, and Eryk Dutkiewicz. Enabling large-scale federated learning over wireless edge networks. In 2021 IEEE Global Communications Conference (GLOBECOM), pp. 01-06, 2021b.
Rick Durrett. Probability: Theory and Examples. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, edition, 2010.
T. Fukuda, Masayuki Suzuki, Gakuto Kurata, Samuel Thomas, Jia Cui, and Bhuvana Ramabhadran. Efficient knowledge distillation from ensemble of teachers. pp. 3697-3701, 08 2017.
Robin Geyer, Tassilo Klein, and Moin Nabi. Differentially private federated learning: A client level perspective. arXiv preprint arxiv:1712.07557 2017.
Haowei He, Gao Huang, and Yang Yuan. Asymmetric valleys: Beyond sharp and flat local minima. In Advances Neural Information Processing Systems, volume 32. Curran Associates, Inc., Dec. 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv: 15...55333.3111 Mar. 2015.
10

---

Under review as a conference paper at ICLR 2023
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.063330 2019.
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. SCAFFOLD: Stochastic controlled averaging for federated learning. In Proceedings of the 37th International Conference on Machine Learning, 13-18 Jul. 2020.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arxiiv66099 04866, 2016.
Alex Krizhevsky. Learning multiple layers features from tiny images. Technical report, Aug. 2009.
Y. LeCun, B. Boser, S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backpropagation applied handwritten zip code recognition. Neural Computation, 1(4):541-551, Dec. 1989.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database, 2010. URL http://yant
lecun
Qinbin Li, Yiqun Diao, Quan Chen, and Bingsheng He. Federated learning non-lID data silos: An experimental study. In IEEE 38th International Conference on Data Engineering (ICDE), pp. 965-978, 2022.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems, 2:429 450, 2020.
Wei Yang Bryan Lim, Nguyen Cong Luong, Dinh Thai Hoang, Yutao Jiao, Ling-Chang Liang, Qiang Yang, Dusit Niyato, and Chumyan Miao. Federated learning in mobile edge networks: A comprehensive survey. IEEE Communications Surveys Tutorials, 22(3):2031 2020.
Lumin Liu, Jun Zhang, S.H. Song, and Khaled B. Letaief. Client-edge-cloud hierarchical federated learning. In ICC 2020 2020 IEEE International Conference on Communicaatons (ICC), pp. 1-6, 2020a.
Shengchao Liu, Dimitris Papailiopoulos, and Dimitris Achlioptas. Bad global minima exist and sgd can reach them. Advances in Neural Information Processing Systems, 33:8543-8552 2020b.
Yuang Liu, Wei Zhang, and Jun Wang. Adaptive multi-teacher multi-level knowledge distillation. Neurocomputing, 415:106-11 2020c.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), Dec. 2015.
Nguyen Cong Luong, Dinh Thai Hoang, Ping Wang, Dusit Niyato, Dong In Kim, and Zhu Han. Data collection and wireless communication in internet of things (IoT) using economic analysis and pricing models: A survey. IEEE Communications Surveys Tutorials, 18(4):2546-255 2016.
David J. C. MacKay. Information Theory, Inference, and Learning Algorithms. Copyright Cambridge University Press, 2003.
H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. Communication--fficine learning of deep networks from decentralized data, 2017.
Konstantin Mishchenko, Eduard Gorbunov, Martin Takáč, and Peter Richtárik. Distributed learning with compressed gradient differences. arXiv preprint arXiv: 190.0099,,,,929, 2019.
Arvind Neelakantan, Luke Vilnis, Quoc V. Le, Ilya Sutskever, Lukasz Kaiser, Karol Kurach, and James Martens. Adding gradient noise improves learning for very deep networks, Nov. 2015.
Dunng Nguyen, Sang-Min Lee, Quoc-Viae Pham, Dinh Thai Hoang, Diep N. Nguyen, and federated
Won-Joo Hwang. HCFL: A high compression approach for communication-effin learning in very large scale loT networks. IEEE Transactions on Mobile Computing, pp. 1-13, Jun. 2022a.
11

---

Under review as a conference paper at ICLR 2023
Tung-Anh Nguyen, Tuan Dung Nguyen, Long Tan Le, Canh T Dinh, and Nguyen H Tran. On the generalization of Wasserstein robust federated learning. arXiv preprint arXiv:2206.01432, 2022b.
Xuan-Tung Nguyen, Minh-Duong Nguyen, Quoc- Quoc-Viet Pham, Vinh-Quang Do, and Won-Joo Hwang. Resource allocation for compression-aidee federated learning with high distortion rate. arXiv preprint arXiv:2206.06977 2022c.
Quoc- Viet Pham, Fang Fang. Vu Nguyen Ha, Md. Jalil Piran, Mai Le, Long Bao Le, Won-Joo Hwang. and Zhiguo Ding. A survey of multi-access edge computing in 5G and beyond: Fundamentals, technology integration, and state-of-the-art. IEEE Access, 8:116974-117017, 2020.
Herbert Robbins and Sutton Monro. stochastic approximation method. The Annals of Mathematical Stalistics, 22(3):400-400 Sep. 1951.
Yuris Mulya Saputra, Diep N. Nguyen, Dinh Thai Hoang, and Eryk Dutkiewicz. Incentive mech- anism for ai-based mobile applications with coded federated learning. In 2021 IEEE Global Communications Conference (GL.BECOM)) pp. 1-6, 2021.
Juris.Mulya. Saputra, Diep Nguyen, Hoang. Thai. Dinh, Quoc-Vic Pham, Eryk Dutkiewicz, and Won-Joo Hwang. Federated learning framework with straggling mitigation and privacy-awareness for ai-based mobile application services. IEEE Transactions on Mobile Computing, pp. 1-1, 2022.
Chengchao Shen, Mengqi Xue, Xinchao Wang, Jie Song, Li Sun, and Mingli Song. Customizing student networks from heterogeneous teachers via adaptive knowledge amalgamation. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprint arXiv: 1103v:1170000//00 2017.
Samuel Smith, Erich Elsen, and Soham De. On the generalization benefit of noise in stochastic gradient descent. In Proceedings of the 37th International Conference on Machine Learning, pp. 9058-9067, 13-18 Jul 2020.
Wonchul Son, Jaemin Na, Junyong Choi, and Wonjun Hwang. Densely guided knowledge distillation using multiple teacher assistants. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9395-9404, October 2021.
Linh Tran, Bastiaan S. Veeling, Kevin Roth, Jakub Swiatkowski. Joshua V. Dillon, Jasper Snoek, Stephan Mandt, Tim Salimans. Sebastian Nowozin, and Rodolphe Jenatton. Hydra: Preserving ensemble diversity for model distillation. CoRR, abs/2001 1.4614, Mar. 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H. Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. In H. Larochelle. M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 7611-7623. Curran Associates, Inc., Dec. 2020.
Blake Woodworth, Kumar Kshitij Patel, Sebastian Stich, Zhen Dai, Brian Bullins, Brendan Mcmahan, Ohad Shamir, and Nathan Srebro. Is local SGD better than minibatch SGD? In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 10334-10343. PMLR, 13-18 Jul. 2020.
Zhaohui Yang, Mingzhe Chen, Walid Saad, Choong Seon Hong, and Mohammad Shikh-Bahaci Energy efficient federated learning over wireless communication networks. IEEE Transactions on Wireless Communications, 2033):1935-194) 1949, 2021.
12

---

Under review as a conference paper at ICLR 2023
Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, and Yasaman Khazaeni. Bayesian nonparametric federated learning of neural networks. Kama- lika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Con- ference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. ings. ml mwt.m...rre///7//71
7252-7261. PMLR, 09-15 Jun. 2019. URL yurochkin19a.hes html
Hailin Zhang, Defang Chen, and Can Wang. Confidence-aware multi-teacher knowledge distillation. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4498 4502.. IEEF, 2022.
Yue Zhao, Meng Li, Liangzhen Lai. Naveen Suda, Damon Civin, and Vikas Chandra. Federated Learning with non-lID Data, Jun. 2018.
Xiatian Zhu, Shaogang Gong, et al. Knowledge distillation by on-the-fly native ensemble. Advances in neural information processing systems, 31, 2018.
Yichen Zhu and Yi Wang. Student customized knowledge distillation: Bridging the gap between student and teacher. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 5057-5066, October 2021.
Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou. Data-frec knowledge distillation for heterogencous federated learning. In International Conference Machine Learning, pp. 12878-12889, PMLR, 2021.
13


Reviews:
[{"review_id": "v_ZGXu0TZG", "reviewer": "Reviewer_3LcY", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a novel FL framework called Full-stack FL (F2L). It aims at solving two important problems in FL - scalability, and robustness in the presence of heterogeneous data. To solve the first problem, they propose a hierarchical design in FL where several smaller FL systems are connected via a global server. To handle heterogeneity, they propose a new label-driven knowledge distillation (LKD) technique at the global server which leverages the advantage of the hierarchy in the design to learn useful information from the sub-regions in the network to achieve fast convergence by reducing the generalization gap between regions. ", "summary_of_the_review": "Both hierarchical learning and KD exist in literature. The authors claim their KD method is novel though. The contributions are significant only if we have a well represented root dataset in my opinion, which may not be able to cope with scalability in practice, which is in fact the main contribution claim of the paper.", "strength_and_weaknesses": "Strengths - \n1.\tThey offer to solve the two problems of scalability and data heterogeneity simultaneously.\n2.\tSince a hierarchical model is able to train the sub-regions before doing a global aggregation, the proposed design is claimed to also achieve computational efficiency.\n3.\tThe objective function for online distillation in Eq. (9) is well formulated that includes the goal of reducing the generalization gap between the regions and changes in the global model; this is so that the model does not forget the crucial characteristics of the old global model.\n4.\tEquations (11) and (12) recommend how to set the \\lambda_2 and \\lambda_3 in terms of \\lambda_1. This reduces the number of hyperparameters to be chosen by the user.\n5.\tF2L is compatible and integrable with other FL techniques. \n\nWeaknesses - \n1.\tThe main drawback of the design is that it relies heavily on a centrally available dataset. One of the two primary goals of the system is to handle non-IIDness in the data, which raises the question - how does the performance of F2L depend on the quality of the root dataset at the server. How well does the root dataset represent the non-IIDness present among the clients? How is scalability affected if the root dataset is not updated to well represent the newly joined clients? More experiments are required to convince the reader that the system can do well even when the root dataset does not exactly represent the data distribution among the clients.\n2.\tTable 1 shows that F2L performs significantly better than Fed-Distill. The lower performance of the other benchmarks can be attributed to the fact that they do not leverage any information from a root dataset. What essentially leads to this improvement with respect to Fed-Distill? Do they both use the same root datasets? Is Fed-Distill well tuned for best performance?\n3.\tFigure 2c shows the performance of F2L when a client is injected into the system midway during the training. F2L can be seen to perform better than vanilla FL. Can this be attributed to knowledge distillation? How would it compare with Fed-Distill? How sensitive are the observations with respect to the knowledge distillation parameters - lambda and temperature?\n4.\tF2L relies on switching between LKD and FedAvg after sufficient convergence has happened. How is this threshold chosen? What can be a general way to choose this value for any dataset?\n5.\tFigure 3 shows that a student can outperform a teacher in F2L. This experiment was performed on EMNIST. Does this observation hold in general, independent of the dataset? If not, what conditions does this depend on?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "gbz5zK4nBD", "reviewer": "Reviewer_jW2m", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a FL framework named F2L. It combines hierarchical network design and knowledge distillation to solve the problem of non-IID of data. Its contributions are threefold:(1) it shows the traditional FL algorithms are unstable for non-IID data. (2) it proposes a new knowledge distillation algorithm Label-Driven Knowledge Distillation. (3) it proposes a new FL algorithm F2L.", "summary_of_the_review": "See the Weaknesses.", "strength_and_weaknesses": "**Strength**\n\n(1) it shows the traditional FL algorithms are unstable for non-IID data. (2) it proposes a new knowledge distillation algorithm Label-Driven Knowledge Distillation. (3) it proposes a new FL algorithm F2L.\n\n**Weaknesses**\n\n(1) what is the meaning of the first step of FL, direct LKD may get better results. (2) an additional dataset (Reference Datapool in Fig. 1) is introduced. Whether the distribution of this dataset is similar to the distribution of the teachers' training dataset. The comparison with other methods is not fair and the experimental results are not convincing. (3) lack of explanation of some symbols in the paper makes it less readable. (4) some lines in Fig. 1 are misplaced. (5) some references are missing. Many FL works have introduced KD. For example, \n[1] Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learning. [2] FedBE: Making Bayesian Model Ensemble Applicable to Federated Learning. [3] FedMD: Heterogenous Federated Learning via Model Distillation. [4] Ensemble Distillation for Robust Model Fusion in Federated Learning.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "G4k-ss2-e7", "reviewer": "Reviewer_yujU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a hierarchical FL framework with a new label-driven distillation method to handle non-iid FL scenarios", "summary_of_the_review": "In general, I think that the hierarchy structure has potential in real FL systems. Meanwhile, the challenge of building such a system may be more in other parts, e.g., communication/node joining & dropping, instead of gradient aggregation. For the label-driven distillation, I think the authors' comparison with baselines is a bit unfair, as the authors' mechanism has more information (i.e., test data in the server).", "strength_and_weaknesses": "S1. The proposal of a hierarchical structure for FL is reasonable.\n\nW1. The paper is very hard to follow, as a lot of design considerations are proposed, but which parts are the most novel ones are unclear.\n\nW2. The comparison with baselines is unfair. It seems that the authors assume there is some test data in the server so that every region’s model can be validated for each label; then label-driven distillation can be proposed. However, the baselines seem not to have this assumption and can work well without the centralized test data. In other words, the authors’ method uses external knowledge (i.e., server test data), then performance improvement is expected.\n\nW3. In practice, it is very hard to collect test data for the server; even if the server can collect some, such data may be very biased (because many clients will not allow the server to collect data; this is actually why FL is needed). Then, the server’s validation accuracy may be doubtful. Hence, the application of the proposed mechanism in practice is doubtful.\n\nW4. The experiment settings are manually controlled so whether the regional hierarchy can work in practice is still unknown. From my reading of the paper, I think the mechanism may work well when different regions have non-iid samples (the experimental setting); however, if non-iid happens for the clients in a region, the mechanism may still fail. The authors need to use realistic data/region partitions instead manually controlled ones to validate the usefulness of the proposed mechanism.\n\nW5. While the hierarchy structure of FL is sensible, the authors may have real experiments to deploy hundreds of clients in different regions to verify the practicality of the hierarchy structure. Otherwise, how this structure is useful in practice is unclear.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject