Adaptive Self-Distillation for Minimizing Client Drift in Heterogeneous Federated Learning
Authors: M Yashwanth
Gaurav Kumar Nayak
Arya Singh
Yogesh Simmhan
Anirban Chakraborty
Venue: Accepted by TMLR ()

Abstract: Federated Learning (FL) is a machine learning paradigm that enables clients to jointly train a global model by aggregating the locally trained models without sharing any local training data. In practice, there can often be substantial heterogeneity (e.g., class imbalance) across the local data distributions observed by each of these clients. Under such non-iid label distributions across clients, FL suffers from the `client-drift’ problem where every client drifts to its own local optimum. This results in slower convergence and poor performance of the aggregated model. To address this limitation, we propose a novel regularization technique based on adaptive self-distillation (ASD) for training models on the client side. Our regularization scheme adaptively adjusts to each client's training data based on the global model's prediction entropy and the client-data label distribution. We show in this paper that our proposed regularization (ASD) can be easily integrated atop existing, state-of-the-art FL algorithms, leading to a further boost in the performance of these off-the-shelf methods. We theoretically explain how incorporation of ASD regularizer leads to reduction in client-drift and empirically justify the generalization ability of the trained model. We demonstrate the efficacy of our approach through extensive experiments on multiple real-world benchmarks and show substantial gains in performance when the proposed regularizer is combined with popular FL methods. The code is provided as supplementary material.

Published in Transactions on Machine Learning Research (December/ 20224)
Adaptive Self-Distillation for Minimizing Client Drift in Heterogeneous Federated Learning
M. Yashwanth
yushuanthon
Indian Institute of Science
Gaurav Kumar Nayak Indian Institute of Technology (ITT) Roorkee
Arya Singh
f20180763y@olammns wit.::iiiibbi---litti.....
Indian Institute of Science
Yogesh Simmhan Indian Institute of Science
simmban Wrisc.
Anirban Chakraborty Indian Institute of Science
anirban Wissc. ac
Reviewed on OpenReview: https: //::://ooppe openreview. net/ forum? id= K58m87DE4s
Abstract
Federated Learning (FL) is a machine learning paradigm that enables clients to jointly train global model by aggregating the locally trained models without sharing any local training data. In practice, there can often be substantial heterogencity (e.g., class imbalance) across the local data distributions observed by each of these clients. Under such non-iid label distributions across clients, FL suffers from the 'client-drift' problem where every client drifts to its own local optimum. This results in slower convergence and poor performance of the aggregated model. To address this limitation, we propose a novel regularization technique based on adaptive self-distillation (ASD) for training models on the client side. Our regularization scheme adaptively adjusts each client's training data based the global model's prediction entropy and the client-data label distribution. We show in this paper that our proposed regularization (ASD) can be easily integrated atop existing, state-of-the-art FL algorithms, leading to further boost in the performance of these off-the-shelf methods. We theorctically explain how incorporation of ASD regularizer leads to reduction in client-drift and empirically justify the generalization ability of the trained model. We demonstrate the efficacy of our approach through extensive experiments on multiple real-world benchmarks and show substantial gains in performance when the proposed regularizer combined with popular FL methods. The wttp:iiiccceaaaaatieeeeeeeeeeeee
link to the code is httpp:: https://githhbl
1 Introduction
Foderated Learning (FL) is maching learning paradigm where the clients collaboratively Iearn a shared model under the orchestration of the server without sharing any of their local training data with other clients or the server. Due to the
nature of FL, has found many applications in smartphones (Hard al. 2018 Ramaswamy al.] 2019). the Internet of Things (IoT), healthcare organizations (Ricke al. 2020 Xu al.) (202)), where training data generated edge devices or from privacy-sensition sensitive domains. As originally introduced in (McMahan et al. 2017). FI. involves model training across architecture consisting

---

Published in Transactions on Machine Learning Research (December 2024)
Test Accuracy Difference (line)
Accuracy Differncce
(horr
-.1
class Index
dass Index
client
client
Figure 1: Impact of one round of local training on the accuracy of two clients with different label distribution sampled from CIFAR-10 dataset: The effect of local learning on test accuracy is analyzed by mcasuring the change in accuracy before and after local training, with positive values indicating improved model performance. Interestingly, in scenarios where classes with low probability of occurrence under- represented, models trained using FedAvg frequently exhibit decline in accuracy post-training. In contrast, incorporating our proposed adaptive self-distillation regularizer (ASD) into FedAvg (FedAvg+ASD) not only effectively captures knowledge from well-represented classes but also preserves information about under- represented classes. A similar pattern is observed with FedNTD and FedNTD+ASD.
of one server and multiple clients. In traditional FL, each client securely holds its training data due to privacy concerns well to avoid large communication overheads while transmitting the same. At the same time, these clients aim to collaboratively train a generalized model that can leverage the entirety of the training data disjointly distributed across clients.
Data ingested at the edge/client devices are often highly heterogeneous as a consequence of the data generation process. They can differ in terms of quantity imbalance (the number samples at each client are different), label imbalance (empirical label distribution across the clients widely vary). and feature imbalance (features of the data across the clients are non-iid). When there exists label or feature imbalance, the objective for every client becomes different the local minimum for every client objective will be different. In such settings, during the local training, the client's 's model starts to drift towards its local minimum and farther away from the global objective. This is undesirable as the goal of FL is to converge to a global model that generalizes well across all the clients. This phenomenon, known as 'client-drift' is introduced and explored in carlier works (Karimireddy et al., 2020 Acar et al. 2021 Wang et al. 2021). In this work, we will be considering only the label heterogeneity. In any given FL round, the client initializes its model with global model weights and then starts training its model using the local data. Due to this, the client training often leads to overfitting the local data and cannot retain the knowledge acquired from the global model in earlier FL round.
Recently (He et al. 2022b introduced a class-wise adaptive weighting scheme (FedCAD) at the server side. The major drawback of FedCAD is that it assumes the presence of related auxiliary data and reliability on the server to compute the weights for the clients. Dependency on the server for computing the adaptive class-wise weights necessitates the availability of auxiliary data the server. Another work (Lee al. 2022) proposes FedNTD which poses the client-drift a local forgetting problem. It cannot mitigate client-drift effectively since it assigns uniform weights to regularization loss for all samples, independent of label distribution. Consequently, it treats high and low probability samples similarly, biasing the client model towards those with higher probability of occurrence, thus degrading performance. To address these issucs and motivated by client model regularization in mitigating client drift and lo remove the server's dependency on computing client-side weights, we introduce computationally efficient strategy known as Adaptive Delf-Distillation (ASD) for Federated Learning. Importantly, ASD does not require any auxiliary data. We use the KL divergence between the global and local models as the regularizer. For every sample, the weight assigned to the regularization loss is adaptively adjusted based the global model's prediction entropy and the
2

---

Published in Transactions on Machine Learning Research (December/ 2occeber/2222/ 22024)
empirical label distribution of the client's data. Specifically, when the server model encounters samples with high entropy, we reduce the weighting on the regularization loss, whereas, for samples with low probability of occurrence, we prioritize the learning from the global model. This adaptive approach enables local models to effectively learn from the cross-entropy loss for more frequent labels while leveraging the global model's guidance for less frequent labels. The adaptive weights are computed without relying on external or proxy data, unlike methods such as FedCAD which relies on external data. Morcover, the additional computational burden on clients is minimal, involving only single forward pass of the training data.
In Fig D we explain how the ASD regularization with adaptive weights helps mitigate the client-drift. We analyze the impact of client-drift by observing one round of local training on a particular client model with the CIFAR-10 dataset. We see that FedAvg substantially deteriorates the performance on the labels that have sparse or no representation in the client's local data. After adding the ASD loss, the impact is reduced. The ASD with (adaptive weights) performs the best in terms of local learning and preserving the global model knowledge on the sparse classes. We theorctically explain the client-drift reduction through our proposed ASD regularizer. In addition, we also provide justification on how ASD leads to improved generalization of the global model. This novel design of our proposed method allows the regularizer to be easily integrated atop any existing FL methods, and this results in substantial performance gains, making it an attractive and compelling solution to the federated learning problem. To the best of our knowledge, this is the first work where the adaptive weights are used for the distillation loss in the FI. framework without requiring access to auxiliary data and without the assistance of the server. We would like to clearly point out that the goal of this work is not to directly compete against any particular regularization method used in FL. Our proposed ASD regularizer is of true plug-and-play nature. With very negligible computational overhead (discussed additional regularization top of any off-tho-shelf FL method (either with
in Sec. B ASD can be used regularized FL methods such FedProx, FedDyn FL methods without regularization such as FedSAM, FedAvg etc.) and further boost their performance across the benchmark datasets such as CIFAR-100/ 10 and Tiny-ImageNet. for both IID and non-IID settings. As a validation, we combine our proposed method with some of the popular off- offtthesheelf FL methods such as FedAvg (McMahan et al. 2017). FedProx (Li et al., 2020), FedDyn (Acar al.) (022)) FedSpeed (Sun et al. 2023), FedNTD (Lee et al. 2022), FedSAM (Caldarola et al. 2022) and FedDisco (Ye et al.) 2023) and consistently observe performance improvement.
In summary, the key contributions of this work are:
We introduced novel computationally efficient regularization method ASD in the context of Federated Learning that alleviates the client drift problem by adaptively weighting the regularization loss for each sample based on the global model's prediction entropy and the label distribution of client data.
We demonstrate the efficiency of our method by extensive experiments on datasets such as CIFAR-10, 100, and Tiny-ImageNet datasets by combining our proposed ASD regularizer with the popular FL methods and improving their performance.
We present a theoretical analysis of the client-drift. and show that our regularizer minimizes the client-drift. We also empirically show that ASD promotes better generalization by converging to a flat minimum.
2 Related Work
2.1 Federated Learning (FL)
In recent times, addressing heterogencity in Federated Learning has become active area of research, and the field is developing rapidly. For brevity, we discuss few related works here. In FedAvg (McMahan et all 2017), the two main challenges explored are reducing communication costs (Yadav & Yadav 2016) and ensuring privacy by avoiding having to share the data. There are some studies based on gradient inversion (Geiping et al., 2020) raising privacy concerns owing to gradient sharing while some studies have proposed in defense of sharing the gradients (Kairouz et al. 2021 Huang et al.] 2021). ). FedAvg is the generalization of local SGD (Stich) 2018) by increasing the number of local updates, significantly reducing communication
costs for an iid setting, but does not give similar improvements for non-iid data. Several works perform
3

---

Published in Transactions on Machine Learning Research (December/ (iccebber/202g 22024)
SGD-type analysis that involves the fulll device participation, and this breaks the important constraint in FI. setup of partial device participation. Some of these attempt to compress the models to reduce the communication cost (Mishchenko et al. 2019). A few works include regularization methods on the client side (Zhu et al., 2021), and one-shot methods where clients send the condensed data and the server trains on the condensed data (Zhou et al. 2020). In (Hsu et al., 2020 an adaplive weighting scheme is considered on task-specific loss to minimize the learning from samples whose representation is negligible. Flatness-basod
methods based on SAM called as FedSAM is introduced in (Qu et al., 2022 Caldarola et al., 2022).
2.2 Client-Drift in FL
Due to data heterogeneity, federated training suffers from client drift. To address this, momentum-bant based server aggregation was proposed in (Wang et al.; Hsu al.) 2019)) which was later extended to handle any client and server updates in (Reddi et al.p FedProx (Li al. 2020) introduced a proximal term to penalize deviations of client weights from the globally initialized model. SCAFFOLD (Karimireddy et al.) 2020) tackled the issue as one of objective inconsistency, introducing a gradient correction term regularizer Subsequently, FedDyn (Acar et al. 2021) enhanced this with a dynamic regularization term. In (Kim et al. 2024h, a proximal term was introduced in the client's 's optimization based on the accelerated global model, and momentum was applied on the server to track its updates.
2.3 Federated Learning Using Knowledge Distillation
Knowledge Distillation (KD) introduced by (Hinton et al. 2015) is a technique to transfer the knowledge from a pre-trained teacher model to the student model by matching the predicted probabilities. Self-distillation was introduced in (Zhang et al.) 2019) where the student distills from the same model to the sub-networks of the model. The teacher model predictions are updated every batch. In our method, distillation happens with the full network, and the teacher's predictions are updated after every communication round. Adaptive distillation was used in (Tang et al. 2019). The server-side KD methods such as FedGen (Seo et al. 2022 use KD to train the generator at the server and the generator is broadcasted to the clients in the subsequent round. The clients use the generator to generate the data to provide the inductive bias. This method incurs extra communication of generator parameters along the model and training of the generator in general is difficult. In FedDF (Lin et al. 2020) KD is used at the server that relies on the external data. The KD is performed on an ensemble of client models, especially client models acts as a separate teacher model and then the knowledge is distilled into single student model (global model). In FedNTD (Loe al. 2022) the non-true class logits are used for distillation. This method gives uniform weights to all the samples. In FedCAD (He et al.) 2022b) and FedSSD) (He et al. 2022ab, the client-drift problem is posed as forgetting problem, and weighting scheme has been proposed. Importantly, the computation of adaptive weights of the client samples is done with the help of the server with the assumption that the server has access to auxiliary data. One shortcoming of this method is the assumption of the availability of auxiliary data on the server, which is impractical. In (Zhang et al. 2022) logits were calibrated based on the label distribution. This is totally different from our approach as we are adjusting the weights of the distillation loss. Unlike all of these approaches, we propose a novel ASD strategy that aims to mitigate the challenge of client drift due to non-iid data without relying on the server and access to any form of auxiliary data to compute the adaptive weights.
3 Method
We first describe the traditional federated optimization problem, then explain the proposed method of adaptive self-distillation (ASD) in section 3.2 We provide the theoretical and empirical analysis in the sections 3.3 and 3.4 respectively.
3.1 Problem Setup
We assume there is a single scrver/cloud and m clients/cdge devices. We further assume that client le has its own training dataset Dj with ny training samples drawn iid from the data distribution Pa y). The data distributions (T, k) across the clients are assumed to be non-iid. In this setup, we perform the
4

---

Published in Transactions on Machine Learning Research (December/ 2aannnber/2222/ 2022)
Client 1
Step 1: Broadcast w
Client
Client l
Adaptive Weights
Adaptive
Local
L
A5P
Server
Step Aggragato
Entropy Lowi,
116
Step
starting point, t-1
Figure 2: Federated Learning with Adaptive Self-Distillation: The figure describes the overview of the proposed approach based on Adaptive distillation. In Step 1. The server broadcasts the model parameters, In Step 2. clients train their models by minimizing both the cross entropy loss and predicted probability distribution over the classes between the global model and the client model by minimizing the KL divergence, the importance of each sample in the batch is decided by the proposed adaptive scheme as a function of label distribution and the KT, term. The server model is fixed while training the client. In Step 3. The server aggregates the client models based on FedAvg aggregation. The process repeats till convergence.
following optimization. (Acar al. 2021 McMahan al.
2017
org min Rd
f(w) A Σ [w()
(1)
kE[K]
where (w( is the client specific objective function and w denotes model parameters. The overall FL framework is described in detail in figure 2
3.2 Adaptive Self-Distillation (ASD) in FL
We now describe the proposed method where each client minimizes the (ww as defined below Eq. 2
fw(w)= & Lx(w) ALASD
(2)
La (w) is given below.
Lx(w) E (ww; (x, y))]
(3)
2.
Here, la is cross-entropy loss. The expectation is computed over training samples drawn from Pa v) of client k. This is approximated the empirical average of the losses corresponding to samples from the Dataset 'D-- T.ASP(w) in Eq. 2 denotes our proposed Adaptive Self-Distillation loss (ASD) term which considers label imbalance and quantifies how casily the predictions of the local model can drift from the global model. ASD loss is designed so that client models learn from the local data and at the same time not. drift too much from the global model. We define (ASD) Loss as follows.
ASD (w) A Blau(a, DKT (g (x, w w")llu((s w))]
(4)
In the above Liq. 4 w represents the global model parameters at FL round and represents the trainable model parameters of client k, initialized with w' at round t. ag(2, y) denotes the weight for the sample æ with label ground truth label y. For simplicity, we denote the global model softmax predictions (g(r, w") as
(x) and client model softmax predictions g (x, w) as (z). DKL is the KL divergence. The Eq. 4 can be approximated by the following equation for a mini-batch.
T,ASD (w)
Σ =E
onla (f'l'',')DDu.le D') Jll (lax
(5)
B
5

---

Published in Transactions on Machine Learning Research (December/ 2occemeer/2025 2224)
where B is the batch size, (a',y') DR. 99 and qu are softmax probabilities on the temperature (r) scaled logits of the global model and client model ht respectively. For a class below Eq. annd Eq. 7 holds.
exp (2g(z))/T
(f(x")
(6)
Emec exp (2)) (3((1///))
cxp(=f(x')/11
(((x') Emeo exp (2f (x*)/T)
(7)
where (2)),, (x(x") are the logits predicted on the imput x' by the global model and client model & respectively. The index denotes the ith sample of the batch. The DKL. (g, (1))))))))))) is given in Eq.
Σ (oglgg (x9()(()))
(8)
where C' is the number of classes. We use the simplified notation of for distillation weights ax ,²) and it is given in below Eq
&&
at Eien Ci
(9)
and de' is defined as below Eq. 10
exp( (ap(-H((x") H(x)))
(10)
Pk
where H(x') is the entropy of the global model predictions and is given by
11
H(x²) = (oog(g
(11)
DKL. in Eq. S captures how close the local model's predictions are to the global model for any given sample w!. Our weighting scheme in Eq. II decides how much learn from the global model for that sample based on the entropy H(x') of the server model predictions and the label distribution of the client data (ok ). H(x') captures the confidence of global model predictions, higher value implies the server predictions are noisy so we tend to reduce the weight, i.c, we give less importance to the global model if its entropy is high. The p is the probability that the sample belong to particular class. We give more weight to the sample if it belongs to the minority class. This promotes learning from the local data for the classes where the representation is sufficient enough and for the minority classes we encourage them to stay closer to the global model. In summary, the choice of alpha is designed to ensure that, when the global model encounters samples with high prediction entropy, we decrease the weighting on the regularization loss. Conversely, for samples with low probability of occurrence, we prioritize learning from the global model. This adaptive approach enables local models effectively learn from the cros--entropy loss for more frequent labels while leveraging the global model's guidance for less frequent labels. In Table 3 we highlight the importance of adaptive weights, where clearly show that ASD with adaptive weights consistently improves performance when combined with off-the-shelf FL methods. We approximate the label distribution P with the empirical label distribution, it is computed as Eq 12]
p EiEDDu Ty [Du]
(12)
where Ty denotes the indicator function and its value is if the label of the 8th training sample belongs to class e clse it is 0. To simplify notation, we use Ph for ph as it depends only on the class for client k. Finally we use Eq and Eq. 11 to compute the LAA (w) defined in Eq The choice of KL divergence in the Eq. is motivated by the seminal work of Hinton et.al., which aims to match the temperature raised softmax values between the pro-trained teacher model and student model for effective knowledge transfer. divergence and
We also analyzed the other statistical divergences such reverse KL and empirically found that KT. divergence is better. More details presented in the Sec. A.13 of the Appendix.

---

Published in Transactions on Machine Learning Research (December/ 2occmberr22024 22024)
3.3 Theoretical Analysis of Gradient Dissimilarity
In this section, we perform the theoretical analysis of the client drift. We now introduce the Gradient dissimilarity G& based on the works of (Li et al. 2020 Lee et al. 2022 as way to measure the extent of client-drift as below.
* E [w)]]]
Ga(w, A)
(13)
[V
Ga(w, A) is function of both the w and X. For convenience, we simply write Ga and mention arguments explicitly when required. (u(W) in the above Eq. 13 is same as Eq. 2 With this, we now establish a series of propositions to show that ASD regularization reduces the Gradient dissimilarity, which as a result, leads to lower client drift.
Proposition 3.1. inf Ga(w, A) is 1, A
The above proposition implies that if all the client's gradients are progressing in the same direction, which means there is no drift = The result follows from Jensen's incquality. The lower value of Ga is desirable and ideally 1. To analyze the Ga, we need V/w(w) which is given in the below proposition.
Proposition 3.2. When the class conditional distribution across the clients is identicol, i.c., P4(1 v) = P(r v) then fa(()) = E. + Xn( where ge. VB]((w;,,u) = c), 8c c and x Pk
VIleepp(-H (ot
The result follows from the tower property of expectation and the assumption that class conditional distribution is the same for all the clients. From the above proposition, we can see that the gradients V (w) only differ due to p which captures the data heterogeneity due to label imbalance. The proof is given Sec. 15 of the appendix.
Assumption 3.3. Class-wise gradients are weakly correlated and similar magnitude gig. < &g.gm. B.g. < gmgm and for c cf
The assumption on weakly correlated class-wise gradients intuitively implies that gradients of loss for a specific class cannot give any significant information on the gradients of the other class.
Proposition 3.4. When the class-conditional distribution across the clients is the same, and the As- sumption 3.5 holds then 3 a range of values for X such that whenever X 2 de we have dx < 0 and Ga(w, < Ga(w.O).
dGa
The proposition implics that there is a value of X 2 X such that the derivative of Ga w.r.t X is negative. The proof is given in Sec. [..15 of the appendix. This indicates that by appropriately selecting the value of we can make the Ga lower which in turn reduces the client drift. One of the key assumptions on the heterogeneity is the existence of the below quantity.
B²(A) = sup Ga(w, X) weRd
(14)
which leads to the following assumption
Assumption 3.5. + E. (VVA S B'A))IVS(W))
This is the bounded gradient dissimilarity assumption used in Li et al. 2020 In the following proposition, we show the existence of such that (²) < (²))), which means that with regularizer we can tightly bound the gradient dissimilarity compared to the case without the regularizer ie., (A = 0).
Proposition 3.6. Suppose the functions S& satisfy Assumplion 3.5 above then we have B² < B²(0).
Proof. From
4 have
B²()) sup Ga(w, WERd
(15)

---

Published in Transactions on Machine Learning Research (December (enemberr/0225 2022)
For fixed À per proposition 3.4 we have the following.
sup Ga(w, A)
sup Ga(w,
(16)
WERd
WERd
The above inequality 16 is true as proposition 3.4 guarantees that the value of Ga(w, X) < Ga(w, 0) for all w when À 2 >e- If inequality [6 is not true, one can find a w that contradicts the proposition 3.4 which is impossible. This means for some value of X 2 de we have (²) < (0) from Eq. 14 and Eq. 16
The key takeaway from the analysis is that by introducing the regularizer we can tightly bound the heterogeneity when compared to the case without the regularizer. Based on the works (Karimireddy et al. |0000- Li et al. 2020) we exploin that lower (X) implies better convergence, which is also supported by empirical evidence. These details are provided the Sec. A.10 10 of the Appendia.
3.4 Discussison on the Generalization of ASD
Table 1:
The table shows the impact of ASD on the algorithms on CIFAR-100 Dataset using the non-iid partition of 8 = 0.3. We consistently see that the top eigenvalue and the trace of the Hessian of the loss of the global model docrease and the accuracy improves when ASD used. This suggests that by using ASD can make global model reach flat minimum towards better generalization.
Algorithm
Top Eigenvalue J Trace
Accuracy 1
FodAvg
53.6
8516
38.67
FedAvg ASD
12.3
2269
42.77
FodDyn
49.4
6675
47.56
FedDyn + ASD
14.2
2241
49.03
FedSpeed
51.9
6937
47.39
FedSpeed ASD
14.6
2063
49.16
Donaity (Log scale
10-1
10
10:
15
20
30
Eiganvalue
Figure 3: Eigen spectrum with and without the ASD regularizer. It is evident that ASD regularizer not only minimizes the top cigenvalue but most of the cigenvalucs and attains the flatness.
The key reason for better generalization the adaptive self-distillation loss. It has been shown in (Mobahi et al. 2020) that self-distillation improves the generalization in centralized settings. It's been empirically shown in (Zhang et al. 2019) that self-distillation helps the model to converge to flat-minimum. Generally, converging to flat minima is indicative of improved generalization, concept explored in prior studies such as (Keskar et al. 2017) and Yao et al. 2020). The top eigenvalue and the trace of the Hessian computed from the training loss are typical measures of 'flatness' of the minimum to which the training converges, i.c., lower values of these measures indicate the presence of a flat minimum. To gain a deeper underslanding of this phenomenon a federated learning setting, we analyzed the top eigenvalue and the trace of the Hessian the cross-entropy loss for global models obtained with and without the ASD regularizer. The following argument establishes that if the client models converge to flat minima, it would also ensure convergence of the resultant global model to flat minimum. We assume the Hessians of the functions fs. (kth client's 's local objective),
8

---

Published in Transactions on Machine Learning Research (December/24) 2024)
f (resultant global objective) exist and are continuous almost everywhere. Since = K Σ fúr have H(f) = E= H(fi) (H(g) denotes the Hessian of function g). This implics μ (H(H))) S K - (H(f:)) (A) denoles top eigenvalue of matrix A). Thus when the local models converge to a flat minimum, it will ensure the convergence of the global model to flat minimum. Following the method of (Yao al, 2020), we computed the top eigenvalue and trace of the Hessian. In Table [ we observe that FedAvg+ASD attains lower values for the top cigenvalue and trace compared to FedAvg, suggesting convergence to flat minimum. The Eigen density plot in the figure 3 also confirms the same. We use the CIFAR-100 dataset. with non-iid data partitioning of 8 = 0.3 (refer to Sec. 4 In Table | we have presented our analysis when ASD is combined with FodAvg, FodDyn and FodSpeed. The results for other algorithms are presented in Sec. A.8 of appendix. similar concept has been explored in FedSAM (Qu et al. 2022 Caldarola et al., 2022;; the issuc with SAM-based methods is they require an extra forward and backward pass, which doubles the computational cost on the resource constrained edge devices. However, our method can be applied to SAM-based methods and further improve its performance. ASD consistently attains the flatness with the other FL algorithms and enhances their generalization.
4 Experiments
We perform the experiments on CIFAR-10, CIFAR-100 (Krizhevsky & Hinton 2009), Tiny- Imagelet (Le & Yang] 2015) datasets with different degrees of heterogeneity in the balanced settings (i.e., the same number of samples per client but the class label distribution of each varies). We set the total number of clients to 100 in all our experiments. We set the client participation rate to 0.1, i.e., 10 percent of clients are sampled on average per communication round, similar to the protocol followed in (Acar et al.) 2021). We build our experiments using publicly available codebase by (Acar et al.] 2021). For generating non-fid data, Dirichlet distribution is used. To simulate the effect of label imbalance, for every client we sample the 'probability distribution' over the classes from the aforementioned Dirichlet distribution pk Dir(ó, C). Every sample of pir is vector of length C and all the elements of this vector are non-negative and sum to 1. This vector represents the label distribution for the client. The parameter known the 'concentration parameter', captures the degree of label helerogeneily. Lower values of å capture high heterogeneity and as the value of increases, the label distribution becomes more uniform. Another parameter of Dirichlet distribution (i.e., C), its value can be interpreted from the training dataset (C = 100 for CIFAR-100). For notational convenience, we omit from Dōr C*) by simply re- re-writing as Dir(6). By configuring the concentration parameter to 0.6 and 0.3, we sample the data using the Dirichlet distribution across the labels for each client from moderate high heterogeneity by controlling ò. This is in line with the approach followed in Acar et al. 2021 and (Yurochkin et al. 2019
Fabley
43
Fedtya
Acounsy
10
106
206
200
460
530
166
206
203
489
$30
Commanication Round
Cammunication Round
(a) CIEAR-100 (6 0.3)
(b) Tiny-ImageNet (5 = 0.3)
Figure 4: Test Accuracy vs Communication rounds: Comparison of algorithms with 5 = 0.3 partitions on C1FAR-100 and Iiny-ImageNet datasets. All the algorithms augmented with proposed regularization (ASD) outperform compared to their original form. FodSpoed+ ASD outperforms all the other algorithms.
9

---

Published in Transactions on Machine Learning Research (December/ 2ocember/20225 2024)
Table 2: Comparison of Ac):racy(%): We show the accuracy attained by the algorithms across the datasets ManAA--o0//iin-mmaaaee al the end of 500 communication rounds. It can be seen that by combining the proposed approach the performance of all the algorithms can be significantly improved.
CIFAR-100
TinylmageNet
Algorithm
0.3
0.6
III
FedAvg (MeMahan et al.
2017
38.67
38.53
37.68
23.89
23.95
10.72
23.48
10.81
FedAvg ASD (Ours)
42.77
40.22
42.54
43.00
25.31
#0.25
26.38
20.21
26.67
=0.10
FedProx (Li al. 2020
37.79
10.99
37.92
10.55
37.94
24.61
11:24
23.57
10.44
23.27
20.11
FedProx- ASD (Ours)
41.31 +0.90
41.67
#0.12
42.30 +0.37
25.49
#0.45
25.62
#0.05
25.58
=0.18
FedNTD Lee et al.
2022
40.40 =1.52
40.50 #0.54
41.23
#0.44
23.71 $0.05
23.28
=0.29
22.95
#0.22
FedNTD + ASD (Ours)
43.01
43.61
43.
27.34
#0.73
27.39
40030
27.41
PodDyn Acar al
2021
47.56 10.41
48, 60
10.00
48,87
10.51
27
10.21
28.58
10.61
28.37
1:020
FedDyn (Ours)
10:24
50.23
10.25
51.44 10.48
29.94 10.67
30.05
1..24
30.76
1.0.4
FedSAM Caldarola et al.
2022
40.89 +0.30
41.41
40.81
24.72 40.04
25.42
=0.49
23.50
#0.94
FedSAM+ + (Ours)
43.99 20.14
44.54
44.77
26.26
26.80
10.17
25.37
=0.36
PodDisco Ye ct al.
2023
38.97 #1.38
38,87
+1.37
37.85
#0.57
24.35
+0.42
24.03
=0.78
23.49 13.49
#0.31
FodDisco ASD (Ours)
41.55 11.06
41.94
10.30
43.09
10.47
25.43
10.46
26.03
10.26
26.56
..0.
FedSpeed Sun et al.
2023
47.39 20.82
48:27
40.18
49.01
10.46
28.60
10.15
29.33
=0.3
29.62
4..31
FedSpeed ASD Ours)
49.16 20.40
49.76
51.99
30.97
30.05
10.24
32.68
=0.53
Falley
Fellosed
Accursry
40
20.0
29.9
20.0
106
128 Communication Round
106
X00
Communication Round
(a) CIFAR-100 (8 0.6)
(b) Tiny-ImageNet (ō = 0.6)
Figure 5: Test Accuracy vs Communication rounds: Comparison algorithms with ó = 0.6 data partitions on CIFAR-100 and Tiny-ImageNet dataset. All the algorithms augmented with proposed regularization (ASD) outperform compared their original form. FedSpeed+ ASD outperforms all the other algorithms.
Fadtye
Faddly
Fad
27.5
Accurary
Accurecy
0.90
150
209
106
489
Communication Round
Communication Round
(a) CIFAR-100 (iid)
(b) Imag-et (fid)
Figure 6: Test Accuracy vs Communication rounds: Comparison of algorithms with iid data partitions CIFAR-100 and Tiny-Image ImageNet datasets. All the algorithms augmented with proposed regularization (ASD)
outperform compared to their original form. FedSpeed+ ASD outperforms all the other algorithms.
10

---

Published in Transactions on Machine Learning Research (December/ 2iccebeer/2024 22024)
5 Results and Discussion
For ovaluation, report accuracy the test dataset as performance metric and the number of communication rounds required to attain the desired accuracy metric to quanlify the communicaatin cost. Specifically, we evaluate the global model on the test set and report its accuracy after every communication round. For comparison, we consider the popular methods for federated learning, such as FedAvg, FedProx, FodDyn, FedSpeed FedNTD, FedSAM and FedDisco. We augment each of these methods with our approach (ASD) and observe a significant boost performance. For fair comparison, we consider the same models used in Fodavg MMcMahan et al. 2017), and FodDyn (Acar ct al., 2021), for CIFAR-10 and CIFAR-100 classification tasks. The model architecture used for CIFAR-100 contains convolution layers followed by 3 fully connected layers. For Tiny- Netge,,,,,,, we use 3 convolution followed by 3 fully connected layers. The detailed architectures are given in See A.2 of the appendix. Hyperparameters. SGD algorithm with a learning rate of 0.1 and decay the learning rate per round of 0.998 is used to train the client models. Temperature T is set to 2.0. We only tune the hyper-parameter X. More hyperparameter setting details and impact of X, 7 are provided in Sec. [A33 and A.6 of the appendix, respectively. The impact of client participation rate and the number of clients on ASD are shown Sec. A.7 of the appendix. Implementation of ASD with other FL methods is discussed in Sec. A.12 of appendix. We compare the convergence of different. schernes for 500 communication rounds. Following the testing protocol of (Acar al. 2021), we average across all the client models and compute the test accuracy on the averaged model, which is reported in our results. In all the tables, we report the test accuracy of the global model in % at the end of 500 communication rounds. All the experiments in the tables are performed over three different initializations, mean and standard deviations of accuracy over the three experiments are reported. We also demonstrate the efficacy of our proposed method with deeper architectures such as ResNet-20 and Vision Transformer (ViT) models in Sec A.4 and See A.5 of the appendix respectively.
5.1 Performance of ASD on CIFAR-10/100 and Tiny-lmagenet
In Table 2 we report the performance of SIFAR-100 and Tiny-ImageNet datasets with various algorithms for non-iid (Dir(ó = 0.3) and Dir(ō = 0.6)) as well as the iid settings. Each experiment is performed over three different initializations, and the mean and standard deviation of the accuracy are reported. On CIFAR-100 we observe that our proposed ASD applied on FedDyn improves its performance by 1.45% for Dir(6 = 0.3) and a 1.6% (5 0.6). Similarly, for Tiny-ImageNet as 2.4% for Dir(6 0.3) and by R 1.4% for Dir(ō 0.6). The lest accuracy vs communication rounds plot on CIFAR-100 100 and Liny-ImageNet datasets is shown in Figures 4
observe that FedDyn+ASD improvos FodDyn by
across non-iid and iid partitions. We can see that adding ASD gives consistent improvement across the round We obtain significant improvements for FodAvg-ASt against FedAvg, FedProx+ASI against FedProx, FedSpeed+
5 6
against FedSpeed, etc. In Sec A.9 of the appendix we present the CIFAR-10 results where we observe that adding ASD consistently gives an improvement of 0.4% 1.99% improvement across the algorithms.
5.2 Comparison with adaptive vs uniform weights
Table 3: Comparison with adaptive weights vs uniform weights on CIFAR-100 dataset with Dirichlet 6 = 0.3
Distillation with Uniform weights
Distillation with Adaptive weights
Algorithu
FodAvg ASD
41.75 20.12
42.77
+0.22
FodN ASD
40.40 =1.52
43.01
+0.34
FedDisco
40.21 10.77
41.55
11.08
FedDyn+AAD
47.90 20.35
49.03
10.24
We analyze the impact of the proposed adaptive weighting scheme. We compare by making all the ai in
Eq 10 to ie, by giving equal weights to all the samples in the mini-batch. We can see from Table 3 that the
In the figures we only compare FedAvg, FedDyn and FedSpeed for better readability. others, please refer the appendix
Sec A.10 of
11

---

Published in Transactions on Machine Learning Research (December/200) 2022)
proposed adaptive weighting scheme yields much better performance than assigning uniform weights, thus establishing the impact of proposed adaptive weights.
5.3 Performance with increased clients and lower client participation
Table 4: Experiments with 500 clients
Method
Accuracy (in %)
FedAvg
27.92
FedAvg+AASD (ours)
31.28
TexlProx
28.09
ASDD
32.13
FodNTD
30,99
FedNTD+ASD
33.65
FodDyn
31.0
FedDyn=AASD (ours)
33.12
FedSperd
34.08
FedSpeed+ ASD (ours)
36.59
In this section, analyze the impact of our ASD regularizer to mimic the cross-device setting. We increase the client participation to 500 clients and only 1% of the clients participate in every round. We consider the CIFAR-100 dataset and the non-iid data partition of = 0.3. In the Table 4 we observe that ASD consistently improves the performance of the algorithms. The accuracies are reported after averaging over three different initializations at the end of 1000 communication rounds. Even in this challenging setting ASD consistently improves the performance of the FL algorithms.
6 Computation Cost
The major computation for the distillation scheme comes from the teacher forward pass, student forward pass, and the student backward pass (Xu et al. 2020). We assume C₂ the total computational cost of server model forward pass and Ck be the total computation cost of client model k the forward pass per epoch. We do not need C. computations every epoch, we only need to compute once and store the values of H(z) while keeping the same backward computation. Specifically, for the computation distillation regularizer we only need Cs + C. local computations compared to E C₁ computations without regularizer. Here denotes the local epochs of the client. Since C, Ck, we have (E + 1] C local computations. Thus, our regularizer introduces minimal forward computation on the edge devices, which typically have low computation. In Sec. A.14 of appendix discuss the computation vs accuracy of ASD.
7 Conclusion
In this work, we presented an efficient and effective method for addressing client data heterogeneity due to label imbalance in federated learning using our proposed Adaptive Self-Distillation (ASD), which does not require any auxiliary data and no extra communccttion cost. We also theoretically showed that ASD has lower client-drift leading to better convergence. Moreover, we performed analysis to show that ASD has better generalization by analyzing the top eigenvalue and trace of the Hessian of the global model's loss. The effectiveness of approach is shown via extensive experiments across datasets such CIFAR-10, CIFAR-100 and Tiny-I ImageNet with different degrees of heterogeneity. Our proposed regularizer (ASD) be integrated casily atop any of the FL frameworks. We evaluated this efficacy by showing improvement in the performance when combined with FedAvg, FedProx, FedDyn, FedSAM, FedDisco, FedNTD and FedSpeed. We have also shown that the computation required to implement ASD is simply an additional forward pass on the client-side training, i.c, all the gains we obtain with ASD requires minimal compute. Our research can inspire the designing of the computationally efficient regularizers that concurrently reduce client-drift and improve the generalization.
12

---

Published in Transactions on Machine Learning Research (December/ (ecember/2004)
References
Durmus Alp Emre Acar, Yue Zhao, Ramon Matas Navarro, Matthew Mattina, Paul N Whatmough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. arXiv preprint or arXiv:211100490 2021.
Debora Caldarola, Barbara Caputo, and Marco Ciecone. Improving generalization in federated learning by secking flat minima. In European Conference on Computer Vision, pp. 654-672. Springer, 2022.
Jonas Geiping, Hartmut Bauermeister, Hannah Dröge, and Michael Moeller. Inverting gradients easy is break privacy in federated learning? Advances in Neural Information Processing Systems, 33: 16937 16947, 2020.
Andrew Hard, Chloé M Kiddon, Daniel Ramage, Francoise Beaufays, Hubert Eichner, Kanishka Rao, Rajiv Mathews, and Sean Augenstein. Federated learning for mobile keyboard prediction, 2018. URL https://arxxv..rrg/bbb////
03604
Yuting He, Yiqiang Chen, XiaoDong Yang, Hanchao Yu, Yi-Hua Huang, and Yang Gu. Learning critically: Selective self-distillation in federated learning on non-iid data. IEEE Transactions on Big Data, pp. 1-12, 2022a. doi: 10.1109/TTDB 2022AA..022.311110 2022.3189703
Yuting He, Yiqiang Chen, Xiaodong Yang, Yingwei Zhang, and Bixiao Zeng. Class-wise adaptive self distillation for heterogeneous federated learning. 2022b.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in neural network. arXiv preprint 1rXiv:1503.02530 2(7), 2015.
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Mcasuring the effects of non-identical data distribution for federated visual classification. arXiv preprint orXit:1909.06330 2019.
Tau-Ming Harry Hsu, Hang Qi, and Matthew Brown. Federated visual classification with real-world data distribution. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part X 16, pp. 76-92. Springer, 2020.
Yangsibo Huang, Samyak Gupta, Zhao Song, Kai Li, and Sanjeev Arora. Evaluating gradient inversion attacks and defenses in federated learning. Advances in Neural Information Processing Systems, 34:7232-722 7241, 2021.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mchdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and Trendso in Machine Learning, 14(1-2):1-21) 2021.
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for foderated learning. In Inter- national Conference on Machine Learning, pp. 1822-143.. PMLR, 2020.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference on Learning Representations, 2017. URL https:://pperrevier.. http://wwwo..ooooooooooooooooooo
Gecho Kim, Jinkyu Kim, and Bohyung Han. Communication-effcieett federated learning with accelerated client gradient. Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 12385 12394, 2024.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Canadian Institute for Advanced Research, 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
13

---

Published in Transactions on Machine Learning Research (December/2000 22024)
Gihun Lee, Minchan Jeong, Yongjin Shin, Sangmin Bae, and Se- Young Yun. Preservation of the global knowledge by not-true distillation in federated learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https:
//openrevies met/forumidd qu332bbb1J
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Amoet Talwalkar, and Virginia Smith. Federated oplimization in heterogeneous networks. Proceedings of Machine Learning and Systems, 2:429-44
2:429-450, 450, 2020.
Tao Lin, Lingjng Kong, Scbastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. Advances in Neural Informalion Processing Systems, 33:2351-2300 2020.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Arcas. Communication- efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp. 1273-1282. PMLR, 2017.
Konstantin Mishchenko, Eduard Gorbunov, Martin Takáë, and Peter Richtárik. Distributed learning with compressed gradient differences. arXiv preprint arXiv:1901.0990 09269, 2019.
Hossein Mobahi, Mchrdad Farajtabar, and Peter Bartlett. Sclf-distillation amplifics regularization in hilbert space. Advances in Neural Information Processing Systems, 33:3351-3361, 2020.
Zhe Qu, Xingyu Li, Rui Duan, Yao Liu, Bo Tang, and Zhuo Lu. Generalized federated learning via sharpness aware minimization. In International Conference on Machine Learning, pp. 19250-18286 PMLR, 2022.
Swaroop Ramaswamy, Rajiv Mathews, Kanishka Rao, and Françoise Beaufays. Federated learning for emoji prediction in a mobile keyboard. Xr preprint arXiu: 04399, 2019.
Sashank J Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and Hugh Brendan McMahan. Adaptive federated optimization. In International Conference on Learning Representations.
Nicola Rieke, Jonny Hancox, Wenqi Li, Fausto Milletari, Holger R Roth, Shadi Albarqouni, Spyridon Bakas, Mathieu N Galtier, Bennett A Landman, Klaus Maier- Hein, et al. The future of digital health with federated learning. NPJ digital medicine, 3(1):1-7, 2020.
Hyowoon Seo, Jihong Park, Seungeun Oh, Mehdi Bennis, and Seong-Lyun Kim. 16 federated knowledge distillation. Machine Learning and Wireless Communications, pp. 457, 2022.
Sebastian U Stich. Local sgd converges fast and communicates little. arXiv preprint ar. 2018.
Yan Sun, Li Shen, Tiansheng Huang, Liang Ding, and Dacheng Tao. Fedspeed: Larger local interval, less communication round, and higher generalization accuracy. In The Eleventh International Conference Learning Representations, 2023. URL https::/ wwwpaarwwiwwwweeeeeeeeeeeemeeeee ntt/ forummorumm (dd
Shitao Tang, Litong Feng, Wenqi Shao, Zhanghui Kuang, Wei Zhang, and Yimin Chen. Learning efficient. detector with semi-supervised adaptive distillation. arXio preprint srXiw:1901.00300 2019.
Jianyu Wang, Vinayak Tantia, Nicolas Ballas, and Michael Rabbat. Slowmo: Improving commuicationn efficient distributed sgd with slow momentum. In International Conference on Learning Representations.
Jianyu Wang, Zachary Charles, Zheng Xu, Gauri Joshi, H Brendan McMahan, Maruan Al-Shedivat, Galen Andrew, Salman Avestimehr, Katharine Daly, Deepesh Data, et al. A field guide to federated optimization. Xix preprint arXiv:2107.00 06917, 2021.
Kan Wu, Jinnian Zhang, Houwen Peng, Mengchen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan. Tinyvit: Fast pretraining distillation for small vision transformers. In European conference computer vision, pp. 68 85. Springer, 2022.
Guodong Xu, Ziwei Liu, and Chen Change Loy. Computation-effiient knowledge distillation via uncertainty- aware mixup. Xiv preprint or Xrx:2000 209411,,,, 2020.
14

---

Published in Transactions on Machine Learning Research (December/ 2ocember/2022/ 2024)
Jie Xu, Benjamin S Glicksberg, Chang Su, Peter Walker, Jiang Bian, and Fei Wang. Federated learning for healthcare informatics. Journal of Healthoure Informatics Research, 5:1-19, 2021.
Sarika Yadav and Rama Shankar Yadav. A review on energy efficient protocols in wireless sensor networks. Wireless Networks, 22(1):335-335 350, 2016.
Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W Mahoney. Pyhessian: Neural networks through the lens of the hessian. In 2020 IEEE international conference on big data (Big data), pp. 581-590. IEEE, 2020.
Rui Ye, Mingkai Xu, Jianyu Wang, Chenxin Xu, Siheng Chen, and Yanfeng Wang. Feddisco: Federated learning with discrepancy-aware collaboration. 2023.
Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, and Yasaman Khazaeni. Bayesian nonparametric federated learning of neural networks. In International Conference on Machine Learning, pp. 7252-7261. PMLR, 2019.
Zelin Zang, Siyuan Li, Di Wu, Ge Wang, Kai Wang, Lei Shang, Baigui Sun, Hao Li, and Stan Z Li. Dlme: Deep local-flatness manifold embodding. In European Conference on Computer Vision, pp. 576-592. Springer, 2022.
Jie Zhang, Zhiqi Li, Bo Li, Jianghe Xu, Shuang Wu, Shouhong Ding, and Chao Wu. Federated learning with label distribution skew via logits calibration. In International Conference on Machine Learning, pp- 26311 1-3299. PMLR, 2022.
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE/CVE international conference computer vision, pp. 3713 3722, 2019.
Yanlin Zhou, George Pu, Xiyao Ma, Xiaolin Li, and Dapeng Wu. Distilled one-shot federated learning. arXiv preprint XrX:::2009. 07999, 2020.
Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou. Data-free knowledge distillation for heterogencous fodcrated learning. In International Conference on Machine Learning, pp. 12878 12889. PMLR, 2021.
A Appendix
A.1 Notations and Definitions
H(x) denotes the entropy of the model under consideration for input æ.
p denotes the probability that the client has the input i belonging to class
H(g) denotes the Hessian of the function g-
(A)
denotes the top eigenvalue of matrix A.
Del denotes the KT. divergence.
inf denotes the infimum and sup denotes the supremum.
is used for denoting the heterogeneity generated based on Dirichlet distribution.
À denotes the ASD regularizer strength.
Ga(w, X) denotes the gradient dissimilarity.
B(.) denotes the expectation.
Ta (w) is the loss of client (cross-entropy loss).
15

---

Published in Transactions on Machine Learning Research (December/ (iccember/022g 2024)
L* (
ASD is the Adaptive Self-Distillation loss for client k.
D represents the dataset of client k.
9)) represents the data distribution of the client k.
A.2 Model Architectures
In Table the model architecture is shown. We use PyTTorch style representation. For example conv layer(3,64.)) means 3 input channels, 64 output channels and the kernel size 5. (2xpool(2,2) represents the kernel size of 2 and stride of 2. CollyConncctel(3842222
represents input dimension of 384 and outpul dimension of 200. The architecture for CIFAR-100 is exactly the same as used in (Acar el al. 2021
Table 5: Models used for Tiny-Imaged and CIFAR-100 datasets.
"iny-ImageNet Model
ConvLayer 3364,33
GroupNorm(4.64
CIFAR-10/100 Model
Relu
MaxtPool
ConyLayer (61,61133
GroupNormt 4,64)
(onrLayerr3.64.5
Relu
Rein
2
MaxPool12.2)
ConvLayer (64,64.3)
(B4,Layer(44,61.5
GroupNorm(4.00
Relu
Relu
MaxPooo((222]
MaxPPool/222)
Flatien
Flasten
FullyCommeetoc(100...55 Relu
FullyConnoctedl( (096,,5122
Reiu
(all(Comeette((355.222)
TallsConneeted(1112..35
Relu
FullyConnecedd 192.100)
Culyyoonnetedd3811.0
A.3 Hyper-Parameter Settings
The value of À is specified in units of batch-size B. We chose À from {10, 20, 30). We set A 20 for all the Tine-ImageNeet experiments. For CIFAR-10/100 we chose X to be 10 and 30 respectively. The batch-size (B) of 50 and learning rate of 0.1 with decay 0.998 is employed for all the experiments unless specified. All the experiments are carried out with 100 clients and with 10% client participation.
A.4 Experiments with Deeper Models (CNN's)
In this section, perform experiments with the deep models such as ResNet-20 CIFAR-100 dataset with Dirichlet 5 = 0.3. For this experiment we have used 300 communication rounds, the number of clients as 30, and the client participation rate is set to 20%. In the table G we report the numbers averaged over 3 different trials. We observe that the addition of our proposed regularizer ASD atop mutiple popular FI. methods leads to consistent improvements, thereby further justifying the efficacy of our proposed method.
Table 6: Experiments ResNet-20
Method
Accuracy (in %)
FedAvg
46.35
FedAvg-1 ASD (ours)
47.90
FedDyn
53.60
FedDyn- (ours)
55.15
FedSpeed
54.42
FedSpeed + ASD (ours)
55.82
16

---

Published in Transactions on Machine Learning Research (December/ 2ocember/22024 22024)
A.5 Experiments with Deeper Models (ViT)
We perform experiments with ViT architecture using the Tiny- ViT (Wu al 2022) as client models on ImageNet-1000 dataset (Zang al. 2022) with non-iid data partitioning of Dirichlet 0.3. The choice of Tiny-Vil VIT is motivated by the fact that edge devices are traditionally computational resource-constramned and Tiny-ViT is designed for such applications. For this experiment, the number of clients is set to 200, and the client participation rate is set to 5%. We have used 300 communication rounds. In the Table 7 we report the numbers averaged over different trials. We observe that the addition of our proposed regularizer ASD atop FedAvg and FedDyn leads to consistent improvements, thereby further justifying the efficacy of our proposed method on the deeper architectures.
Table Experiments using Tiny-Vis VIT on ImageNet- 100 dataset with the non-iid partitioning of d = 0.3
Method
Accuracy (in %)
FedAvg
18.12
FodArg ASD (ours)
22.10
FodDyn
28 28.02
+oDD (ours)
36.70
In Table we performed an experiment on ViT-Small architecture on CIFAR-100. We observe that adding our ASD regularizer improves the baseline FedAvg by 1.4% and 1.7% for 0.3 and å = 0.6, respectively. In this setup we consider 100 clients with 10% participation and the accuracy is reported at the end of 300 rounds.
Table 8: Experiments using ViT-Small with CIFAR-100 with the non-iid data partitioning of 0.3 and 8 0.6.
Accuracy (%)
Method
8 0.3
8 0.6
FedAvg
53.22
52.63
FedAvg ASD (Ous)
54.67
54.34
A.6 Impact on the choice of hyperparameters and т
A=
20
30
4-40
A-60
Temperature
Figure 7: Impact of and T on CIFAR-100 dataset with non-iid partitioning of 8 = 0.3 with FedAvg+ ASD.
We study the impact of changing the hyper-parameters À and 7 on the CIFAR-100 datasct with the Dirichlet non-iid partition of 5 = 0.3. We report the accuracy at the end of 500 rounds. When using FedAvg+ASD algorithm. In Figure 7 we see that the accuracy the model increases with À and then slightly drops after a critical point. This is expected as too less value of similar to FodAvg and very high value of À will ignore the local learning. can also be seen that for all the values of À the Accuracy peaks experiments set the temperature parameter set to 2.0.
In all of our
17

---

Published in Transactions on Machine Learning Research (December/ 2occeber/22024 22224)
A.7 Impact of client participation number of clients on ASD
We fix the client participation 2% and vary the number of clients from 100 500. We perform this ablation using the CIFAR-100 dataset with a non-iid Dirichlet data partitioning of å 0.3. We summarize our observations in the Tables 9 and 10 below. It can be seen that ASD improves the performance of the baselines FedAvg and Fn in all the settings. In particular, we would like to highlight the point here that despite increasing the number of clients, the total number of training data samples across all the clients remains constant (for CIFAR-100). Thus as the number clients increases, the number of data samples per client decreases. This further aggravates the adverse impact of label heterogeneity across clients, and hence accuracy degrades in general. However, we are happy observe and report that, even under such a challenging setup, our proposed adaptive self-distillttion-baeedd strategy consistently improves the accuracy when combined on top of the existing baseline algorithms.
Table 9: Impact of increasing the number of clients on the Accuracy (%) when the participation rate is fixed to 2% and non-iid partitioning of å 0.3.
Number of Clients
Method
100
200
300
400
500
FedAvg
31.15
31.86
30.05
28.70
26.12
FodAvg+ASD (ours)
37.67
35.56
32.86
29.98
27.16
FedDyn
39.17
36.11
34.24
31.09
26.87
FedDyu ASD (ours)
39.34
40.08
36.83
33.61
28.05
Table 10: Impact of increasing the number of clients the Accuracy (%) when the participation rate is fixed to 2% and non-iid partitioning of 5 = 0.6.
Number of Clients
Method
200
300
400
500
FedAvg
35.17
32.06
30.12
28.31
25.71
FedAvg+ASh (ours)
39.61
35.49
32.16
29.43
27.61
FedDyn
37.96
36.56
34.71
30.37
26.45
FedDyn+ ASD (ours)
39.40
40.49
37.48
33.23
28.60
In Table 11 unlike the previous ablation, here we fix the number clients to 100 and vary the client participation rate from 5%, 10% and 15%. We consider the CIFAR-100 dataset with non-iid partitioning of (6 = 0.3). As expected, the accuracy of the FL-trained models improve with an increase in the client participation rate. We would also like highlight here that, by adding our proposed ASD strategy consistently improves the accuracy when combined top of the existing bascline algorithms such as FedAvg and FedDyn.
Table 11: Impact of increasing the client participation rate on the Accuracy (%) with number of clients fixed to 100.
nou iid partition (6 0.3)
non-iid partition (ō
0.6)
Method
client paticipation
client participation
5%
10%
15%
5%
10%
15%
FodAvg
38.22 22
38.67
38.85
39.04
38.53
38.00
Fed ASD (Ours)
43.04
42.77
43.59
43.51
42.54
42.90
FedDyn
44.68
47.56
47.87
45.18
48.60
48.74
PedDyn+ (Ours)
47.51
49.08
50.32
47.81
50.23
51.48
A.8 Hessian Analysis
In the Table 12 we analyze the top eigenvalue and the trace of the Hessian of the global model when ASD is applied to methods such FFdProx, FedNTD, FedSAM and FedDisco.
18

---

Published in Transactions on Machine Learning Research (December/ 2ocember/22024 2224)
The table shows the impact of ASD on the algorithms on CIFAR-100 Dataset. We consistently see that the top eigenvalue and the trace of the Hessian decrease and the Accuracy improves when ASD is used. This suggests that using ASD makes the global model reach to flat minimum for better generalization.
Table 12:
Algorithm
Top Eigenvalue
Trace 4
Accuracy
FedProx
45.2
8683
37.79
FedProx + ASD
41.31
FedNTD
16.3
3517
40.40
FedNTD + ASD
17.5
2840
43.01
FedSAM
19.04
4022
40,89
FedSAM ASD
43.99
FesDisco
46.7
8771
38.97
FedDisco ASD
12.2
2334
41.55
A.9 Performance on CIFAR-10 dataset
In Table 13 we show the results for the CIFAR-10 dataset, we find that applying the ASD improves the performance of all the algorithms consistently.
Table 13:
We show the accuracy attained by the algorithms CIFAR-10 at the end of 500 communication rounds. It can be seen that by combining the proposed approach the performance of all the algorithms is improved.
Algorithm
8= 0.3
FodAvg
78.15 10.78
78.66
10.10
80.99
10.09
FedAwg+AAD (Ours)
79.01 10.33
79.93
#021
81.83
10.19
FedPrux
78.25 10.68
78.81
10.69
81.04
1:0.44
FedProx+ASD (Ours)
78.77 #0.49
79.91
=0.12
81.74
#0.06
FedNTD
76.79 +0.37
78.55
+0.31
80.98
+0.21
FedNID+ASD (Ours)
78.78 10.86
80.13
20.49
81.80
#0.11
FedDyn
81.08 10.52
81.48
1.0.35
83.51
10.27
FodDyn ASD (Ours)
81.82 10.56
82.33
10.39
84.09
10.15
FedDisno
78.21 +0.45
78.76
+0.32
81.04
+0.30
ASD (Ours)
78.97 +0.01
79.98
+0.35
81.71
+0.21
FedSpeed
81.28 #0.32
81.83
0.336
83.67
#0.14
FodSpeed ASD (Ours)
81.70 10.20
82.62
10.26
84.57
10.24
A.10 Accuracy vs Communication rounds
In the below figures g and 10 we present how the accuracy is evolving across the communication rounds for the FL methods FedNTD, FedProx, FedDisco with and without the ASD regularizer. We present these results for non-iid (8 0.3 and d 0.6) and with the iid data partitions for both the CIFAR-100 and Tiny-ImageNet datascts. It can be seen that adding ASD to these off-tho-shelf FL methods consistently improves the performance.
A.11 Privacy of Proposed Method
In our method, which is ASD regularizer, the adaptive weights are computed by the client without depending on the server and it does not assume access to any auxiliary data at the server as assumed methods such as FodCAD (He al. (022b) and FodDF (Lin al.) (020). In our method, only model paramcters are communicated with the server similar to FedArg (MeMahan al. (017). Thus our privacy is similar to the Avg method at the same time obtaining significant improvements in the performance.
19

---

Published in Transactions on Machine Learning Research (December (anembber/2225 2022)
Pariber
AcCUaCy
Communication
Communicatio Round
(a) CIFAR-100 0.3)
(b) Tmny-ImageNet (6 0.3)
Figure 8: Test Accuracy Communication rounds: Comparison of algorithms with 6 = 0.3, data partition CIFAR-100 and Tiny-ImageNee datasets. All the algorithms augmented with proposed regularization (ASD) outperform compared to their original form.
20.2
Accura
Accurg
380
180
Communication
Communicatin Round
(8) CIFAR-100 0.6)
(b) Liny-ImageNet 0.6)
Figure 9: Test Accuracy Commuicationn rounds: Comparison of algorithms with 5 0.6 data partition CIFAR-100 and Tiny-ImageNet datasets. All the algorithms augmented with proposed regularization (ASD) outperform compared to their original form.
Fack 1V
28.
Accunac
Accuracc
134
communicaton Round
C.mmunicttonRuus
(a) CIFAR-100 (lid)
(b) Tiny-ImageNe (iid)
Figure 10: Test Accuracy vs Communication rounds: Comparison of algorithms with iid data partitions on CIFAR-100 and TmageImageNet datasets. All the algorithms augmented with proposed regularization (ASD) outperform compared to their original form.
20

---

Published in Transactions on Machine Learning Research (December/ (iccebber/2024 22224)
A.12 Implementation of ASD with the FL Methods
We now present the integration of ASD loss with the existing FL methods. For all the methods FedAvg, FedDyn, FedSpeed, FedProx, FedDisco and FedSAM, we augment the client loss of each of these methods with our proposed ASD loss in the Eq FodNTD (Lec et al. 2022) uses the non-true distillation loss, it distills the knowledge only from the non-true classes.
DNID(ag(=')llaattt
E (ogr""oo(((("""")(( eay
(17)
The above equation represents the FedNTD loss on the sample i, when the true class label is y. We now use the adaptive weights defined in Eq. 20 to update the FedNTD loss as below.
L
Load-nid nld (w)
aad
Σ xE[
(="))l
(18)
So the final loss used for optimizing FedNTD with adaptive self-distillation is given below.
fx(w) [a(w)= + ALgad-ntd ntad (w)
(19)
where (x(w) is definod as in Eq. of the main paper.
A.13 On the choice of KL divergence
The distillation loss introduced by the seminal work of (Hinton et al. 2015 matches the temperature-raised softmax values between the pre-trained teacher model and student model for effective knowledge transfer. It essentially cross entropy between two softmax vectors. KT, divergence differs from cross entropy by a constant and hence achieves the same optimization objective. In our context, treat the server model the teacher model and the client model the student model. Other divergence measures such reverse KL and JS can also be considered, but did not any significant performance difference empirically. In fact KL divergence performed better compared to reverso-KL and JS divergence as shown in Table below. For this experiment we used the CIFAR-100 dataset with 100 clients and 10% client participation rate.
Table 14: Comparison of Statistical Divergences
Method KL [ours]
Accuracy (in %)
42.77
reverso-Kl. KL
42.04
Jensen: Soasen-Shannon Shannon
42.21
Accuracy
22.5
1580
2003
Computation
Figure 11: Comparison of the Communication vs Computation for FedAvg and ASdA It can be
seen that over the communication rounds for given amount of computation, FedAvg+ASD attains better accuracy compared to FedAvg.
21

---

Published in Transactions on Machine Learning Research (December/ 2occeberr22024
A.14 Computation vs Accuracy
In the Figure 1 we have compared the compulation with the accuracy for FedAvg and FedAvg+ASD methods. In particular, we observe that at fixed cumulative computation cost of 2500 units FedAvg attains 38.67 % Accuracy while FodAvg+ASD attains 42.3% accuracy. Here the onc unit denotes the computation required for the single forward pass.
A.15 Proofs of Propositions:
We rewrite the adaptive weighting equations for convenience as below.
O EiEB oi
0
(20)
and di' is defined as below Eq. 21
exp( (HH(x²)
6
(21)
Pk
Proposition A.1. inf WERU Ga(w, 2) 1,
Proof.
1 Ex V
G. = K
(22)
|VS|| = 11 R EVEI V
(23)
We observe that the function |1-112 is Convex. By applying Jensen's inequality, we get the desired result. The expectation is taken over the discrete probability measure.
1 K
(24)
Lemma A.2. For any function of the form ((x)
az bar ax tor
satisfying Cn > ca , 3 xe 2 such that
d((m) da
<0Vx2 Ze
Proof.
dC(x)
(2ax + b)xz + be + ca) (2ac + (ax² + bx + cn)
(25)
dr
(ax² bx + ca)2
By re-arranging and simplifying the above we get the following
d((x) 2x(aca aca) b(ca - cn)
(26)
dr
(aax² bx c2)2
We interested in knowing when the numerator negative.
x2a(ca -c)) S b(cn, ca)
(27)
Since Cn Cd, have
b
x2a(cn - ca) 2 -b(cn - ca) - x ス
(28)
2a
22

---

Published in Transactions on Machine Learning Research (December/ 2aannnberr2222// 2024)
assuming 2e | 2m
We have the desired condition for a 2 Zec This conchudes the proof.
Proposition A.3. When the class conditional distribulion across the clients is identical, i.e, Px(æ
v) P(xx | y) then Vf((w) E. Pg. >n(gg), where ge VE[[(w;,,)) y = c), and ge
where v P
Proof. We re-write the equations for fx (w) from See 3.3 of main paper, LA (w) and A (w) from the Sec 3.2 of main paper for convenience.
Sx(w) In(w) + ALA
(29)
Lx(w)= E
aNEED [la (xx,
(30)
47 LSD (w) A Blan((a (2, D)t gu (x))
(31)
By applying the tower property of expectation, we expand Eq.
30 as below
La(w)= = Σ pi Elle = c]
(32)
If assume the class-conditional distribution across the clients to be identical the value of (.4,W;; is same for all the clients. Under such assumptions, drop the client index & and rewrite the Eq. 32 follows
[x(w)= Σ
Pk
(33)
(L((W) Σ Pf VE[[(w; c)
(34)
We further simplify the notation by denoting ge = vB(((w;,,y) yy= cl.
VL+(w) Píg.
(35)
To make the analysis tractable, Eq. 21
we use the un-normalized weighting scheme as the constant can be
absorbed into we can re-write Eq. 21 as below
a 2h ex)()(x:))
(36)
where 2/6 =4 With the above assumptions we can interpret the Eq. 31 as follows. Px
L
E
Id (w; (I.z))
(37)
where Idia((w; (I, w)) = 2i DRt (go (x) (x)))
By following the similar line of arguments from Eq. 32 to Eq. 34 we can write the following
V L ASD () = Σ P
(38)
23

---

Published in Transactions on Machine Learning Research (December/ (accember/202// 2024)
Vj&(w)= ΣEdle Pg + +x(
(39)
Lemma A.4. If Cn = K > k= 2
Cd = Emi=1 E12-1 -e=1 (PhaPh2).
where ph 2 Ak, c, and
Ec = 1, then od 2 1.
Proof. We need to show that
Σ K
2 (ph)²
k=
(40)
L
k1=1 2k2=1 2c=1 (p8 p12)
R
K
By rewriting the denominator we get
Σ K
(pf)² 1
k=
(41)
K E (2)))²
K
Consider rewriting the denominator of the L.H.S of above equation.
K
p()) Σ (p))T
(42)
where pd pp and is the all one vector of size K
Applying the Cauchy Schwartz incquality to the R.H.S of the Eq.
42 we get the following.
C KK
1p)
ΣΣ
(43)
By combining the Eq. 42 and Eq. 43 the result follows.
Proposition A.5. When the class-condittonal distribution. across the clients is the same, and the As- sumption 3.9 holds then a range of values for X such that whenever X 2 do we have dGa Ga(w.A) Ga(w,0)
dx < and
Proof.
1 G. K Ek /
(44)
|V
From Sec 3.2 of the main paper we have the following, We drop the argument w for the functions fa simplify the notation
C (g + Xre
V
Σ
(45)
24

---

Published in Transactions on Machine Learning Research (December/ (enemberr2022// 22224)
C
|
Σ Σ
Pk Pk (B.1 + A (B + Ank Bx2) Be2)
c2
cl=1 221
C
22
1 go2 + X Be Bo2 + x el
Be2 + 23 c ( 2
Pk Pk
(46)
cl 22
C
(pt)
ge) +
Σ
Pk Bml Bee + x Σ Σ
el
Pk
Brl Be2 + Xa E (
c=112=1
d=1221
c=1
C
C
Σ
(PE) + 2A
Σ c1=1e2=1
Σ
Pk 8. + X²C
el
In the above equation the second equality obtained by simply expanding the product, the third approximation by weakly correlated assumption of the gradients. The last two equalities used the fact that 26 1. We also assume that gradients are normalized to unit magnitude. Finally, we have the following
pk
K
C
K
C
(pE)
22
Σ
Σ
dl
8 82
T
X²KO)
(47)
K
K
k=1d=122-1
1
(and2 + bn +c)
K2
where
an K²C
(48)
K 0 c
bn = 2K
Pk g 82
4=
K
On = K
(pi)²
(49)
k= 0=1
K C
1 14 (1)
(g. 2ngg.)))
K
Σ
Pki (gl + AAA +
K
k1=1 22- 62ml
C
ΣΣ
Σ (g.g.
K
k1=112-1
Σ
( M + Pir
(A p R.
7
cl=1e2=1
C
K
(PIPPE
Σ
PRi
Vig 2
Σ
g. + +XRKC
c2=
c2=1
K
C
2AK
ΣΣΣ
Σ
Pk
8 +3²22)
K2
kld=112
(aut2 + +ba + cc)
k2
(50)
25

---

Published in Transactions on Machine Learning Research (December/ (enember/22024 2024)
By defining
Po a K²C
(51)
ba 2K
Σ
Σ
Pk gl
gal
x=1el=1 c2=1
K K
Cd =
Σ
Σ
(hiPk2)
(52)
k1=1
By substituting Eq.
and Eq. 50
Eq.
we get
Ga(w, A) an + bnn + Cn.
(53)
add2 + bad + Cd
Comparing Eq.
48 and Eq. 51
we see that a = On = Od, bn = ba.
Also On Cd assuming pf is non-degenerate.
Using the Lemma A.2 on Eq. 53 we get the value of A such that Ga(w, A) is reduced.
We also gel
by analyzing the values of for which Ga(w, A) Ga(w, 0) holds.
Thus choosing the A A supp mar(s. 1-)) guarantecs Ga(w, A) < Ga(w, 0), for all
This concludes the proof.
A.16 Discussion Impact of Gradient Dissimilarity the Convergence
We now study how the gradient diversity impacts the convergence of the FL algorithms such as FedProx and FedAvg. We omit the dependence of X on B. (for these algorithms so B is nothing but B(0) in notation) We have the gradient dissimilarity assumption below
Assumption A. K Ex (VS) & B²]
A.16.1 FedProx
Suppose the functions fe are lipschiltz smooth and their exists T__ such that Hʃx 2 I.I. With ft 0.
where a is FedProx regularization. If fú satisfies the assumption A.6 then acccording to Theorem 6 of Li
et al. 2020) the FedProx, after T = O A)) We have the gradient contraction as 1Σ [w] S The value of p is given below.
1
B
B(1 B(1+7)((2)
L(I B2
L(1+7) KJ²
p
(2v2K + >
(54)
P
1
Ap
2n²
for some 7 > and A f(w") f(w*), f(w" is the local minimum.
It can be seen that the convergence inversely related to p. High value of p leads to faster convergence. From Eq. 54 we can see that p can be increased by decreasing the value of B. Thus reducing the value of B helps in better convergence.
26

---

Published in Transactions on Machine Learning Research (December (iccember/202// 2202))
A.16.2 FedAvg
Assumption A.7. We now analyze the convergence of FedAvg, we consider the following assumptions
[V/((x) - [x)x = X||x-XX (3 smoothness)
Assumption A.8. Gradients have bounded Variance.
Suppose that f(w) and [ww,, satisfies Assumptions A.6
A.7 and
A.8
Let w = arg f(w) the w
local step-size be og- The theorem V in (Karimireddy al. 2020 shows that FedAvg algorithm will have w') and for constant M, then in R rounds, the
contracting gradients. If Initial model is w", F == f(w")
model " satisfies E(IVf(( 1| & O BMVF BB²P
RRS
R
We see the convergence rate O (BMM
We can see that convergence has direct dependence on B², This is the only term that is linked to heterogeneity assumption. So the lower value of B implies faster convergence. This motivates to have tighter bound on heterogencity. ASD achieves this by introducing the regularizer and choosing the appropriate value of A. In the figure 12 we empirically we verify the impact of ASD on the convergence. We plot the smoothed estimates of the norm of the difference of the global model parameters between the successive communication rounds ie ||w - w!
FB²F).
VRLS
Secive
Fedtrrx
12
toskkg ASD
1.2
Pedtrockas lsd
1.0
0.4
0.4
02
02
0
200
400
900
802
1000
1200
1402
200
430
6.0
500
1000
1200
1400
commuricaioon
(a) FedAvg
(b) FedProx
Figure 12: Impact of ASD on the convergence CIFAR-100 dataset with non-iid partition of d = 0.3
27


Reviews:
[{"review_id": "865R7OKPdv", "reviewer": "Reviewer_J9kY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n+ The paper presents a simple, intuitive, and easy-to-understand algorithm. \n+ The algorithm has some theoretical justification.\n+ Results on the presented experiments are promising. The method is presented as a drop-in replacement to other methods and there seems to be consistent improvement of at least 1-3%, where the bigger improvements are with simpler baseline models and architectures.\n+ The paper also features nice visuals, figures, and explanations, making it easy to understand where it works and where it doesn't.\n\nWeaknesses\n\n- While the paper seems simple, the devil is in the details, which in this case is the hyperparameters and the optimization. To me, it is clear and logical to drop back to the global model in the absence of labeled data for training. However, the key question is what comprises sparse data and labels, when to decide to start relying on the global model more, what happens if suddenly more labelled data kick in making the client not sparsely labelled, etc? How sensitive is the algorithm to these hyperparameter choices?\n- More generally, I would expect a more thorough ablation experimentation on hyperparameters and experimental settings, including the value lambda, the size of the dataset overall as well as per different server-client ratios, temperature tau, different IID settings or ways of distributing labels, with larger datasets, and more modern and expressive architectures, eg Transformer based ones.\n- It has been a while since I worked with federated learning, but from what I know, optimization has always been the major problem. It is not entirely clear how is the presented algorithm is supposed to address problems with the optimization? What if the statistics of the server and client change over time, such that it is unclear when to switch from the global to the local model and vice versa?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes an algorithm for better federated learning, especially in the setting of non IID clients where the learning must adapt to different label distributions. More specifically, the algorithm is an adaptive self-distillation regularization loss that encourages the models to rely more on the local nodes when there is sufficient labels and data points, while relying more on the global model for sparse labels. Additionally, the paper discusses some theoretical results regarding the gradients of the model, as well as how to use statistics of the Hessian matrix of the loss function estimated on the training data to determine flatness of the minima, and thus generalization. The experiments are on CIFAR-10, CIFAR-100, and Tiny ImageNet, using mostly shallow architectures.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See above. I would the very least expect\n\n- more extensive ablations on variuos hyperparameters and under different experimental settings and datasets\n- try different and bigger datasets, including at the very least ImageNet, which by now is not even considered such a big dataset\n- try more modern and high-performing architectures, mainly Transformer-based ones.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "apXawWDdwM", "reviewer": "Reviewer_zf56", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The paper is well-written and easy to understand.\n- The method is simple yet effective, theoretically proven, and empirically demonstrated.\n- The discussion is thorough, and the authors' claims are justified and well-addressed.\n- ASD shows improved performance with any algorithm, showing its effectiveness independently from the algorithm and with different tasks.\n\n**Weaknesses**\n\n- The setting where ASD could shine is not thoroughly discussed. The experiments look closer to the cross-silo FL scenario, as there are only 100 clients and a high participation rate (the paper reports 10%), but statistical heterogeneity and client drift are an issue for cross-device FL as well [3].\n\n[3] Kairouz, Peter, et al. \"Advances and open problems in federated learning.\" Foundations and trends® in machine learning 14.1–2 (2021): 1-210.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes Adaptive Self-Distillation (ASD), a plug-and-play method to be used on top of any FL algorithm to address the client drift problem in Federated Learning (FL). \n\nThe technique adds a regularization term to the local loss function to penalize the mismatch between the local model's softmax predictions and the global model's softmax predictions using the Kullback-Leibler divergence. The Self-Distillation is *Adaptive* in that the contributions to the novel loss term are of different intensity depending on the specific sample, controlled by adaptive weights $\\alpha_k$, which are functions of the entropy of the global model and the frequency of the class the sample belongs to. The ASD loss is constructed so that $\\alpha_k$ is low when the server predictions are noisy, giving more importance to the local predictions and vice versa.\n\nASD is mathematically proven to reduce client drift according to the gradient dissimilarity measure used in [1, 2]. Moreover, this paper discusses flat minima by analyzing the top eigenvalue and the trace of the Hessian of the training loss. It also provides empirical evidence of how applying ASD ensures convergence to flat minima, guaranteeing better generalization properties.\n\n[1] Li, Tian, et al. \"Federated optimization in heterogeneous networks.\" Proceedings of Machine learning and systems 2 (2020): 429-450.\n\n[2] Lee, Gihun, et al. \"Preservation of the global knowledge by not-true distillation in federated learning.\" Advances in Neural Information Processing Systems 35 (2022): 38461-38474.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "This work presents no critical issues. However, some suggestions to improve its quality are detailed below.\n\n- In the introduction, page 2, the paper states that \"the primary motivation behind distillation and regularization works in the context of FL is that the global model will have better representation than the global models\". The authors do not justify this statement, nor are some references provided. Please consider clarifying this aspect.\n- In the introduction, page 2, the authors should consider discussing more related client drift works or, even better, including a subsection on addressing client drift in Related Works, as ASD is designed to reduce client drift specifically. Some possible citations are [4-7].\n- Page 8, it is unclear why $f = \\sum_{i=1}^K f_i \\implies \\mu_1(H(f)) \\leq \\frac{1}{K} \\sum_{i=1}^K \\mu_1(H(f_i))$.\n- The authors may consider adding results using the datasets introduced in [8] for cross-device FL.\n- Line plots could be improved for visibility. For instance, the authors may consider pairing the lines associated with the same method with a similar color (e.g., FedAvg blue, FedAvg + ASD light blue) and/or with different line styles (e.g. [method]: solid, [method] + ASD: dashed).\n- The authors may consider including plots to study the required computation for the empirical experiments (x-axis: average computation required per client; y-axis: accuracy).\n- Proposition A.1.: I have a doubt about a possible missing assumption for this proposition to hold. The gradient must be a convex function to apply the Jensen's inequality, but $\\nabla f$ is convex if $H(f)$ is positive semi-definite, so I think that there is a missing assumption on $H(f)$.\n- Lemma A.4.: It is unclear why the R.H.S of Eq. 40 is $\\frac{1}{K}$ and not $1$.\n\n[4] Wang, Jianyu, et al. \"Slowmo: Improving communication-efficient distributed sgd with slow momentum.\" ICLR 2020.\n\n[5] Reddi, Sashank, et al. \"Adaptive federated optimization.\" ICLR 2021.\n\n[6] Hsu, Tzu-Ming Harry, Hang Qi, and Matthew Brown. \"Measuring the effects of non-identical data distribution for federated visual classification.\"\n\n[7] Kim, Geeho, Jinkyu Kim, and Bohyung Han. \"Communication-efficient federated learning with accelerated client gradient.\" CVPR 2024.\n\n[8] Hsu, Tzu-Ming Harry, Hang Qi, and Matthew Brown. \"Federated visual classification with real-world data distribution.\" ECCV 2020.", "broader_impact_concerns": "I have no concerns about the ethical impacts of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vIySUC7amh", "reviewer": "Reviewer_SCnE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strong points\n\n- The contexts and background are clearly presented. Indeed I am not an expert in FL and just familiar with some well-known papers. The problem setup, introduction and proposed losses are clearly illustrated. \n- The motivation is clearly illustrated. Indeed, Fig 1 clearly demonstrates the limitations of other methods. Then naturally introducing the proposed method. I appreciate this narrative. \n- The experiments seem strong. The empirical results seem significant. \n\nWeak points\n\n- In the introduction. Presenting other’s related work seems a bit convoluted. In general, this will render the paper a bit inaccessible. \n- In the proposed losses. I could understand the importance of weights. But I could not understand why $\\alpha$ is defined in this manner. \n- Why KL divergence in the distillation is preferred? Why not other divergences? I feel a bit confused about the rationale of divergence choice. \n\nOverall I feel like this paper is worthy for publication. But I feel like some parts are still lacking convincing points to justify.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper considered heterogeneity federated learning, where the label distributions changes significantly across the clients. This paper proposed a specific regularization ASD - Adaptive Self-Distillation. The proposed regularization is a weighted KL divergence to ensure the local update on clients similar to the global model.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would suggest authors to revised the following parts\n\n- The introduction. Current introduction seems a bit convoluted by previous work and the actual contribution. I would suggest to briefly discuss the most important points. \n\n- Please address the concerns on the choice of $\\alpha$.\n\n- Please address the concerns on the choice of statistical divergence.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Three reviewers recommended acceptance. The authors have extensively experimented with the requested ablations on hyperparameters and experimental settings, including the value lambda, the size of the dataset overall as well as per different server-client ratios, temperature tau, different IID settings or ways of distributing labels, with larger datasets, and more modern and expressive architectures. They demonstrated sufficient generalization and performance improvement.

One reviewer pointed out that certain confusions remain in the introduction, where too many elements are convoluted. For non-experts in FL, this may present a clear barrier. Some suggestions in this respect include: clarifying in the title that the topic concerns class-imbalance federated learning; using Figure 1 to clearly introduce the problem and explain the proposed solution (it should be explicitly noted in the caption that the figure represents one round of local training on a specific client model; perhaps the figure can be resized and supplemented with illustrations involving several clients); briefly explaining FedAvg and emphasizing the truly plug-and-play nature of FedNTD + Ours, which should also be illustrated.
Additionally, the authors are encouraged to review the referencing style. For instance, the second sentence in the introduction should include proper bracketing, such as: many applications in smartphones **(**Hard et al. (2018); Ramaswamy et al. (2019)**)**, the Internet of Things (IoT), healthcare organizations **(**Rieke et al. (2020); Xu et al. (2021)**)**.
Decision: Accept (poster)