Provably Near-Optimal Federated Ensemble Distillation with Negligible Overhead
Authors: Won-Jun Jang
Hyeon-Seo Park
Si-Hyeon Lee
Venue: ICML 2025 poster (2025)
Keywords: Federated learning
ensemble distillation
data heterogeneity
generative adversarial network
Primary area: optimization->large_scale_parallel_and_distributed

Abstract: Federated ensemble distillation addresses client heterogeneity by generating pseudo-labels for an unlabeled server dataset based on client predictions and training the server model using the pseudo-labeled dataset. The unlabeled server dataset can either be pre-existing or generated through a data-free approach. The effectiveness of this approach critically depends on the method of assigning weights to client predictions when creating pseudo-labels, especially in highly heterogeneous settings. Inspired by theoretical results from GANs, we propose a provably near-optimal weighting method that leverages client discriminators trained with a server-distributed generator and local datasets. Our experiments on various image classification tasks demonstrate that the proposed method significantly outperforms baselines.  Furthermore, we show that the additional communication cost, client-side privacy leakage, and client-side computational overhead introduced by our method are negligible, both in scenarios with and without a pre-existing server dataset.

Provably Near-Optimal Federated Ensemble Distillation with Negligible Overhead
Won-Jun Jang' Hyeon-Seo Park
Si-Hyeon Lee!
Abstract
Wang et al., 2020b; Li & Zhan, 2021; Kairouz al., 2021;
Huang et al., 2023; Karimireddy et al., 2020).
Federated ensemble distillation addresses client heterogeneity by generating pseudo-labels for an unlabeled server dataset based on client predic- tions and training the server model using the pseudo-labeled dataset. The unlabeled server dataset either be pre-existing generated through a data-free approach. The effectiveness of this approach critically depends on the method of assigning weights to client predictions when cre- ating pseudo-labels, especially in highly hetero- geneous settings. Inspired by theoretical results from GANs, we propose a provably near-optimal weighting method that leverages client discrimi- nators trained with server-distributed generator and local datasets. Our experiments on various im- age classification tasks demonstrate that the pro- posed method significantly outperforms baselines. Furthermore, we show that the additional com- munication cost, client-side privacy leakage, and client-side computational overhead introduced by our method are negligible, both in scenarios with and without pre-existing server dataset.
To address the data heterogeneity issue, various approaches have been taken, including regularizing the objectives of the client models (Karimireddy et al., 2020; Li al., 2020a; Liang al., 2019; Yao et al., 2021; Mendieta al., 2022: Son et al., 2024), normalizing features weights (Dong et al., 2022; Kim et al.), utilizing past round models (Yao et al., 2021; Wang et al., 2023b), sharing feature informa- tion (Dai et al., 2023; Yang et al., 2024; Tang et al., 2024; Zhang et al., 2024), introducing personalized layers (Huang et al., 2023), and learning the average input-output rela- tion of client models through ensemble distillation (Chang et al., 2019; Gong al., 2021; Deng al., 2023; Sattler et al. 2020; Lin et al., 2020; Cho et al., 2022; Xing et al., 2022; Park et al., 2024; Wang et al., 2023a; Tang et al., 2022; Zhang et al., 2022; 2023a). In particular, the last ap- proach, federated ensemble distillation, has recently gained significant attention for effectiveness in mitigating data heterogeneity and for its advantage of being effectively ap- plicable heterogeneous client models. requires an un- labeled dataset at the server, for which pseudo labels created based on client predictions. By training on this pseudo-labeled dataset at the server, the server distills the knowledge from the clients. This additional dataset can be either public (Chang et al., 2019; Gong et al., 2021; Deng et al., 2023; Sattler et al., 2020), held only by the server due to its exceptional data collection capability (Lin et al., 2020; Cho et al., 2022; Xing et al., 2022: Park et al., 2024), or obtained through data-free approach (Wang et al., 2023a; Tang et al., 2022; Zhang et al., 2022; 2023a). Note that the performance of ensemble distillation depends on the quality of the pseudo-labels, which ultimately translates into problem of appropriately assigning weights to client predictions for each data point, particularly situations of data heterogeneity. In this research stream of federated ensemble distillation, early studies like FedDF (Lin et al., 2020) applied uniform weighting. Subsequently, algorithms such as Fed-ET' (Cho et al., 2022), FedHKT (Deng et al., 2023), FedDS (Park et al., 2024), and DaFKD (Wang et al., 2023a) emerged, which utilize metries like variance, en- tropy, and judgement of client discriminator as indicators of confidence in client predictions for weighting. However,
1. Introduction
Federated learning (FL.) (McMahan et al., 2017) has ceived substantial attention in both industry and academia as promising distributed learning approach. It enables numer- ous clients to collaboratively train global model without sharing their private data. A major concern in deploying FL in practice the severe data heterogeneity across clients. In the real world, it's probable that clients possess non-IID (identical and independently distributed) data distributions. It is known that the data heterogeneity results in unstable convergence and performance degradation (Li et al., 2020b;
'School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, South Korea, Corre- spondence to: Si- Hi-Hyonn Lee <sihyeon @ kr.>..
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Provably Near-Optimal Federated Ensemble Distillation with Negligible Overhead
Averaged model
Uniform weighting Variance weighting
Entropy weighting
Domain weighting
Our weighting
Figure 1. A toy example of decision boundaries of aggregated models. Each point represents data, and its color represents the label. The background color represents the decision boundary of each model in the RGB channels. The oracle decision boundary, shown by the black lines, corresponds to the a-axis and y-axis. For aggregated models, we consider the parameter-averaged model (McMahan et al., 2017) and eeeebbee-isttlles models using uniform weighting (Lin et al., 2020), variance weighting (Cho et al., 2022), entropy weighting (Deng al., 2023; Park et al., 2024), domain domain-aware aware weighting (Wang et al., 2023a), and ours. Detailed settings are provided in Appendix E.
analysis regarding the rationale behind optimal weighting remains scarce.
In this paper, we suggest novel weighting method for federated ensemble distillation that outperforms previous methods (Fig. 1), with theoretically justified optimality based on some results in generative adversarial networks (GANs) (Goodfellow et al., 2014). Our main contributions summarized in the following:
2. System Model and Related Work
Federated Learning In federated learning, the goal is to cooperatively train a global model based on data distributed among K clients, by exchanging the models between a server and the clients.
We focus on classification tasks in this paper. Let X' denote the data domain and denote the labeling function that out- puts the label of the data æ E X. A model f(:; 0) is parame- terized by 8 E θ where + is the set of model parameters and H = (b) = (:: 0), E }} denotes the class of param- eterized models. For distribution q on X, h denotes the expected loss minimizer q. i.e., h2 arg mink (h), where L. (h) = (h) u(:)) and / is the loss function. Client possesses a (labeled) dataset Sş of T2 data points, sampled over X' i.i.d. according to distribution pt- Then K
We propose FedGO: Federated Ensemble Distillation with GAN-based Optimality, Our algorithm incorpo- rates novel weighting method using the client dis- criminators that are trained at the clients based on the generator distributed from the server.
The optimality of our proposed weighting method is theoretically justified. We define an optimal model ensemble and show that cnowledge-distilled model from an optimal model ensemble achieves the opti- mal performance, within an inherent gap due to the difference between the spanned hypothesis class of ensemble model and the hypothesis class of single model. Then, based on the theoretical result for vanilla GAN (Goodfellow et al., 2014), we show that our weighting method using client discriminators consti- tutes an optimal model ensemble.
p Σ k=1 mk - Pk. where T& client data distribution. The objective of federated learning is given as follows:
is the average of
Σ "gr
min NER
(h) min (1((((()),)(x)))
(1)
hER
K
min NEW
Σ m& (m(x((((
min Eye (h).
We experimentally demonstrate significant improve- ments of FedGO over existing research both in final performance and convergence speed on multiple image datasets (CIFAR- PEFFARR-10100 10/100, ImageNet100), In particular, we demonstrate performance across various scenarios, including cases where the server holds an unlabeled dataset different from the client datasets and where the server does not hold an unlabeled dataset, requiring data-free approaches. Furthermore, we provide com- prehensive analysis of communication cost, privacy leakage, and computational burden for the proposed method, showing that client-side overheads are negligi- ble both in scenarios with and without a pre-existing server dataset.
1=1
k=1
In each communication round f, subset A' of clients down- loads the current server model and trains it based on Sy with the objective of minimizing Lp, (h). Then it sends the trained model to the server. The server aggregates these client models to update the server model. The aforemen- tioned procedure is repeated at the next communication round. For the aggregation of client models at the server, the FedAVG algorithm (MeMahan et al., 2017) constructs the server model with parameter 6 for round of model parameters 0f for k E A' received in round (line 7 of Algorithm 1). When the client data distributions are homogeneous, each Ps is same as p and hence Ep, becomes same as Cp. However, when the client data distributions are heterogeneous, Lpp and Lp are not same, leading to a
the average
For ease of reproduction, our code is open-sourced
2

---

Provably Near-Optimal Federated Ensemble Distillation with Negligible Overhead
Algorithm 1 Federated learning with K clients for T' communication rounds, with ensemble distillation exploiting unlabeled dataset on the server. Client k possesses ny data points, and the fraction C' of clients participate in cach communication round. [+: f) stands for the model with parameter 6, and μ stands for the step size.
Client labeled dataset {S server unlabeled dataset Initialize server model f(= with parameter 0 for communication round = 1 to T do A' < sample |C K] clients
Require:
for parallel client & E A do Sk))
0f +- ClientU ClientUppaaee(( end for
D Gradient update 0--1
with Sk.
0g < ExEA' Eieat k
for server train epoch = to Eg do for unlabeled minibatch u E U do j(u) + - (EREA ux(u) f(u; 8% - 0% - Vog (L(j((), (f
>o Label as a weighted sum of client predictions D Ensemble distillation
end for end for end for return f(:07)
significant degradation in the convergence rate of FedAVG to the global optimum (Li et al., 2020b).
determines the weighting of client k's output. Therefore, designing a better- netter-performing ensemble distillation during the server update ultimately boils down to designing a better- performing weighting function.
In the following, we introduce federated ensemble distil- lation using an unlabeled dataset on the server to address client data heterogeneity.
For the weighting function, FedDF (Lin al., 2020) uses uniform weights for each client, i.e., wx(x) E for all k in A'. Subsequently, algorithms assigning higher weights to the outputs of more confident clients have been proposed. In Fed-ET (Cho et al., 2022), higher weights are assigned to models with larger output logit vari- (ar(f(x:00)) ance, i.e., (x(xx) = Eseat [ar(f(=: FedHKT (Deng et al., 2023) and FedDS (Park et al., 2024) allocate higher weights models with smaller output softmax entropy, i.c., Einrro-yoo (a:00 M1)/
Federated Ensemble Distillation To address client data heterogeneity, there has been a line of research on feder- ated ensemble distillation using an unlabeled dataset on the server. This unlabeled dataset may either be available from the outset (Lin et al., 2020; Cho al., 2022; Deng et al., 2023; Park et al., 2024) produced through a generator trained as part of FL. by taking a data-free approach (Ra- souli et al., 2020; Guerraoui et al.. 2020; Li et al., 2022:
wx(xx) Eieat exp(p Entrrpyy (or (x):))))) where is the temperature parameter. In DaFKD (Wang et al., 2023a), while training global generator and client discriminators at each round, ensemble distillation is performed on unla- beled dataset generated by the global generator by assigning higher weights to models with larger discriminator outputs, i.e., wk (x) = E:EAt (z()) where D is the client k's dis- criminator against the global generator at round t.
Wang et al., 2023e; Fan & Liu, 2020; Behera et al., 2022;
Hardy et al., 2019; Xiong et al., 2023; Zhang et al., 2021;
2023b; Wang et al., 2023a; Zhang et al., 2022; 2023a). With the unlabeled dataset, the server model undergoes additional training to learn the average input-output relationship of client models.
Dī(x)
Algorithm/ describes this federated ensemble distillation, when the client and server model structures are the same. Here a represents the softmax function, and KL denotes the Kullback-Leible divergence. If the model output al- ready includes the softmax activation, then the softmax function is omitted in lines 10 and 11. After averaging client model parameters in line 7, the performance of the server model depends on the quality of the pseudo-labels, as the server model undergoes additional training with those pseudo-labes -abuls.. Morcover, the quality of pseudo-labels 5(() relies on designing the weighting function (+).),, which
For theoretical aspects, generalization bounds of an ensem- ble model are presented in Table I for a binary classification task under é, loss. For n = 22 bound for an ensemble model with fixed weights a1...., aK with Σ. og et al., 2022), with weight function wx (x(x) Eie D()))) is given as (1.2) (Wang et al., 2023a), and with our weight function w described in Section 3.1 is given as (1.3).
π for all k. a generalization
is given as (1.1) (Lin et al., 2020; Cho
(x)x)
3

---

Provably Near-Optimal Federated Ensemble Distillation with Negligible Overhead
Table 1. Generalization bound of the ensemble model of ensemble distillation algorithms. For any ó e (0, 1) and 0, the generalization bounds hold with probability - 6. Here, Pu is the empirical distribution by sampling ny data points i.id. according to pe. dHAR denotes the discrepancy between two distributions, A = infa Lpe (h) + (h),, and TH is growth function bounded by polynomial of the VC-dimension of H.
Algorithm
Generalization Bound
FedDF (Lin al., 2020), Fed-ET (Cho et al., 2022)
- ak Cp (Kip2) + EdussH(PHsP) + dx + O Cp(Enr anhp)) S Ex=n
log(6
)
(1.1)
DaFKD (Wang et al., 2023a)
Cp(Ek= K wi 1%.)) S (K 1). Ek=r ke
Cpn (h7,)) )
0² log 2x
(1.2)
201
f wt M3, S Ex Th (1pe) 4+ (ogge
Ours, Theorem C.I
(1.3)
(8/K))/2m
These bounds relate the loss of an ensemble model (the LHS of (1.1), (1.2) and (1.3) to the average empirical loss of client models (the first term in the RHS of (1.1), (1.2) and (1.3).. Note that the bound (1.1) assumes a fixed weight per client irrelevant to data points, hence there is a lack of anal- ysis for assigning varying weights per data point. Further- more, (1.1) provides a generalization bound that becomes loose it scales with the average distribution discrepancy between p and py across clients. The bound (1.2) assumes specific weighting function of each data point, but it is too loose because it becomes vacuous as K increases. In contrast, (1.3), which is derived in this paper, provides a refined generalization bound for weighting functions. To the best of our knowledge, this is the tightest bound cur- rently provided, detailed in Theorem C. Furthermore, we also emphasize that the bound (1.3) is independent of client data heterogeneity, as it does not include any client data discrepancy term.
learning manner, requiring no class labels. The discrimina- tor aims to distinguish between real images from the dataset and fake images generated by the generator. Meanwhile, the generator strives to produce images that can fool the discriminator.
In Goodfellow et al. (2014), the authors showed that the output of an optimal discriminator against fixed generator can be expressed in terms of distributions of real and fake images.
Theorem 2.1. (Goodfellow et al., 2014, Proposition 1.) For a fixed generator G, let Py and Paata denote the density functions of the generated distribution by G and the real data distribution, respectively. Then the output of an optimal discriminator D for imput data x is given as follows:
Peiata (x)
D(x)
Petato (I) Pg (x)
Moreover, in federated ensemble distillation, our ultimate in- terest is in the loss of the server model, knowledge-distilee from the ensemble model. Note that the hypothesis class of ensemble models is in general larger than that of single models, and hence there exists an inherent gap between the losses of an ensemble model and the knowledge-distille model. However, the above bounds do not provide an analy- sis on this gap.
Using the above result, we develop method of assigning weights to client predictions in Section 3.
3. Proposed Method
In this section, we propose a weighting method for federated ensemble distillation. First, theoretical results are presented in Section 3.1. In Section 3. we define an optimal model ensemble and give bound on the loss of the server model knowledge-distille from an optimal model ensemble. Next, in Section 1.2. we propose client weighting method to construct an optimal model ensemble, based on Theorem 2. In Section 3.2, we introduce our FedGO algorithm, lever- aging the theoretical results. We note that generalization bound of an ensemble model with our proposed weighting method comparable with (1.1) is provided in Appendix C.
In Section 3. we define an optimal model ensemble and show that the server model knowledge-distilled from an op- timal model ensemble achieves the optimal loss within the gap arising from the distillation step, which depends on the inherent difference between the hypothesis classes of the server model and the ensemble model, along with the distri- bution discrepancy between the average client distribution p and the distribution ps of unlabeled data on the server.
3.1. Thcoretical Results
Generative Adversarial Network The generative adver- sarial networks (GANs) are a class of powerful generative models composed of a generator and discriminator (Good- fellow et al., 2014; Gulrajani et al., 2017; Radford et al., 2016; Chen et al., 2016; Zhu et al., 2017: Choi et al., 2018: Karras et al., 2019). They are trained in an unsupervised
1. ENSEMBLE DISTILLATION WITH OPTIMAL MODEL ENSEMBLE
We first define an optimal model ensemble.
Definition 3.1. For K clients, the ensemble of their models
4

---

Provably Near-Optimal Federated Ensemble Distillation with Negligible Overhead
and weight functions {(hy, wy ) is said to be an optimal model ensemble if the following holds:
difference between h and Σ wa l2 over Ps. and 3) the distribution discrepancy between p and ps-
K Σ k=1
Ep [
The following corollary is direct consequence of Theorem 3.2 and Definition 3.1.
Cp w
he
Σ (u(a)) ho(())
Corollary 3.3. (Informal) For an optimal model ensemble
min Cp((i) (1)) hEH
(4)
{(hx, wx)) k= the following holds for any h E H:
K
We remind that the objective of federated learning is train model that minimizes the expected loss over the average client distribution as shown in (1). If {hhg, w&) K k= is an optimal model ensemble, its expected loss over p is less than or equal to the minimum expected loss over achievable by a single model, i.e., minggh (h).
(p(N)) S (p(h) S Cp(h))) - Cp. (h.
Σ A=1
wk h4)
1 + 2 dnan(p.p.).
(7)
Corollary 3.3 demonstrates the powerfulness of an opti- mal model ensemble. If an optimal model ensemble is constituted, the difference between the expected loss of the server model over p and the minimum expected loss (p))) minscH (h) is bounded by the distillation loss, which depends on the inherent difference between the hypothesis class H and the spanned hypothesis class H. along with the distribution discrepancy between p and ps-
However, we cannot guarantee that knowledge-distille model from an optimal model ensemble would be optimal, i.e. achieve minneH (h),, due to the following two rea- sons: 1) the ensemble model Σ Wk- hy may lie outside the hypothesis class H. of single model and 2) the distri- bution used for knowledge distillation (the distribution Ps of unlabeled data on the server) can be different from p. In the following theorem, we present a bound on the expected loss over p of a single model by taking into account these factors. For two hypotheses h, h' E H and a distribution q over X, the expected difference between h and h' over q. denoted Cg (h, h'), is defined as follows:
In the next subsection, we propose a weighting method to constitute an optimal model ensemble.
3.1.2 CLIENT WEIGHTING FOR OPTIMAL MODEL ENSEMBLE
Let us assume that the server has models (hip J trained by clients based on their respective data distributions {p* In the following theorem, we present weight functions {ur K such that the ensemble of (h we 1 constitutes an optimal model ensemble.
(1, A E. (((())))(())))
(5)
Theorem 3.2. (Informal) Let H 2 1E we hy hy E H. wy : + [0. 1}, Ex=1 We (x))
K, x E x} be the spanned hypothesis class, Ps be a distribution on X, and {(ha, w be an ensemble of client models and weight functions. Then for any h H. the following holds:
= =1,
Theorem 3.4. Let the loss function I be convex. Define the client weight functions {wā)
kk=1 as follows:
N& Px(
(x)
K
K
(x(x)
(8)
Ch(h) S Cp(
E wk ha) + f.p (h.
Σ
we ha)
EK n4 P=(xx)
Σ
Ti Pc(x))
1=1
1=1
Then, the ensemble (he wt] is optimal model en- semble, i.e., C, (Ex wn K( S (1t)..
1
+ 2 dRAR(P.Pa).
Theorem 3.4 follows from some manipulations based on the convexity of the loss and the definitions of w 's and hip 's, and its full proof is provided in Appendix B.
The formal statement and the proof of the above theorem are in Appendix A. Let provide brief sketch of the proof. Utilizing the results from Ben-David et al. (2006) and Cram- mer et al. (2008), we have (h) S Ek Wi h+) + (h, Σ Wk- hu). Then from the triangular incqual-
Theorem 3.4 demonstrates that for data point x, weighting according to each client's proportion of having x constitutes an optimal model ensemble. However, even weighting each client according to Theorem 3.4 constitues an optimal model ensemble, it is not feasible without knowing the data distribution p& of each client. Theorem 3.6 addresses this issue based on Theorem 2.1 and provides hints on how to implement an optimal model ensemble.
ity, we obtain (h, Σ Wp hx) < L.p Σk= Wk- ha) 11, (h, ΣK_ wh+hx) - Lp. (h. ΣK WR -he) Now the desired inequality is obtained by applying the results from Ben-Davin et al., Lemma 3.
From Theorem 3.2, we can ascertain the following. The loss of the server model h is bounded by the sum of three losses: 1) expected loss of the ensemble model over p. 2)
Definition 3.5. (Odds): For o E (0, 1), its odds value is defined as (()) I-
5

---

Provably Near-Optimal Federated Ensemble Distillation with Negligible Overhead
Table 2. A comprehensive analysis of additional communication burden, privacy Icakage, and computational burden caused by the
proposed weighting method, compared to FedAVG.
Extra Server Dataset
Generator Preparation
Distillation Dataset
Communication
Privacy Leakage
Client-side Computational Burden
Cost
Server-side
Client-side
S1
GI
DI
Negligihle
Non-negligible Non lon-negligible
Negligible
Negligible
S1
G2
DI
Negligible
Negligible
Negligible
S2
G2
D2
Negligible
Negligible
Negligible
S2
G3
D2
Negligible
Negligible
Negligible
Theorem 3.6. For a fixed generator G with generating distribution Pg. let Dx be an optimal discriminator for gen- erator G and client k's distribution ps. Assume that D& outputs value over (0, 1) using a sigmoid activation ooniv tion, and let & (x(x)) (D)( Then, for x E supp(pa), the following holds:
The main-FL stage operates according to Algorithm I, ex- cept that the server assigns weights for pseudo-labeling according to (9) using the client discriminators. For the server unlabeled dataset U used for distillation, which we call distillation dataset, we consider the following cases: (D1) use the same dataset held by the server, which is pos- sible under (S1); (D2) produce a distillation dataset using the generator.
Tug (x)
aKk (k(x)
Σ i n4 (x(
ΣE TT (z)
A comprehensive analysis of additional communication cost, privacy leakage, and computational burden according to the methods for obtaining the generator and distillation set is provided in Table 2, which shows the trade-off among the methods. In particular, an extra dataset at the server makes the communication cost and the client-side privacy and com- putational burden negligible, the expense of server-side privacy leakage. In the absence of server dataset, the use of off-thhesheeff generator makes all the burdens negligible, but it can be challenging to secure an off- off-the-shelf generator whose generation distribution is similar to the client data distribution. Lastly, the data-free approach (G3)+(D2) does not require an extra server dataset or an external generator. While it requires extra communication to prepare the gener- ator, the additional communication burden, privacy leakage and computational burden on the client side remains negli- gible due to the relatively small number of communication rounds involved, unlike existing data-free methods such as DaFKD (Wang et al., 2023a), which involve both genera- tor and model exchanges every communication round. By decoupling generator preparation from model training, FedGO with (G3)+(D2) provides a lightweight solution.
Theorem 3.6 is a direct consequence of Theorem 2.1, be- cause xxxx)) = pa(ar) from Theorem 2. Theorem 3.6 indicates that if the server once receives the optimal dis- criminators { discriminators to calculate the weights for optimal model ensemble. Note that the generator G only needs to generate a wide distribution capable of producing sufficiently diverse samples. Therefore, one can use an off-the-shelf generator pretrained on a large dataset.
Pz(z]
k=1 trained by the clients, it can use those
3.2. Proposed Algorithm: FedGO
By leveraging the thcoretical results in Section 3.1, we pro- pose FedGO that constitutes an optimal model ensemble and performs knowledge distillation. The main technical nov- elty of FedGO lies in implementing the optimal weighting function wt using client discriminators, which is a versa- tile technique that can be integrated to both the following scenarios with/without extra server dataset: (S1) the server holds an extra unlabeled dataset; (S2) the server holds no unlabeled dataset, thus a data-free approach is needed.
For completeness, let us describe how the FedGO algo- rithm be adapted depending on the cases (S1) and (S2). FedGO largely consists of two stages: pre-FL and ma. In the pre-FL stage, the server and the clients exchange the generator and the discriminators. First, the server obtains a generator through one of the following three methods, and distributes the generator to the clients: (GI) train a generator with an unlabeled dataset on the server, which is possible un- der (S1): (G2) load an off thel- generator pretrained sufficiently rich dataset; or (G3) train a generator through an Fl. approach, e.g., using FedGAN (Rasouli et al., 2020).
A detailed description of FedGO and explanation for Table 2 can be found in Appendices D and G, respectively.
4. Experimental Results
In this section, we present the experimental results. All ex- perimental results were obtained using five different random seeds, and the reported results are presented the mean + standard deviation.
4.1. Experimental Setting
After receiving the generator, each client trains its own dis- criminator based on its dataset and sends the discriminator to the server.
Datasets and FL Setup We employed datasets CIFAR- 10/100 (Krizhevsky, 2009) (MIT license) and downsam-
6

---

Provably Near-Optimal Federated Ensemble Distillation with Negligible Overhead
Table Server test accuracy (%) of our FedGO and baselines on three image datasets at the 100-th communication round. A smaller indicates higher heterogeneity.
CIFAR-10
CIFAR-100
ImageNet100
0.1
0.05
a 0.1
0.05
a 0.1
a 0.05
Central Training
05.2
51.72-0.65 0.65
43.20
1.00
FedAVG
58.65:15.75
46.61
38.93:10.74
36.66-10977
29.44110.41
27.58=1088
FedProx
64.69-22115
55.56=9.86
38.21=00955
34.44=1.26
29.96=00.66
26.99=0.97
SCAFFOLD
61.20:13.9
50.10=10.00
38.15=0.80
36.14=1.06
29.13#0.79
27.08=0.69
FedDisco
56.78:17.22
48.08-8.35
38.81+1.02
36.86-10.88
29.69-10.66
27.54
FedUV
62.58 + 4.83
53.80 3 5.68
38.84 + 0.79
36.17
30.09 1.09
27.32 + 0.65
FedTGP
01.16:6..9
7.78
39.58 0.10
36.56 0.11
29.21 1.13
26.34 1.02
FedDF
71.56+55.09
59.53-9.88
42.74
37.18+1.003
33.48+11.00
30.94+1.60
FedGKD+
12.59:14.10
59.96=8.60
43.35
40.47=1.00
34.10110.67
31.42-0.93
DaFKD
71.52:15.56 49.62+
10.77
44.12:42.25
39.50=1088
33.34100.69
31.59
FedGO (ours)
72.35=9.01
44.66+11.27
41.04=0.99
34.20+:0.71
31.70
Table 4. The number of communication rounds to achieve test accuracy of at least AcCarget-
CIFAR-10
CIFAR-100
ImageNet100
0.1
0.05
0.1
a 0.05
0.1
0.05
AcCurgot
60%
45%
35%
35%
25%
25%
FedAVG
65.6+-22.8
47.4
42.4++12.8
7.50+
22.2++3.1
43.8++7.3
FedProx
38.0=9.1
33.0
45.6-:5.9
86.0
20.8
47.6
SCAFFOLD
14.2
45.8
47.4
76.219.0
22.2-2222
47.8188.4
FedDisco
63.4+:27.8
44.0#11.6
44.6=44.5
76.2+99.
21.8=33.2
51.2++5.6
FedUV
+
+
5.7
4
2..4
4..0
FedTGP
58.6 16.1
40.4 = 4.5
63.6 17.5
69.6 3.9
2.5
4.22
FedDF
1.4
6.0
15.2=5.7
78.0
9.41199
22.04:5.7
FedGKD+
5.6+1.6
4.2+1.2
12.6+333
39.8+19.6
9.0+1.4
14.8++2.5
DaFKD
5.6+1.4
3.0+0.66
13.4+5.4
50.24:27.9
9.0
15.6+4..1
FedGO (ours) 3.0.L0.9
2.0
11.0-2.1
25.4
8.411.0
12.6
pled ImageNet100 (ImageNet100 dataset; Chrabaszez et al., 2017). Unless specified otherwise, the entire client dataset corresponds to half of the specified client dataset (half for each class), and each client dataset is sampled from the en- tire client dataset according to Dirichlet(a)), akin to setups in Lin al. (2020); Cho al. (2022). set to 0.1 and 0.05 to represent data-heterogennouus scenarios. The server dataset corresponds half of the specified server dataset (half for each class) without labels. If not otherwise speci- fied, the server dataset and the client datasets partition the same dataset disjointly. We considered 20 and 100 clients (20 clients not specified otherwise), assuming that 40% of clients participate in each communication round.
2021) and DaFKD (Wang et al., 2023a) that incorporate ensemble distillation. For comparison with other weight- ing methods, we considered the variance-based weighting method of Cho et al. (2022), the entropy-based methods of Deng et al. (2023) and Park et al. (2024), and the domain- aware method of Wang al. (2023a), described Section 2. As an upper bound of the performance, we also compared with central training that trains the using the entire client dataset. FedGO and DaFKD require image generators and discriminators. For the generator, we considered the three approaches (GI), (G2), and (G3) in Sec- tion 3.2. For (GI) and (G3), we adopted the model architec- ture and training method proposed in WGAN-GP (Gulrajani et al., 2017). For (G2), we employed StyleGAN-Xx (Sauer ImageNet (Krizhevsky al., 2012). Unless specified otherwise, we assume (Gl). For dis- criminators, we utilized 4-layer CNN. More experimental details are provided in Appendix E.2.
model directly
et al., 2022), pretrained
Models and Baselines For architecture, we employed ResNet-18 (He al., 2016) with batch normalization lay- (loffe & Szegedy, 2015). For baselines, considered the vanilla FedAVG (McMahan et al., 2017), FedProx (Li et al., 2020a), SCAFFOLD (Karimireddy et al., 2020), Fed- Disco (Ye et al., 2023), FedUV (Son et al., 2024) and FedTGP (Zhang et al., 2024) that not perform ensemble distillation, FedDF (Lin al., 2020), FedGKD+ (Yao al.,
4.2. Results
Test Accuracy and Convergence Speed Table 3 shows the test accuracy of the server model and Table 4 presents

---

Provably Near-Optimal Federated Ensemble Distillation with Negligible Overhead
Table 5. Server test accuracy (%) of our FedGO with generator trained with the unlabcled dataset on the server (Scratch) and with an off-the-shelf generator pretrained on ImageNet (Pretrained) on three image datasets with a 0.05.
CIFAR-10
CIFAR-100
ImageNet100
Generator
Scratch
Pretrained
Scratch
Pretrained
Scratch
Pretrained
Accuracy
72.35+9.01
74.40=6.97
41.04=:.99
41.04
31.70=1.55
32.72:10.18
the communication rounds required for the server model to achieve target accuracy (Acerareet) for the first time, for the baselines and FedGO, on CIFAR-10/100 and ImageNer100 datasets. Our FedGO algorithm exhibits the smallest perfor- mance gap from the central training and the fastest conver- gence speed across all the datasets and data heterogeneity settings.
Ensemble accuracy
8.0
70-
60-
Uniform weighting
50-
Variance weighting
Entropy wei ghtting
40
Domain we ghting
For CIFAR-10 with a = 0.1, our FedGO algorithm shows performance improvement of over 7%p compared to the observe diminishing gain for C10 and ImageNet100. We argue in Appendix F.2 that this is not due to the marginal improvement in FedGO's ensemble performance, but rather due to larger distillation loss as the server model more struggles to keep up with the performance of the ensemble model.
30
weighting
20
40
100
Communicat round
baselines. However,
(a) a = 0.1
80
Ensemble accuracy
70
60-
Uniform weighting
50
Variance weighting Entropy wei ghtiig
Comparison of Weighting Methods Figure 2 shows the ensemble test accuracy along with communication rounds on the CIFAR-10 dataset, according to weighting methods. We evaluated ensemble test accuracy compare the efficacy of each method in generating pscudo-labels. For the baseline weighting methods, we considered the uniform (Lin et al., 2020), the variance-based (Cho et al., 2022), the entropy- based (Deng et al., 2023; Park al., 2024), and the domain- aware (Wang et al., 2023a) methods. For fair comparison, all the baselines follow the same steps except the weighting methods. The effectiveness of our weighting method is demonstrated by its ensemble test accuracy outperforming all the other weighting methods over all communication rounds.
40-
Domain ghtting
Our weighting
20
40
60
80
100
Communicat round
(b) a 0.05
Figure 2. Ensemble test accuracy (%) of FedGO and other baseline weighting methods over communication rounds CIFAR-10 10 with a = 0.1 and a 0.05.
to performance enhancement seems be the larger model structure of the pretrained generator and its training with richer dataset. This enhances the generalization perfor- mance of client discriminators, enabling optimal weighting even for test data. However, since the assumption of The- orem 3.6 does not hold for x E supp(p) \ supp(p,). the portion of data for which an optimal weighting is guaran- teed decreases as the portion of p's support not covered by Pg increases, potentially leading to performance degrada- tion. We note that ImageNet100 is subset of ImageNet, and ImageNet includes the classes of CIFAR- 10 except deer. However, there are several classes of CIFAR-100 not included ImageNet, which could possibly result in no performance gain.
FedGO with a Pretrained Generator If there exists pretrained generator capable of generating sufficiently di- verse data, the server can distribute the pretrained generator to clients instead of training generator from scratch using the server's 's unlabeled dataset, which corresponds to the case (G2) in Section 3.2. This approach has the advantage of sav- ing the server's s computing resources required for training a generator.
Table 5 reports the performance of FedGO for various datasets with 0.05, when using generator trained with the server's unlabeled dataset versus using generator pre- trained on ImageNet (Krizhevsky et al., 2012). We observe that utilizing the pretrained generator results in superior performance on CIFAR-10 and ImageNet100, whereas it remains the same for CIFAR-100 key factor contributing
More Results In Appendix F, we provide more experi- mental results. We report ensemble test accuracy of the baselines and FedGO, demonstrating a larger improvement compared to test accuracy. We also provide results for sce- narios involving larger clients or different model structures, cases where the server dataset is different from the client
8

---

Provably Near-Optimal Federated Ensemble Distillation with Negligible Overhead
datasets, and data-free approaches when no server dataset is available, showing significant performance gains the baselines. Additionally, report the performance of FedGO with reduced server dataset and various discrim- inator training epochs, showing that even with only 20% of the server dataset, FedGO achieves a performance gain of 15%p over FedAVG. Furthermore, FedGO outperforms the baselines even with significantly fewer discriminator training epochs. Finally, we demonstrate the robustness of FedGO under adversarial conditions where subset of clients behave in a Byzantine manner, showing that our method maintains superior performance even when up to 50% the client discriminators are compromised.
References
Adlam, B., Weill, C., and Kapoor, A. Investigating un- der and overfitting in wasserstein generative adversarial networks, arXiv preprint (ttp::9990.111177 2019.
Behera, M. R., Upadhyay, S., Shetty, S., Priyadarshini, S., Patel, P., and Lee, K. F. Fedsyn: Synthetic data generation using federated learning. arXiv preprint 0rXiv:2203.05931, 2022.
Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., and Vaughan, J. W. A theory of learning from different domains. https: / //wwwaa.. alw.kulesza.ea... com com/pubs//ee hann/uubs/ddapt.. adapt_malllmsaaaa ml pdf.
In Appendix G, a comprehensive analysis of communica- tion costs, privacy, and computational costs for FedGO and baselines is provided.
Ben-David, S., Blitzer, J., Crammer, K., Pereira, and Fer- nando. Analysis of representations for domain adaptation. Advances neural information processing systems, 19, 2006.
5. Conclusion
We proposed the FedGO algorithm, which effectively ad- dresses the challenge of client data heterogeneity. Our algorithm was proposed based on theoretical analysis of optimal ensemble distillation, and various experimental re- sults demonstrated its high performance and fast conver- gence rate under various scenarios with and without extra server dataset. The limitation of our work is provided in Appendix H.
Chang, H., Shejwalkar, V., Shokri, R., and Houmansadr, A. Cronus: Robust and heterogeneous collaborative learn- ing with black-box knowledge transfer. arXiv preprint 1et.: 2019.
Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., and Abbeel, Infogan: Interpretable representation learning by information maximizing generative adver- sarial nets. Advances in neural information processing systems, 29, 2016.
Acknowledgements
This work supported in part by the Institute of Informa- tion & Communications Technology Planning & Evaluation (IITP) through the Next Generation Semantic Communica- tion Network Research Center Grant (RS-2024-00398948) funded by the Korean Government (MSIT), and in part by the IITP through 6G Cloud Research and Education Open Hub Grant (0S-0024000428780 funded by the Korea gov- ernment (MSIT).
Cho, Y. J., Manoel, adis, D. Heterogeneous ensemble knowledge transfer for training large models in federated learning. In Proceed- ings of the Thirty-Fira International Joint Conference on Artificial Intelligence (IJCAI) Main Track, 2022.
A., Joshi, G., Sim, R., and Dimitri-
Choi, Y., Choi, M., Kim, M., Ha, J.-W., Kim, S., and Choo, J. Stargan: Unified generative adversarial networks for multi-domain image-to-imas translation. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 8789-8797, 2018.
Impact Statement
In this work, we proposed a federated learning algorithm that demonstrates strong performance in scenarios where client data is heterogeneous. This capability makes our approach highly effective for distributed learning in many practical situations, where data across different clients can vary significantly. By efficiently handling such data di- versity, our algorithm holds the potential to enhance the applicability and robustness of federated learning systems in real-world applications.
Chrabaszez, P., Loshchilov, I., and Hutter, F. A downsam- pled variant of ImageNet as an alternative to the CIFAR datasets. CORR, 2017.
Crammer, K., Kearns, M., and Wortman, Learning from multiple sources. Journal of Machine Learning Research, 9(8), 2008
Dai, Y., Chen, 7.,, Li, Heinecke, S., Sun, L.,, and Xu, R. Tackling data heterogeneity in federated learning with class prototypes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 7314-7322, 2023.
9

---

Provably Near-Optimal Federated Ensemble Distillation with Negligible Overhead
Deng, Y., Ren, J., Tang, C., Lyu, F., Liu, Y., and Zhang, Y. A hierarchical knowledge transfer framework for heterogeneous federated learning. In IEEE INFOCOM 2023-IEEE Conference on Computer Communications, pp. 1-10. IEEE, 2023.
Hinton, G. Distilling the knowledge in neural network. arXiv preprint arXiv: 2015.
Hinton, G., sh Srivastava, N., and Swersky, K. Neural networks for machine learning.
https: http///wwwwccooo cs twwotto..oooo...ooo www.oto..e.ottiiiiiit. edw/ 2ac32//slides//eeeeeeesssss /taetee///eeeeereee slides lecll.eeee. ..cc6.p. lec..pdf.
Dong, X., Zhang. S. Q., Li, A., and Kung, H. Spherefed: Hy- perspherical federated learning. In European Conference Computer Viston, pp. 184.184.20 Springer, 2022.
Huang, W., Shi, Y., Cai, Z., and Suzuki, Understand- ing convergence and generalization in federated learning through feature learning theory. In The Twelfth Interna-
Dwork, C., McSherry. F., Nissim, K., and Smith, A. Cal- ibrating noise to sensitivity in private data analysis. In Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings pp. 265-284. Springer, 2006.
tional Conference on Learning Representations, 2023.
ImageNer100 dataset. https: /wwww/wwww/oo.oo/ https///wwwkkkggg.eem kaw.ke.cgglll.ommmm datasets/ambityye imagemet100.
Dwork, C., Roth, A., al. The algorithmic foundations of differential privacy. Foundations and Trends@ in Theo- retical Computer Science, 9(3-4):211-40) 2014.
loffe, and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference machine learning, pp. 448- 456. pmlr, 2015.
Fan, C. and Liu, P. Federated generative adversarial learning. In Pattern Recognition and Computer Vision: Third Chi- Conference, PRCV 2020, Nanjing, China, October 16-18, 2020, Proceedings, Part III 3, pp. 3-15. Springer, 2020.
Kairouz, P., McMahan, H. B., Avent, B., Bellet, A., Bennis, M., Bhagoji. A. N., Bonawitz, K., Charles, Z., Cormode, G., Cummings, R., et al. Advances and open problems in federated learning. Foundations and trends@ in machine learning, 14(1-2):1-210, 2021.
Gong. X., Sharma, A., Karanam, S., Wu, Z., Chen, T., Doer- mann, D., and Innanje, A. Ensemble attention distillation for preeeciires federated learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15076-15086, 2021.
Karimireddy, S. P., Kale, S., Mohri, M., Reddi, S., Stich, S., and Suresh, A. T. Scaffold: Stochastic controlled averag- ing for federated learning. In International conference on machine learning, pp. 5132-5143 PMLR, 2020.
Goodfellow, I., Bouget-Abadic J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. Advances in neural informa- tion processing systems, 27, 2014.
Karras, T., Laine, S., and Aila, T. style-based generator architecture for generative adversarial networks. In Pro- ceedings of the IEEE/CVF conference on computer vision and puttern recognition, pp. 1401-4410, 2019.
Guerraoui, R., Guirguis, A., Kermarree, A.-M., and Merrer, E. L. Fegan: Scaling distributed gans. In Proceedings of the 21 International Middleware Conference, pp. 193-
Kifer, D., data streams. In VLDB, volume 4, pp. 180-191. Toronto, Canada, 2004.
Ben- Ren-David, S., and Gehrke, J. Detecting change in
206, 2020.
Kim, S., Lee, G., Oh, J., and Yun, S.-Y. FedFN: Feature normalization for alleviating data heterogeneity problem in federated learning. In International Workshop Fed- erated Learning in the Age of Foundation Models in Con- junction with NeurlPS 2023.
Gulrajani, L., Ahmed, F., Arjovsky, M., Dumoulin, V., and Courville, A. C. Improved training of wasserstein gans. Advances in neural information processing systems, 30, 2017.
Hardy, C., Le Merrer, E.,, and Sericola, B. Md-gan: Multi- discriminator generative adversarial networks for dis- tributed datasets. In 2019 IEEE international parallel and distributed processing symposium (IPDPS), pp. 866-877. IEEE, 2019.
Kingma, D. P. and Ba, J. Adam: A method for stochas- tic optimization. In Bengio, Y. and LeCun, Y. (eds.), 3rd International Conference on Learning Represen- tations, ICLR 2015, San Diego, CA. USA, May 7-9, 2015, Conference Track Proceedings, 2015, http: / //arxiv. org. abs 111//11111.2 6980.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual leamn ing for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Krizhevsky, A. Learning multiple layers of features from tiny images. 2009. URL https: /ttttss:/appi.. /api. semanticscholes wwt.sh/.......rr/.o..p...ee.eee.
10

---

Provably Near-Optimal Federated Ensemble Distillation with Negligible Overhead
Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
Park, J.-M., Jang, W.-., Oh, T.-H., and Lee, S.-H. Over- coming client data deficiency in federated learning by exploiting unlabeled data the server. IEEE Access, 2024.
Li, H., Cai, Z., Wang. J., Tang, J., Ding, W., Lin, C.-T., and Shi, Y. Fedtp: Federated learning by transformer personalization, IEEE transactions on neural networks and learning systems, 2023.
Radford, A., Metz, L., and Chintala, S. Unsupervised representation learning with deep convolutional gener- ative adversarial networks. Bengio, Y. and LeCun, Y. (eds.), 4th International Conference on Learning Rep- resentations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http:/ /aaaaaiivv.../ .wwg/ abss 1111/1111110 06434.
Li, T., Sahu, A. K., Zaheer, M., Sanjabi, M., Talwalkar, A., and Smith, V. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429-450, 2020a.
Rasouli, M., Sun, T., and Rajagopal, R. Fedgan: Federated generative adversarial networks for distributed data. arXiv preprint arXiv:2006.072288 2020.
Li, W., Chen, J., Wang, Z., Shen, Z., Ma, C., and Cui, X. Ift-gan: Improved federated learning generative adver- sarial network with maximum mean discrepancy model aggregation. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Sattler, F., Marban, A., Rischke, R., and Samek, W. Communication-effiieent federated distillation. arXiv preprint arXiv:0012.00622 2020.
Li, X., Huang, K., Yang. W., Wang, S., and Zhang, Z. On the convergence of fedAvg on Non-IID data. In International Conference on Learning Representations, 2020b.
Sauer, A., Schwarz, K., and Geiger, A. Stylegan-xl: Scaling stylegan to large diverse datasets. In ACM SIGGRAPH 2022 conference proceedings, pp. 1-10, 2022.
Li, X.-C. and Zhan, D.-. Fedrs: Federated learning with restricted softmax for label distribution non-iid data. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 995-1005, 2021.
SSwarzz, S. and Ben-David, S. Under- standing machine learning: From theory al- gorithms. schpts///oneerttaaniiiggeeeeeeen understanding wachnnngaaaan---aeeeee machinne-leermnnn-theee -a -algori wwrs.htttt... .dd.
https://ww..s..hii.ac....
Liang, X., Shen, S., Liu, J., Pan, Z., Chen, E., and Cheng, Y. Variance reduced local sgd with lower communication complexity. arXiv preprint arXiv: /9/2. 12844, 2019.
Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. In Bengio, Y. and LeCun, Y. (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceed- ings, 2015. URL htth:///// /ttp///aaxvvvvvvv htrg/abb/1440 1556.
Lin, T., Kong, L., Stich, S. U., and Jaggi, M. Ensemble distillation for robust model fusion in federated learning. Advances in Neural Information Processing Systems, 33: 2351-2363, 2020.
Loshchilov, I. and Hutter, F. Sgdr: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2022.
Son, H. M., Kim, M.-H., Chung, T.-M., Huang, C., and Liu, X. Feduv: uniformity and variance for heterogencous federated learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2024.
Marfoq, O., Neglia, G., Vidal, R., and Kameni, Per- sonalized federated learning through local memorization. In International Conference on Machine Learning, pp. 15070-15092. PMLR, 2022.
pp.
McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A. Communication--ffcce efficieen learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273-1282. PMLR, 2017.
Tang. Z.. Zhang, Y., Shi, S., He, X., Han, B., and Chu, X. Virtual homogeneity learning: Defending against data heterogeneity in federated learning. In International Con- ference Machine Learning, pp. 21.2111111111 21111-21132. PMLR, 2022.
Mendieta, M., Yang, T., Wang, P., Lee, M., Ding, Z., and Chen, C. Local learning matters: Rethinking data het- erogeneity in federated learning. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8397-8406, 2022.
Tang, Z., Zhang, Y., Shi, S., Tian, X., Liu, T., Han, B., and Chu, X. FedImpro: Measuring and improving client update in federated learning. In The Twelfth International Conference on Learning Representations, 2024.

---

Provably Near-Optimal Federated Ensemble Distillation with Negligible Overhead
Wang, H., Yurochkin, M., Sun, Y., Papailiopoolos, D., and Khazaeni, Y. Federated learning with matched averaging- In International Conference on Learning Representations, 2020a.
Yoon, Y., Hu, D., Weissburg, I., Qin, Y., and Jeong, H. Redifine: Reusable diffusion finetuning for mitigating degradation in the chain of diffusion. arXiv preprint arXiv:2407. 17499,, 2024.
Wang, Hi, Yi, Xu, W., R., Zhan, Y., and Zeng, Z. Dafkd: Domain-aware federated knowledge distillation. In Proceedings of the IEEE/CVF conference Computer Vision and Pattern Recognition, pp.
Zhang, J., Guo, S., Guo, J., Zeng. D., Zhou, J., and Zomaya. A. Y. Towards data-independent knowledge transfer in moneel-heterogeeeeuus federated learning. IEEE Transac- tions on Computers, 72(10):2888 2901, 2023a.
2023a. Wang, H., Xu, H., Li, Y., Xu, Y., Li, R., and Zhang. T. Fed- CDA: Federated learning with cross-rounds divergence- aware aggregation. In The Twelfth International Confer- ence on Learning Representations, 2023b.
Zhang, J., Zhao, L., Yu, K., Min, G., Al-Dubai, A. Y., and Zomaya, A. Y. A novel federated learning scheme for generative adversarial networks. IEEE Transactions Mobile Computing, 2023b.
Wang. J., Liu, Q., Liang, H., Joshi, G., and Poor, V. Tack- ling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 13:7611-7622 2020b.
Zhang, J., Liu, Y., Hua, Y., and Cao, Fedtgp: Trainable global prototypes with adlative-margin-ennannee con- trastive learning for data and model heterogeneity in fed- erated learning. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pp. 16768-16776. 2024.
Wang, J., Xie, G., Huang, Y., Lyu, J., Zheng, F., Zheng, Y., and Jin, Y. FedMed-GANY Federated domain translation unsupervised cross-modality brain image synthesis. Neurocomputing, 546: 126282, 2023c.
Zhang, L., Wu, D., and Yuan, X. Fedzkt: Zero-shot knowledge transfer towards resource-constraing feder- ated learning with heterogeneous on-device models. In 2022 IEEE 42nd International Conference on Distributed Computing Systems (ICDCS), pp. 928-938. IEEE, 2022.
Wang. Y., Zhang, J., and Wang, Y. Do generated data always help contrastive learning? In The Twelfth International Conference Learning Representations, 2024.
Zhang, X., Zhu, X., Wang, J., Bao, W., and Yang, L. T. Dance: Distributed generative adversarial networks with communication compression. ACM Transactions on In- temet Technology (TOIT), 22(2):1-32, 2021.
Xing, H., Xiao, Z., Qu, R., Zhu, Z., and Zhao, B. An efficient federated distillation learning system for mul- titask time series classification. IEEE Transactions on Instrumentation and Measurement, 71:1-12, 2022.
Zhu, J.-Y., Park, T., Isola, P., and Efros, A. A. Unpaired image-to taaggeto-immag translation using cycle- yylle-consistent adver- sarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 2223-2233 22322, 2017.
Xiong, Z., Li, W., and Cai, Z. Federated generative model on multi-source heterogeneous data in iot. Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 10537-10545, 2023.
Yang, C., Shen, Y., Xu, Y., Zhao, D., Dai, B., and Zhou, B. Improving gans with a dynamic discriminator. Advances in Neural Information Processing Systems, 35:15093- 15 10144,20 2022.
Yang, Z., Zhang, Y., Zheng, Y., Tian, X., Peng, H., Liu, T., and Han, B. FedFed: Feature distillation against data heterogeneity in federated learning. Advances in Neural Information Processing Systems, 36, 2024.
Yao, D., Pan, W., Dai, Y., Wan, Y.. Ding, X., Jin, H., Xu, Z., and Sun, L. Local-global knowledge distillation in heterogeneous federated learning with non-iid data. arXiv preprint arXiv:2107.0000 2021.
Ye, R., Xu, M., Wang. J., Xu, C., Chen, S., and Wang, Y. FedDisco: Federated learning with discrepancy-awart collaboration. In International Conference on Machine Learning, pp. 39979-39902. PMLR, 2023.
12

---

Provably Near-Optimal Federated Ensemble Distillation with Negligible Overhead
A. Formal Statement and Proof of Theorem 3.2
In addition to the setups and definitions introduced in Section we assume binary classification task, i.e., y(œ) E [0, 1] and h(z) {0, 1}. coupled with & loss.
We first present some definitions and lemma.
Definition A.1. (Kifer et al., 2004, Definition 1) For two distributions and g' over domain X' let H. denote hypothesis class on X' and (h) for h E H. denote the set { E X h(x) = 1}. The H-divergence between y and g' is
dx(9. q¹) sup heH
Pro Pre [I(h]]].
(10)
Definition A.2. For a hypothesis space H. the symmetric difference hypothesis space HAH is the of hypotheses
g H&H + g(z) = h(x) @ (xx) for some h' H
(11)
where @ is the XOR function.
Lemma A.3. For hypotheses h, h'' E H and distributions 9 on X, we have 1
(hg(h,h') Lor (h, duanlggg')
(12)
Proof. By the definition of H/HH-iistace,, we have
duau(a, d') = sup Pranglh(x)+ + [x)]]] - Prow [1(x) + (x)))
(13)
hEA
sup Lo h1) - for (h, R())
(14)
NE7
(h, h²) (h.
(15)
which completes the proof.
Now we are ready to present the formal statement and proof of Theorem 3.2.
Theorem A.1. For binary classification task with ε loss, consider hypothesis class H. such that h H outputs 0 or I and
its spanned hypothesis class H 2 {E = Wy hig l4g E H, wk : + [0, for all k = I,... K, ER=1 Wi = 13. For any
h E H and (E Wk hx) E H. the following holds:
I wk ha) #2 _dRAR(U.P.)
(x(h) cp
wk he) f.p (h1, Σ
(16)
k=1
Proof. We have
(h(h) = Epl((h()) v(x)
(17)
< Ep((((x))
E we hx)( + E, E,P((
hu (x)) xx() wn
(18)
k=1
K
Cp
Ws
hx) + C, Σ wp hx)
(19)
Jmm
1=11
by triangle inequality (Ben-David et al., 2006; Crammer et al., 2008).
Since A < B AA B), letting A Ly Ek= wp hx), B Cp, (h, R== we ha), the RHS of (19) is upper-bounded
by
K
K
K
K
Cpt
wkk hx) + Cp, (h,
Σ
wk hx) + (1, Σ
wk - hwa) - Lp. (h, E wk hp)l
(20)
k=
k
Cp1
wk hx)) + Ly. (h,
Σ
wk hx) d
(21)
1=1
13

---

Provably Near-Optimal Federated Ensemble Distillation with Negligible Overhead
by the definition of dRAR and Lemma A.3. Thus, we prove Theorem A.I.
B. Proof of Theorem 3.4
Before we start the proof of Theorem 3.4, we present the following lemma with the setups and definitions introduced in Section 2.
Lemma B.1. Let the loss function be convex and {hu Jim C H. For the functions {wttkkm KK 3.4, the following holds:
defined in Theorem
Cp hx) S En Tk Lpe (hx).
(22)
Proof. Note that
Cx ut -hx) = Egup [< (z(x)) (x(x),
(23)
/ 15 wi(()) (x) p(r)dr
(24)
/ (<Ex ( (x())) E, TJ P,(2)dr
(25)
/ ((())) (t) v(z)) mg Ds(x)dx
(26)
$ / wi) (x
Ej Ty (r)dr
(27)
/ Ex Te(o) DELTEPI
(x) v(x)) Σn as (5(r)dr
(28)
Σ / Th (x())) (he(z)) de
=
(29)
Σ Tk' / ((hx(a)) px(x)dr
(30)
Σ Th (hx),
(31)
where (27) holds due to the convexity of loss function I(:,)). This completes the proof.
Now we present the proof of Theorem 3.4.
Proof. For h E H. we have
(h) Excp (M(x),
(32)
/ (1(()),(()) p(z)de
(33)
/ ((b(o), Ee Th Pa (x)
(34)
En Th S [1(x))),, pa(x)dr
(35)
Et Th Cm (h)
(36)
Ee Fx Lp (hip))
(37)
Hence, it suffices to show that
2. Ex kipd) Ex Th Lpe (hip),
(38)
and this the direct result of Lemma B.1 with (hx) = (11 1
14

---

Provably Near-Optimal Federated Ensemble Distillation with Negligible Overhead
C. Generalization Bound with Empirical Loss Minimizer
In this section, we present the generalization loss bound the ensemble of empirical loss minimizers of clients with our weighting method.
Theorem C.1. For binary classification task with & loss, the following holds for our weighting function {wg)k= k==== in Theorem 3.4:
defined
K I
+ Vlog(TH(2n))
Cpl
wkk
Σ
Lp, (ha
(39)
(/K)y
22n /
k=1
1=1
4 v log((H THe (2mx)) T
$ (bg)
(40)
(8/K) 22
where Pr is the empirical distribution by sampling T& data points i.i.d. according to P = Ek= Pk. and TH is growth function bounded by polynomial of the VC-dimension of H.
k=
Compared to (1.2),
that the ensemble of empirical loss minimizers with weighting method has tighter generalization bound without the factor of (K + 1).
Before we prove Theorem C.I, we present the following theorem.
Theorem C.2. (Shalev-Swwartz & Ben-David, Theorem 6.11) Let H be hypothesiss class and let TH be its growth function. Then, for every distribution q on X and every 5 (0, 1), with probability of at least 1 5 over the m i.idd choice of S qt with its empirical distribution we have
(h)(h) - fa(h)) 4+ Vlog(TH (2m))
(41)
8/2m
We also present the bound of growth function TH-
Lemma C.1. Shaarv-Shwartz & Ben-Davia Lemma 60) Let H. be a hypothesis class with VC-dimension of H. is smaller than d, i.e. VCDim(H) < d < co. Then, for all m, (H(m) S >i=o (77). In particular, if m > + 1, then TH(m)) & (em/dy", where is Euler's 's number.
Now we present the proof of Theorem C.I.
Proof. By the result of Lemma B.1 with [K = (1, Jim we can derive
Ex wh )
S SE Th Lyp (hice)
(42)
Also we note that S₁ ~ Pk nk We can derive the following inequality for k = 1,... K using Theorem C.2. With probability at least of (8/K).
Cp. (h S Lp. (hT.) + + /log(Tn(2m)) (6/K) 2n1
(43)
where TH growth function bounded by polynomial of the VC-dimension of
15

---

Provably Near-Optimal Federated Ensemble Distillation with Negligible Overhead
By the union bound, have
Cp S Cpn (k%pe) 4+ Vlog( TH (((((H((22))) (2nx)) (6/K) 22n
P
(44)
K
+ v (log(TT (2mx) +
P
pp (hppg) 2 Cpe (h2p.)
(45)
(6/ ()) 2mm
-k=1
K
4 + /log( THe (2mx)
2 Σ P
Cpa (12p)) 2 Lp (17pe)
+
(46)
(ō/ /
k=1
21 Σ (6/K)
(47)
21
(48)
Hence, with probability at least 8, following inequality holds for all 1, ... K:
Cpe (hie) I Cpe (hpc)) + + v (og (2rxx))
(49)
(8/K)2n))
Furthermore, by definition of p,
Cp(h)) En Th Lpx (h1)) Σx Th Lpa (hzc))
(50)
(51)
By combining the above results, with probability of at least - 8, we have
Cp(E*=
k== wi
5
Σx Tk
Lpe (hip)
(52)
K
+ 4+ (2
Tk.
Cpp (1%p))
(53)
(6/K)y K) (KKK)/n
1=1
K
4 + v
(TH [og((T
Leu (1ipe)
+ m
(54)
(8/K) 2
k=1
4 + Log( TH (2mx))
$ Cp(h)) + Σ
T&
(55)
(5/K) (5/K)/2n) 2mk
1mmm
This completes the proof.
D. Description of FedGO
Figure illustrates the operation of FedGO, Algorithm 2 presents a pseudo-code of FedGO. For training discriminators. each client optimizes the GAN loss with respect its labeled dataset. A pseudo-code of client discriminator update is provided in Algorithm 3.
E. Experimental Details
All experiments were conducted in Python 3.8.12 environment using a 64-core Intel 2.90GHz Xeon Gold 6226R CPU with 512GB memory, and an RTX 3090 GPU. We also implemented the algorithms using PyTorch with version 1.1..0.
E. Detailed Experimental Setting and Analysis of Toy Example (Figure 1)
For the toy example in Figure 1, the dataset generated from a mixture of four Gaussian distributions, each with a variance of 3. The top row of Figure 4 shows the global data distribution and the datasets held by four clients. Each point represents
16

---

Provably Near-Optimal Federated Ensemble Distillation with Negligible Overhead
1. Pre-FL Client discriminators preparation
Server
Server
(G3) local train
Client 1
G
Unlabeled dataset
Client
Di
G
(G2) load
G
G
D2
Pretrained
Generator
(G3) FL train
Client C
Client discriminators
G
Dr
1. Generator preparation (G1 or G2 or G3)
2. Client discriminator training
3. Client discriminators aggregation
2. Main FL Ensemble distillation along with client discriminators
Client 1
Server
Label)
fi
Client 2
Label)
D
Pseudo- label
Client C
Label)
fc
Client models
Client discriminators
Server dataset
Server model
Server dataset pseudo-labeling
4. Server model training & distribution
1. Client model training
2. Client models aggregation
Figure Illustration of FedGO algorithm.
data, with its color indicating the class label: data from Gaussians with means at (4, 4) and (-4, -4) are labeled as Red, data from the Gaussian with mean at (-4, 4) as Blue, and data from the Gaussian with mean at (4, -4) as Green. Each Gaussian provides 300 data samples. Each client holds 90% of data from the Gaussian whose mean is in a certain quadrant (the 3rd, 4th, 2nd, 1st quadrants for Clients 1, 2, 3, and 4, respectively), and the remaining 10% from Gaussians with means other quadrants. The clients' global dataset comprises 1200 samples. with 300 from each Gaussian. The server unlabeled dataset comprises 300 data, uniformly distributed the square [ 12, 12] x [- 12, 12].
Each client trains 3-layer MLP classifier for 2 epochs using its dataset, and 3-layer discriminator for epoch using its dataset as real dataset and server dataset as fake dataset. We used Adam (Kingma & Ba, 2015) with learning rate 0.001 and (B1, (2) = (0.9, 0.999) for classifier optimizer, and RMSprop (Hinton et al.) with learning rate 0.00005 for discriminator optimizer. Also we used a batch size of 64 for both.
The bottom row of Figure (same Figure 1) illustrates the decision boundaries of server models. The leftmost plot from the model with averaged client model parameters, while the remaining plots are from the server models trained via ensemble distillation for 2 epochs using pseudo-labeled dataset: the global dataset is lscull-labeled using uniform weighting (Lin et al., 2020), variance weighting (Cho et al., 2022), entropy weighting (Deng et al., 2023; Park et al., 2024), domain-aware weighting (Wang et al., 2023a), and our weighting method. The background color indicates the decision boundary in RGB channels. Given the Gaussian distributions, the optimal decision rule is red in the 1st and 3rd quadrants, blue the 2nd quadrant, and green in the 4th quadrant. Thus, the oracle decision boundary aligns with the x-axis and y-axis, depicted by black lines.
17

---

Provably Near-Optimal Federated Ensemble Distillation with Negligible Overhead
Algorithm 2 FedGO algorithm with K clients for T communication rounds. f(: stands for the model with parameter stands for the step size, and (x)) stands for the odds value of Pa(x).
Require: Client labeled dataset {5
Initialize server model [(; 02) with parameter 0%
Prepare generate and unlabeled dataset U for parallel client & {1, 2, .... K] do
By one of the methods in Table
D₁ Discriminalor pdate Ss)
Detailed in Algorithm 3
end for for communication round = 1 to T' do
A' t sample |C' K clients
for parallel client & E A' do
0% f- C'ieetUU ClitetUpdatee(!"" Skk) end for 8% ExEA' EiEnt 0%
> Gradient update 0--1 with Sk.
for server train epoch = 1 to E, do for unlabeled minibatch u E U do
j(u) + o(EEA wg(u)= f( 0 < 03 p Vog KL. (L.) 0)))
ng. D W (w) = EiEd (.))
end for end for end for return f(:05)
Algorithm 3 Discriminator update for E, epochs. Pl stands for the step size and D(: 0) is the parameterized discriminator with parameter A.
Require: Generator G, labeled dataset S Initialize discriminator model D(: 69) with parameter 0º for epoch e = to Eg do 0% + -62 -6-1
for minibatch m E S do
(Freats 8) < (images, labels) pair of minimatch m
I. fake + generated images by generator G (Ds 02)) t log(D(z 0%) + loog(1 - D(Z 07))
8% - 8 V (osSLAN(D((:)))))) end for end for return D(::0e))
Update with gradient for vanilla GAN loss
The averaged parameter model exhibits a blurred decision boundary compared to models trained via ensemble distillation. Furthermore, among the models with ensemble distillation, the decision boundary of the model trained via our weighting method is closest to the oracle decision boundary.
E.2. Detailed Experimental Settings for Image Classification Tasks
Hyperparameter Tuning We identified the best-performing hyperparameters on CIFAR-100 with Dirichlet a 0.05 and used the same values for other settings. During the ensemble distillation process, trained both clients and server with the Adam optimizer (Kingma & Ba, 2015) at a learning rate of 0.001 with batch size 64, without weight decay. The (81, B2) parameters for Adam were set to (0.9, 0.999). Additionally, we applied cosine annealing (Loshchilov & Hutter, 2022) to decay the server learning rate until the final communication round T 100 as in Lin et al. (2020), except for the results of F33 and F.5.
For the client and server classifier training epochs, we performed grid search to find the optimal number of training
18

---

Provably Near-Optimal Federated Ensemble Distillation with Negligible Overhead
Server data
Global data distribution Client I's data
Client data
Client 3's data
Client 4's data
Averaged model
Uniform weighting
Variance weighting
Entropy weighting
Domain weighting
Our weighting
Figure 4. Top row represents the server and clients' datasets. Bottom row, showing the dccision boundaries of the aggregated models, is the same Figure and copied here for case of analysis.
epochs. The initial grid was {5, 10, 30, 50), and the experiments were conducted with 30 client epochs and 10 server epochs (E, = 10) for CIFAR-10/100. To leverage the increased number of steps due to the additional number of data, experiments on ImageNet100 were conducted with 10 client classifier epochs and 3 server classifier epochs (E, 3).
To train the generator utilized by our FedGO from scratch, we trained the WGAN-GP model following the training method proposed in Gulrajani et al. (2017). The generator and discriminator of WGAN- were trained using the Adam optimizer with learning rate of 0.0002 and (2) (0, 0.9). The training was conducted with batch size of 64 until the generator completed 100,000 gradient steps. The generator updated every 5 steps of the discriminator, and gradient penalty coefficient À of 10 was used.
When training a generator in a data-free setting, i.c., the case (G3), we utilized the FedGAN (Rasouli et al., 2020) algorithm. The generator was trained for only T' communication rounds, with each local generator and discriminator trained for epochs per round.
For the client discriminator, adopted the hyperparameters from http:///gghbbbbooomouuuuuggggaat pytorch/blob/masterrggnn mnist.py and trained it with batch size of 64 for 30 epochs for CIFAR-10/100, and for 10 epochs for ImageNet100. The optimizer Adam was used with learning rate of 0.0002, and (31, BB2 (0.5,0.999)
FedProx (Li et al., 2020a) introduces proximal term to the client training loss, which helps to address heterogeneity by penalizing large deviations from the server model. The proximal term is multiplied by coefficient μ and added to the primary objective loss. We performed a grid search to tune the value f from {0.1, 0.05, 0.01, 0.005, 0.001, 0.0005, 0.0001), and chose the best value μ 0.001.
FedDisco (Ye al., 2023) determines client parameter aggregation coefficient by leveraging both the dataset size and the discrepancy between local and global category distributions. Here, controls the influence of the discrepancy, while b adjusts the baseline aggregation coefficient. Following the FedDisco paper, we adopted the best best-performing parameters = 0.5 and b = 0.1, which were tuned for optimal performance for CIFAR-10.
FedUV (Son et al., 2024) introduces two regularization terms classifier variance and encoder representation uniformity- to emulate IID conditions in federated learning, mitigating local bias in non-IID settings. The loss function combines the standard classification loss with the uniformity term (weighted by X) and the variance term (weighted by a). By following Son et al. (2024), we set the hyperparameters generalization and personalization across clients.
0.5 and 2.5 as they achieved the best trade-off between
FedTGP (Zhang et al., 2024) proposes Trainable Global Prototypes (TGP) with Seapiive-Margi--Eee Enhanced Contrastive Learning to address data heterogeneity by improving class separability and semantic alignment among clients. The model trains prototypes using contrastive learning with a temperature- scaled margin. By following Zhang et al. (2024), for the TGP model, we used a batch size of 10, trained for 1,000 epochs with a learning rate of 0.005 and a margin parameter r
19

---

Provably Near-Optimal Federated Ensemble Distillation with Negligible Overhead
100. The contrastive loss was weighted by À 0.001 to balance its influence with the main task objective.
FedHKT (Deng et al., 2023) and FedDS (Park et al., 2024) introduce temperature parameter > 0, which allows client weights to approach uniform weighting as T increases. By following Deng et al. (2023), we set T = 1.
FedGKD (Yao et al., 2021) introduces an additional buffer of length M on the server, where the server model is stored after each round. The server then creates an additional model with averaged parameters from the models stored in the buffer and sends this model to the clients each round. Each client uses a temperature parameter T to compute the knowledge distillation loss on the received additional model, multiplies this loss by 7/2, and adds to the primary objective loss. Consequently, it is necessary to tune three additional hyperparameters: M, T, and r We conducted grid search with M and in {1, 3, 5, 10} and y in 1. 0.05, 0.01, 0.005, 0.001J. The best performing parameters were M 5, 3, and y 0.001.
Similar to our FedGO, DaFKD (Wang et al., 2023a) utilizes discriminators to implement client weighting function. However, unlike FedGO, DaFKD trains the generator and discriminators collaboratively. To focus on the weighting method, the domain-aware weighting method in Figure 2 is implemented by only modifying the weighting step in our FedGO algorithm.
Model Implementation We used R8 (He et al., 2016) as the classification model, following the implemen- tation from http:///shhh.coom/uuuuu////oooot
Additionally, our FedGO requires extra generator and discriminator models. When training the generator from scratch, we utilized the WGAN-GP model as proposed in Gulrajani et al. (2017), following its official open-source implementation' We re-implemented this code in PyTorrc for our experiments. For a pretrained off-the-shelf generator, we utilized StyleGAN-XL (Sauer et al., 2022) model pretrained on ImageNet (Krizhevsky et al., 2012) with resolution of 32x32. We downloaded the and implemented the model using these param- eters. For the client discriminator, we adopted a simple 4-layer CNN discriminator, following the implementation from mnist.py. To address the widely known overfitting issue of the discriminator (Adlam al., 2019; Yang et al., 2022) and the resulting dominance of client weights, we employed composition of two sigmoid activations for the discriminator output. This ensures that the odds value D for client k's discriminator Dr is constrained between and e.
model parameters from
https://github.. www.eau..oom////// woom///uryatttj//DGGAAA Caw.ei//DDDCAAAAAAATT-SOOoO
FedTGP (Zhang et al., 2024) utilizes an additional TGP model. We implemented TGP model following its official open-source implementation from https:/gittub.com/iiinn22aaaaaf
Heterogeneous Client Data Split To introduce non-iid distributions among client datasets, we ensured that each client's s distribution follows a Dirichlet distribution Dir(a),, similar as in Lin al. (2020); Wang et al. (2020a): Marfoq et al. (2022); Li al. (2023). As the parameter a increases, each client tends have a more homogeneous distribution, whereas smaller values result in increased data heterogeneity among clients. We conducted experiments for each dataset with a values of 0.1 and 0.05, The number of data samples that each client has per class for CIFAR- 100/00 datasets with a values of 0. I and 0.05 is illustrated in Figures 5 and 6. It's worth noting that ImageNet100 also has 100 classes, so the trends observed in CIFAR-100 would likely align with those in ImageNet100. We can observe that when a = 0.05, the difference in the number of data samples per class for each client is more pronounced compared to when a = 0.1. This results in more skewed distributions for individual clients.
Details for Dataset We normalized the pixel values of all image datasets to fall within the range [-1, 1], ensuring that the generated data also has pixel values within this range. Additionally, for both the training datasets of clients and the server's s unlabeled dataset, we conducted further data augmentation using PyTorch's 's random horizontal flip.
Selection of AcCarget We used the highest multiple of 5 of the test accuracy (%) achieved by the FedAVG algorithm within 100 rounds for all five different random seeds as AcCuaged for Table 4.
F. Additional Experimental Results
F.I. Results with 100 Clients
Figure shows (a) the test accuracy of the server model, (b) the test accuracy of the ensemble model, and (c) the test loss of the ensemble model during the training process for K 100 clients on CIFAR-I0 dataset with 0.05. The latter
http:://giiuuuucmmmiill/////mmpp wgan training
20

---

Provably Near-Optimal Federated Ensemble Distillation with Negligible Overhead
9-
7-
7-
5
g 5
3-
3-
1-
1
2
3
6
7 8
10 Client
11
12 18 14
15
16
19
0
4
5
7
8
9
10
13
14
16
17
Client
(a) a 0.1
(b) a 0.05
Figure 5. Client data split for CIFAR-10 with 0.05
90
80
80
70
70
60
60
30
20
20
10
10
0
1 2
3
6 7
8 9 10 11 12 13
14
15
16
17
1819
2
3
4
5
7 8 9 10 Client
12
13
14
15
16
19
Client
(a)
0.1
(b) 0.05
Figure 6. Client data split for CIFAR-100 with = 0.1,
two measures were evaluated only for algorithms incorporating ensemble distillation. FedGO achieves the test accuracy of 69.52%, which slightly lower than 72.35% with 20 clients (Table 3). In comparison, FedAVG, FedProx, FedDF, FedGKD* and DaFKD show significant performance drops 33.40%, 35.07%, 44.36%, 45.44%, and 59.62%, respectively, This demonstrates that even in settings with large number of clients, FedGO exhibits robust performance compared the baselines.
In terms of the test accuracy and the test loss of the ensemble model, FedGO consistently demonstrates superior performance across all rounds compared to the baseline algorithms. Furthermore, unlike the baseline algorithms, whose test loss initially decreases but then becomes unstable and increases from early rounds, FedGO's loss converges with small deviation.
F.2. Ensemble Test Accuracy Comparison and Analysis
Figure 8 shows the ensemble test accuracy on the server's unlabeled dataset during the training process for our FedGO algorithm and the baseline ensemble algorithms: FedDF, FedGKD+ and DaFKD. It demonstrates that using pseudo- labels generated by theoretically guaranteed weighting methods allows the server to achieve higher final performance and faster convergence.
However, in Table of the paper, the performance gap between method and the baselines CIFAR-100 and Ima- g0 was not as large as that CIFAR-10. We infer the reason from Theorem 3.2. The second term the RHS of Theorem 3.2 can be interpreted as the distillation loss due to the difference between the hypothesis class and the spanned hypothesis class. Even our ensemble close to optimal, the mowweede-ddsttilee server model may not follow the performance of the ensemble it hard for a single model to learn the labuds-labels, and we conjectures that it becomes harder as the number of classes increases.
21

---

In the main paper, we conducted experiments with ResNet-18 model structure. In this subsection, we present the results with VGGI1 (Simonyan& & Zisserman, 2015) (with BatchNorm Layers (Ioffe & Szegedy, 2015)) and ResNet-50 models. For VGGI1, both the client and server models are trained using SGD with a learning rate 0.01 and momentum of 0.9, and all the other settings including hyperparameters are kept identical to those in the main paper. We implemented VGG11 For ResNet-50, all the settings including optimizer and Provably Near-Optimal Federated Ensemble Distillation with Negligible Overhead
80
FedAWG
80
FedDF
FedDF
FedProx
FedGKD
Loss of ensembled model
12
FedGKD
SCAFFOLD
Ensemble accurary
DaFKD
10
(Ours)
FedGO (Ours)
50
FedDF
8
FedGKD
DaFKD
30
FedGO (Gurs)
40
20
30
100
60
100
0
60
100
Communication round
Communication round
Communication round
(a) Server test accuracy
(h) Ensemble test accuracy
(c) Ensemble test loss
Figure 7. Server test accuracy (%), test accuracy of the ensemble model (%), and test loss of the ensemble model of our FedGO and baselines for 100 clients CIFAR-I0 dataset with 0.05.
To support our hypothesis, we show the minimum of mean distillation loss for five different random seeds during 100 round of communication rounds in Table 6. The distillation loss increases progressively from CIFAR-10 to CIFAR- to ImageNet 100. In addition, the distillation loss is higher for a = 0.05 than for a = 0.1, which explains why the gap from the central training larger for a = 0.05.
Table Minimum mean distillation loss FedGO on three image datasets with a 0.1, 0.05.
Dataset
CIFAR-10
CIFAR-100
ImageNet100
0.1
0.175
0.237
0.363
0.05
0.266
0.348
0.539
F.3. Ensemble Distillation with a Different Server Dataset
Our theoretical justification constituting an optimal ensemble Corollary 3.3 allows heterogeneity between the server data distribution P. and the client average distribution p. demonstrate the effectiveness FedGO when p. * p which makes more sense in practice, we report the results when clients have the half of the CIFAR-10 10 dataset and the server has the half of the CIFAR-100 100 (unlabeled) dataset, in Table 7. The experimental results demonstrate that ensemble distillation even with heterogeneous server dataset is helpful in improving the performance. Furthermore, by employing optimal model ensemble, our FedGO algorithm, with theoretical performance guarantee, shows improvement over FedDF and DaFKD.
Table 7. Server test accuracy (%) and ensemble test accuracy (%) of our FedGO and baselines with heterogeneous server dataset: CIFAR-I0 for client dataset and CIFAR-10 server's unlabeled dataset.
FedAVG
FedDF
DaFKD
FedGO (ours)
Server test accuracy Ensemble test accuracy
58.6545.75
59.89
60.84:2.65
60.92.11.95
0.1
62.62:1.900
63.88 + 2.02
64.23
Server test accuracy Ensemble test accuracy
46.61
49.21
52.31
52.89-3.447
0.05
56.06-1207
59.30 +1.33
60.43 +0.56
E4. Results with Alternative Model Architectures
based on https://github. https://ithub..om//cennggaaa www.e//wwnn.nggggggggooooooogooe
22

---

Provably Near-Optimg Federated Ensemble Distillation with Negligible Overhead
Ensemble test accuracy (%)
70
70
Ensemble test
60
60
FedDF
Fed
50
FedGKD
50
FedGKD
DaFKD
DaFKD
40
FedGO (Ours)
40
FedGO (Ours)
0
40
80
100
20
60 Communication round
100
Communication round
(a) CIFAR-10 10 with a 0.1
(b) CIFAR-I0 R-10 with a 0.05
45
Ensemble test accuracy (%)
Ensemble test accuracy (%)
40
35
35
30
30
25
FedDF
25
FedDF
FedGKD
FedGKD
20
DaFKD
20
DaFKD
FedGO (Ours)
FedGO (Ours)
150
15 0
20
40
60
80
100
20
40
60
80
100
Communication round
Communication round
(c) CIFAR- with a 0.1
(d) CIFAR- 100 with 0.05
Ensemble test (%6
Ensemble test accuracy (%)
35
35
25
FedDF
FedDF
FedGKD
FedGKD
20
DaFKD
20
DaFKD
FedGO (Ours)
FedGO (Ours)
150
15
20
40
60
80
100
20
40
60
80
100
Communication round
Communication round
(e) ImageNet100 with 0.1
ImageNet100 with 0.05
Figure 8, Ensemble test accuracy (%) of FedGO and basclines over communication rounds on three image datascts with a = 0.1,
hyperparameters are set to the same as the main paper. Table 8 presents the server test accuracy of FedGO and baseline algorithms with the aforementioned model structures on CIFAR-10 with 0.1 after 100 communication rounds.
We can see that our FedGO algorithm consistently achieves performance gains over FedDF and FedGKD + across different
model structures.
23

---

Provably Near-Optimal Federated Ensemble Distillation with Negligible Overhead
Table 8. Server test accuracy (%) of central training, FedDF, FedGKD* and FedGO on CIFAR-I0 with 0.1 after 100 communication rounds, when utilizing VGGI1 and ResNet-50.
VGG11
ResNet-50
Central training
83.27 + 0.60
85.12 + 0.44
FedDF
68.59 + 4.65
65.21 + 4.62
FedGKD + FedGO (ours)
67.81 3.60
66.21 3.01
72.53 4.10
75.52 4.30
F55. Data-free FedGO
In practice, the server may have no extra dataset. In this case, we first prepare a generator and then generate a distillation dataset using the generator. The generator can either be an off off-the-shelf pretrained model or trained through an FL approach (Rasouli al., 2020; Guerraoui al., 2020; Li al., 2022; Wang et al., 2023c; Fan & Liu, 2020; Behera et al., 2022; Hardy et al., 2019; Xiong et al., 2023; Zhang et al., 2021; 2023b), corresponding the 3rd and 4th scenarios in Table 2 in our main paper, respectively.
Figure 9 and Table present the results for the two data-free approaches with 100 clients on CIFAR-10 dataset. We employed styleGAN (Karras et al., 2019) pretrained with ImageNet dataset for the off-the-shelf generator, and applied the FedGAN algorithm (Rasouli et al., 2020) for training generator. For both approaches, our FedGO shows performance gains in server test accuracy, ensemble test accuracy, and ensemble test loss compared to the baseline algorithms, including the uniform weighting of FedDF (Lin et al., 2020) and the domain-aware weighting of DaFKI) (Wang et al., 2023a). Moreover, observe that ensemble test accuracy is higher than server test accuracy, which can be attributed to the increased discrepancy between the distillation dataset distribution and the average client distribution p. This discrepancy amplifies the gap between the ensemble model's loss and the server model's 's loss, as analyzed in Theorem 3.2 of the main paper.
both data-free approaches, we have P, = Pg, under which our weighting method optimal for Va E Ps Py from Theorem 3.6 in our main paper. Note that the distance between and Pg = Ps for the generator trained from FedGAN smaller than that for the off-the-shelf generator. Consequently, despite using a simpler generator trained on smaller dataset, observe that the performance of FedGO using the generator trained from FedGAN is slightly better than that using the off-the-shelff generator.
Finally, we can observe performance degradation compared to the case where the distillation is performed on real dataset. This can attributed to the naive reuse generated images, which has been identified as a cause of performance degradation (Yoon et al., 2024; Wang et al., 2024). An interesting future work would be on improving the performance of knowledge distillation using generated images. Still, experimental results demonstrate that ensemble distillation beneficial in improving performance even with generated images. Furthermore, by employing an optimal model ensemble, our FedGO shows improvement over FedDF and DaFKD.
12
Server accuracy (95)
50
80
accuracy
Ensemble test loss
45
8
30
40
FedDF
20
Da KD
FedDF
FedDF
FedGO (Ours)
DaFKD
DaFKO
10
FedAVG
FedGO
FedGO (Ours)
20
40
25g
20
40
50
0
10
20
40
50
Communication round
Communication round
Communication round
(a) Server test accuracy
(b) Ensemble test accuracy
(c) Ensemble test loss
Figure Test accuracy of server model (%). ensemble accuracy (%), test loss ensemble model of data-free FedGO an off- shels generator (the case (G2)+(D2) of Table 2) and baselines with 100 clients on the CIFAR-10 - dataset with = 0.05.
24

---

Provably Near-Optimal Federated Ensemble Distillation with Negligible Overhead
Table Test accuracy of model (%), ensemble accuracy (%), and test loss of ensemble model of the data-free FedGO 100 communication round, with generator trained from FedGAN (the case (G3)+D2) of Table 2) and baselines with 100 clients on the CIFAR-I0 dataset with a 0.05.
Method
FedAVG
FedProx
SCAFFOLD
FedDisco
FedDF
DaFKI
FedGO (ours)
Server Test Accuracy
18.12:6..00
10.02
16.42+4.669
21.11=7.33
25.81
23.71=t4.99
27.26+t2.32
Ensemble Test Accuracy
37.0115.22
35.07=1.400
38.84-12.45
Ensemble Test Loss
1.76:10.09
1.84:0.15
1.69=0.06
F.6. Impact of Server Model Hyperparameters on Performance
F.6.1. AMOUNT OF UNLABELED DATA
Figure 10 shows the test accuracy of the server model and the test accuracy of the ensemble model during the training process for our FedGO algorithm. We conducted experiments by reducing the server dataset size 50% and 20% of the size assumed in our main CIFAR- 10 experiments. For these experiments, the server epochs were adjusted to ensure the same number of gradient steps: doubled for 50% and quintupled for 20%, while keeping other hyperparameters the same.
Figure 10 demonstrates that when the server dataset size decreases, the test accuracy of the ensemble model remains nearly consistent, while that of the server model decreases. This suggests that even with pseudo-labels of similar quality, the performance the server model can decline as the server dataset size decreases. This can be interpreted as the server model becoming more prone to overfitting as the distillation dataset becomes smaller (Hinton, 2015). Note that FedGO has the performance improvement of about 15% over FedAVG even with only 20% of the dataset, which corresponds 20% of the total client dataset size.
80
Ensemble test acuraay (%)
80
best accuracy
70-
70
60
60
50-
20%
40-
50%
50%
100%
100%
30
30
60
100
20
40
100
Communication round
Communication round
(a) Server test accuracy
(b) Ensemble test accuracy
Figure 10. Server test accuracy (%) and ensemble test accuracy (%) of our FedGO on the CIFAR- 10 dataset with a = 0.05, according to the size of the unlabeled dataset the server. In the legend, X % means that the size of the unlabeled dataset at the server reduced to X% of the size assumed in our main CIFAR-10 setting.
F.6.2. SERVER MODEL TRAINING EPOCHS
Table 10 shows the impact server model training epochs on FedGO's performance on CIFAR-10 with a = 0.1 after 100 communication rounds. Using epochs outperforms epoch, with minimal performance differences beyond epochs. Notably, even with only epoch, FedGO significantly outperforms all the baselines trained with 10 server epochs in Table 3.
Table 10. Server test accuracy (%) and ensemble test accuracy (%) of FedGO on CIFAR- 10 with = 0.1 after 100 communication rounds, according to the number of server model training epochs.
Epoch
5
20
Server Test Accuracy Ensemble Test Accuracy
7..43+6..4 5..36+ 80.9710.87
79.62++4.36 81.56-10.48
78.32-+-.13
77.16
81.39-10.75
25

---

Provably Near-Optimal Federated Ensemble Distillation with Negligible Overhead
F.6.3. SERVER MODEL LEARNING RATE DECAY
In the main paper, we used cosine learning rate decay by following the experimental setting of FedDF. As shown in Table 11, the absence of learning rate decay results in further performance improvement. Specifically, ensemble test accuracy of 85.20% is achieved, which is comparable to the central training model's accuracy of 85.33%, demonstrating the effectiveness of our provably near-optimal weighting method.
Table Server test accuracy (%) and ensemble test accuracy (%) FedGO on CIFAR-10 with 0.1 after 100 communication rounds, with and without learning rate decay during server model training.
FedGO
with LR decay
without LR decay
Server Test Accuracy Ensemble Test Accuracy
79.62:14.36 81.56110.48
80.18 85.20 1.33
F.7. Impact of Generator and Discriminator Quality on Performance
F.7.1. GENERATOR TRAINING STEPS
Table 12 shows the performance of our FedGO with varying generator training steps (100,000 in the main setup) alongside baseline algorithms after 50 communication rounds, while keeping all other settings unchanged from the main setup. FedGO with the generator trained for 25,000 steps performs better than that with the randomly initialized generator (0 steps), with little performance improvement beyond 25,000 steps. Remarkably, even randomly initialized generator outperforms FedDF with uniform weighting and achieves performance comparable to DaFKD with generator trained for 100,000 steps.
Table 12. Server accuracy (%) and ensemble accuracy (%) of FedGO CIFAR-10 with 0.1 after 50 communication rounds, according to the number of generator training steps.
FedDF
DaFKD
FedGO (ours)
Generator Training Steps
100,000
0
25,000
50,000
75,000
100,000
Server Test Accuracy
70.18 + 2.56
71.42 +: 3.11
71.12 +: 2.07
76.74 :: 3,16
78.43 + 0.99
78.89 +: 1.55
78.24 1: 1.61
Ensemble Test Accuracy
73.55 + 2.41
74.54 2.80
74.88 1.63
79.12 + 1.97
80.72 + 0.75
80.87 0.98
80.82 + 0.82
F.7.2. DISCRIMINATOR TRAINING EPOCHS
Table 13 shows the final performance of the FedGO algorithm for different numbers of discriminator training epochs on CIFAR-10 10 with 0.05. It can be seen that training the discriminator more times results in better final performance. Additionally, we note that among the baselines in Table and Figure 2, except DaFKD which originally trains the generator and discriminators at each round, the highest performance is achieved by the variance weighting method, with the test accuracy of 17.51110.77% indicating that there is a performance gain from the FedGO algorithm with just 5 epochs of discriminator training.
Table 13. Server test accuracy (%) of FedGO CIFAR-10 10 with 0.05 at the 100-th communication round, according to the number discriminator training epachs at the clients.
Epoch
5
10
30
50
Accuracy
63.96:19.03
71.38+7.76
70.84+8.88
72.35-19.01
76.92-1-5008
F.7.3. DISCRIMINATOR ARCHITECTURES
Table 14 presents the number of parameters, the number FLOPs required for the forward computation, and the performance of FedGO on CIFAR-10 10 with a = 0.1 at the 100-th communication round, when the following three different client discriminator structures are used:
26

---

Provably Near-Optimal Federated Ensemble Distillation with Negligible Overhead
CNN: The baseline architecture used in the main setting. It consists of four convolutional layers.
CNN+MLP: A variation of the CNN architecture, where the last two convolutional layers in the CNN are replaced by single multi-layer perceptron (MLP) layer, resulting in three-layer shallow network.
ResNet: deeper architecture based on ResNet-8, an 8-layer residual network.
Table 14. Server test accuracy (%) of FedGO CIFAR-10 with 0.1 the 100-th communication round along with the number of parameters and the number of FLOPs for the forward computation, according to different client discriminator structures.
FedGO
Discriminator Structure
CNN
CNN+MLP
ResNet
Number of Parameters
662,528
142,336
1,230,528
FLOPs
17.6 MFLOPs
9.18 MFLOPs
51.1 MFLOPs
Server Test Accuracy
79.62=4.33
79.71
78.73:1.003
Table 14 shows almost identical performances regardless of client discriminator architectures, demonstrating the robustness of FedGO to the discriminator architecture. In particular, the CNN+MLP discriminator, which has less than quarter of the parameters and around the half of the FLOPs compared to the original CNN structure, achieves similar performance.
E88 Impact of Byzantine Clients
We have conducted additional experiments in which 5 and 10 out of 20 clients were Byzantine, outputting only the maximum value for the discriminator. As shown in Table 15, the classification accuracy on CIFAR-10 without any Byzantine clients was 72.35% with a standard deviation of 9.01. When 5 clients were Byzantine, the accuracy dropped to 69.75%, and further decreased to 66.38% when 10 clients were Byzantine. Despite this challenging scenario where up to half the clients were compromised, our method still significantly outperformed all baseline approaches that do not incorporate discriminator, demonstrating its strong robustness against adversarial behavior.
Table 15. Server test accuracy (%) of FedGO CIFAR-I0 with 0.05 the 100-th communication round along with the number of Byzantine clients.
Byzantine Clients
Accuracy (%)
0
72.35+9.01
5
69.75+5.005
10
66.38+14.97
G. Comprehensive Analysis of Communication, Privacy, and Computational Complexity
Let us provide a detailed explanation for Table 2. If the server dataset is available from the outset (first two rows in Table 2), we only need one-shot communication of generator (from the server to the clients) and discriminators (from the clients to the server). Hence, additional communication burden and client-side privacy leakage are negligible. In particular, for our experiments, the parameters of the ResNet-18 classifier are approximately 90MB when stored as a PyTorch state _dict. In comparison, the generator and discriminator models are 4.61MB and 2.53MB, respectively. Over 100 communication rounds, during which ResNet-18 is transmitted repeatedly, the additional communication burden introduced by FedGO is nearly negligible. However, the server dataset is used for distillation for each communication round, incurring negliggle privacy leakage the server side. If there is no server dataset (last two rows in Table 2), there is no additional privacy leakage on the server side. If we use pretrained generator instead (third row), additional communication burden, client-side privacy leakage and computational burden are negligible, but is challenging in general to secure an off- the- generator which generates data with a distribution similar to the client data distribution. To train a generator through FL (last row), multiple rounds of GAN exchanges between the server and clients are required; however, since our FedGO requires a
27

---

Provably Ncar-Optimal Federated Ensemble Distillation with Negligible Overhead
small number of GAN exchanges in pre-FL. compared to the number of model exchanges in main FL., the additional communication burden, client-side privacy leakage, and computational burden are still negligible.
In the following, we provide a quantitative analysis of additional privacy leakage of FedGO compared to FedAVG, and an explicit comparison of computational cost for FedGO and baselines.
G.1. Privacy Analysis
For privacy measure, we consider local differential privacy (LDP) (Dwork et al., 2006) which is widely accepted both in academia and industry. Note that when the data is provided times by independently applying an LDP mechanism with privacy budget e for each provision, the total privacy budget becomes ne from the parallel composition result (Dwork et al., 2014).
Let T denote the total number of communication rounds in the main-FL stage. For the case (G3) in Section 3.2, let T" denote the total number of communication rounds required train a GAN in the pre-Fl. stage, which is 1/20 of T in experiment. For simplicity, we assume that every client participates in FL for each communication round. Let EM; ED and Ea denote the privacy budgets of LDP mechanisms applied to the classifier, discriminator, generator sent from each client at each communication round, respectively. Let éạs and êg denote the privacy budgets of LDP mechanisms applied to the classifier and the generator sent from the server when the server uses its own dataset in case of (S1) for training the generator and for distillation, respectively.
Table 16 shows the client-side and the server-side total privacy leakage of FedAVG and FedGO under various scenarios. For FedAVG, each client provides the classifier T times, and hence the client-side total privacy leakage becomes T CM- Let us first analyze the additional client-side privacy leakage of FedGO under various scenarios. For FedGO with the method (GI)+(DI), (D)), or (G2)+D2), the client sends its discriminator only once, incurring extra privacy leakage of ED. which is negligible with large T. For FedGO with (D2), the clients need to send the discriminator and the generator for T' times to train a GAN in the pre-FL stage, leading to additional privacy leakage of T' (ED + cG); however, since T" only 1/20 of T our experiment, the additional privacy leakage is negligible. Next, server-side privacy issues arise only when the server has its own dataset. If the server trains the generator from its dataset and provides it to the clients for the case of (G1), it yields the privacy leakage of &g. In addition, if the server uses its dataset for distillation and applies an LDP mechanism with privacy budget ĚM to the classifier for each communication round for the case of (D1), it results in non-negligible amount of additional privacy leakage T Tê.-
We consider this as a future work direction: developing methods for training ensemble distillation while preserving LDP constraints in classifiers or discriminators.
Table 16. Quantitative analysis of the client-sid side and the server-side total privacy leakage of FedAVG and FedGO under various scenarios.
Client-side
Server-side
FedAVG
T EM
(GI)+(D))
T- +
êG + T T.M
(G1)+(D1)
T- cM +
T êM
FedGO
(G2)+(D2)
T T.eM tep
(D2)+(D2)
T T (ED + EG) + +T EM + E
G.2. Computational Cost Comparison
Table 17 shows the floating point operations (FLOPs) during CIFAR-10 training for the baselines and FedGO with the four scenarios described in Table 2. 1 MFLOP represents 10th FLOPs.
First, the client side, the computational cost for FedGO with (G1) or (G2) is comparable to that of FedAVG and FedDF, which only optimize the client's 's vanilla supervised loss. The cost is roughly half of the cost of FedGKD+, which includes a regularization term in the client objective. This reduction is because the client only needs to train the discriminator only once during the pre-FL stage. In each round, FedAVG and FedDF compute 4.17 MFLOPs per client update, whereas the computational cost for training a client's ' discriminator is 3.29c+7 MFLOPs less than the cost for one round of classifier training. The additional computation cost for FedGO with (G1) or (G2) is therefore minimal, especially considering its
28

---

Provably Near-Optimal Federated Ensemble Distillation with Negligible Overhead
fast convergence speed.² Note that the client-side computational cost of FedProx is same that of FedAVG because the proximal term computation is negligible. For FedGO with (G3), clients train the generator using an FL approach during the pre-FL stage. Although this process adds computational overhead to the client side, the number of FL rounds required for training the generator is relatively small-oni 1/20 of the rounds needed for the main FL stage in our experiment. Consequently, this results only a slight increase in the overall computational cost compared to FedAVG. In contrast, data-free FL algorithms like DaFKD (Wang et al., 2023a) exchange both the generator and model in every communication round, resulting in significant overhead associated with utilizing the generator when many rounds for model training are needed. By decoupling generator preparation from model training, FedGO with (G3)+(D2) avoids this issue, reducing the additional overhead.
Next, on the server side, in FedGO with (GI)+(DI), training a generator using server dataset involves significant additional computation compared to FedDF due to the 100,000 steps required for training ResNet-based generator and discriminator. However, given that federated learning typically involves a server with ample resources and clients with limited computational resources, this increase in server-side computation is more affordable in practice, compared to increasing the computational burden on clients. Furthermore, while the computation for training the generator irrelevant the number of clients, the computation required for pseudo-labeling scales linearly with the number of clients. Note that the total computational cost in Table 17 assumes 20 clients. In real-world scenarios, where 100+ clients may participate in FL, the relative proportion of the computational cost for training the generator will decrease.
Note that using an off-the-shelf generator reduces the additional server-side computational cost of FedGO. FedGO with (G2)+(DI) requires approximately 2% more computation than FedDF, but achieves a significant performance gain of about 13%p on CIFAR-10 10 with a = 0.05 Table 3. The reason why FedGO with (G2)+(D2) has a higher server-side computational cost compared to FedGO with (G2)+(D1) that FedGO with (G2)+(D2) generates distillation dataset using a heavy generator, StyleGAN.
FedGO with (G3)+(D2) also generates distillation dataset using a generator but the generator used here is lighter than StyleGAN generator used in (D2))(D2). The computational cost for the generation of distillation dataset in FedGO with (D3)+(D2) is 2. 1e+7 MFLOPs which is negligible compared to the computational cost for pseudo-labelling and ensemble distillation which is 5.07c+10 MFLOPs. On the other hand, note that the computational cost of FedGO with (G3)+(D2) is slightly lower than DaFKD while both train a generator using an FL approach. The reduction mainly comes from the difference that FedGO with (D2) generates a distillation dataset only once after the training of the generator, while DaFKD updates the distillation dataset in every communication round. Finally, note that the server-side computational cost of FedGO with (G3)+(D2) is comparable to the case (G2)+(DI), FL approach. This is because the server's ' role is limited to averaging the clients' generator and discriminator, incurring negligible additional computational cost on the server side.
though (G3) trains a generator through an
Table 17. The number of MFLOPs for training our FedGO and baselines on CIFAR-IO for 100 communication rounds.
FedGO (ours)
FedAVG
FedProx
FedDF
FedGKD
DaFKD
(GI)+DI))
(G2)+(D1)
(G2)+(D)
(G3)+(D2)
Client-side
3,33c+10
3.33c+10
3.33e+10
6.67c+10
8.81c+11
3.40c+10
3.40c+10
3.40c+10
3.52c+10
Server-side
7.82c+3
7.82c+3
5.00e+10
5.00e+10
5,28c+10
1.39c+11
5.07c+10
6.01e+10
5.07c+10
Total
3.33e+10
3.33e+10
8.33e+10
1.17c+11
9.34c+11
1.73c+11
8.47c+10
9.41c+10
8.59c+10
H. Limitation
Although our FedGO algorithm can be extended to model heterogeneous scenarios as in FedDF, we found challenging to define optimal model ensemble for multiple hypothesis classes. Consequently, it appears difficult to apply the results of Theorem 3.2 and Corollary 3.3 in such cases.
"We note that the computational exceeds 2%, rather than being below 1%, because in each round, only C 0.4 proportion of clients are sampled to participate federated learning, rather than full client participation.
29


Reviews:
[{"review_id": "TycmRDKHRf", "reviewer": "Reviewer_i9VB", "summary": "The paper presents FedGO, a novel federated ensemble distillation method, aimed at addressing client data heterogeneity in federated learning. The authors propose a weighting method for ensemble distillation that is provably near-optimal by leveraging theoretical results from GANs. The method trains client-side discriminators using a generator distributed from the server, which allows the server to assign optimal weights to client predictions when generating pseudo-labels for unlabeled server data. The paper establishes theoretical guarantees for the proposed weighting scheme and demonstrates its effectiveness through experiments on image classification datasets (CIFAR-10, CIFAR-100, ImageNet100). FedGO significantly outperforms existing baselines in terms of accuracy and convergence speed while maintaining negligible communication and computational overhead.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Comment 1:** *Important Limitation: The theoretical analysis — including the derivation of the optimal weighting functions and generalization bounds — is restricted to binary classification tasks. This limitation is underemphasized in the main text, yet all experiments are conducted on multi-class classification problems. While the empirical results are compelling, it remains unclear how well the theoretical results translate to the multi-class case. Clarifying or extending the theoretical framework to multi-class settings would strengthen the paper's claims considerably.*\n\nThank you for reviewing our paper. In the following, we have clarified the assumption of binary classification tasks in deriving the optimal weighting functions and generalization bounds.  \n\nThe optimality of our proposed weighting function, based on Theorem 3.4 and Theorem 3.6, is not restricted to binary classification tasks. Theorem 3.4 only requires the convexity of loss function and Theorem 3.6 does not require either the convexity of the loss function or binary classification tasks.\n\nOur generalization bound in Theorem C.1 indeed assumes binary classification tasks. However, we would like to highlight that obtaining a tight generalization bound is challenging even in binary classification tasks and remains an active area of research. For example, FedDF, Fed-ET, and DaFKD derived generalization bounds under binary classification assumptions, but their bounds either tend to degrade in data-heterogeneous settings or become vacuous in large-client scenarios. However, our bound is tighter than those bounds, making it a significant contribution even within the binary classification framework. For the multi-class case, there are some existing results on obtaining generalization bounds in a single-model (non-federated learning) setup, but they remain limited and loose.", "claims_and_evidence": "**Claim 1. Near-optimality of Proposed Weighting Scheme:** The authors theoretically justify their weighting method using GAN-based results and provide generalization bounds to support its optimality. \n\n**Claim 2. Performance improvements:** Experimental results provide strong empirical support that FedGO outperforms FedDF, DaFKD, and other baseline methods in terms of accuracy and convergence speed.\n\n**Important Limitation:** The theoretical analysis — including the derivation of the optimal weighting functions and generalization bounds — is restricted to binary classification tasks. This limitation is underemphasized in the main text, yet all experiments are conducted on multi-class classification problems. While the empirical results are compelling, it remains unclear how well the theoretical results translate to the multi-class case. Clarifying or extending the theoretical framework to multi-class settings would strengthen the paper's claims considerably.", "methods_and_evaluation_criteria": "The paper uses well-established benchmark datasets (CIFAR-10, CIFAR-100, ImageNet100) and evaluation metrics (test accuracy of server model and communication efficiency) for FL. Comparisons with state-of-the-art baselines (FedDF, FedGKD+, and DaFKD) are provided.", "theoretical_claims": "We reviewed the main theoretical claims, including Theorem 3.4 and Theorem 3.6, as well as the generalization bound in Theorem C.1. The derivations appear correct, though they were not carefully checked. Theorem 3.4 relies on the convexity of the loss function to invoke Jensen's inequality and uses knowledge of the true client distributions to derive the weighting function. Theorem 3.6 is a direct application of the standard GAN result from Goodfellow et al. (2014), mapping client-specific data densities to discriminator outputs via the odds function. The generalization bound in Theorem C.1 closely follows prior domain adaptation analyses.", "experimental_designs_or_analyses": "The experimental setup is well-designed, with multiple datasets, varying levels of data heterogeneity, and different FL configurations. The paper presents:\n\n- Performance comparisons across baselines\n- Convergence speed comparisons\n- Ablation studies on different weighting methods\n- Experiments with different generator settings (pretrained vs. scratch-trained)\n- Analysis of overhead in terms of communication, privacy, and computational cost\n\nThe results consistently demonstrate that FedGO achieves superior performance with faster convergence and negligible overhead.", "relation_to_broader_scientific_literature": "The paper builds on several key areas:\n\n- **Federated Learning:** It extends works such as FedAVG (McMahan et al., 2017) and FedDF (Lin et al., 2020) by improving model aggregation in heterogeneous settings.\n- **Ensemble Distillation:** Prior works like FedHKT (Deng et al., 2023) and DaFKD (Wang et al., 2023) explored ensemble distillation but lacked strong theoretical guarantees for weighting strategies.\n- **GANs:** The authors leverage insights from Goodfellow et al. (2014) on GAN discriminators, which is a novel contribution to FL.\n\nBy integrating ideas from these domains, FedGO represents a well-motivated and significant advancement in federated learning.", "essential_references_not_discussed": "The references are satisfactory.", "other_strengths_and_weaknesses": "The paper presents rigorous theoretical analyses with provable guarantees, supported by strong empirical results.", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lsE0DJfnEU", "reviewer": "Reviewer_LviH", "summary": "This paper, inspired by the theoretical results of Generative Adversarial Networks (GAN), proposes a weight assignment method for federated ensemble distillation. The method first trains the generator on the server side via a federated learning algorithm and trains the discriminator on the client side using a local dataset. Subsequently, the server assigns weights to each client based on the generated data samples (or unlabeled server datasets) and the outputs of the client-side discriminators to achieve near-optimal performance. The paper provides relevant theoretical proofs of the effectiveness of the method in a federated environment.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Comments on pre-trained generator robustness**\n\nOur theoretical analysis already takes into account such discrepancy: Theorem 3.6 says that our weighting method produces the optimal weight $w_k^*$ for the data point on supp$(p)\\cap$  supp$(p_g)$, where $p$ is the average client data distribution and  $p_g$ is the generator’s data distribution. Thus, as we mentioned in the last paragraph of Section 3.1, it is theoretically guaranteed that our proposed method works properly as long as the generator is capable of producing sufficiently diverse samples. However, empirically, we showed even stronger results in Appendix F.7.1: FedGO with a completely untrained generator still performs better than FedDF.\n\n**Comments on communication overhead**\n  \nWe confirm that the additional communication cost of our method is indeed negligible, because (1) the number of parameters for the generator and the discriminator is much smaller than that of the classifier architecture and (2) a small number of communication rounds is sufficient for training generator and discriminator, i.e., only 5 rounds for our experiment. More specifically, in Appendix G, we showed that FedGO incurs only an additional 0.47% in communication overhead even in the most challenging scenario where the generator needs to be federated, i.e., case (G3) + (D2). Note that for the scenario where the number of parameters for the generator and discriminator should be large due to the input dimension, the number of parameters for the classifier will also be large, thereby ensuring that the relative overhead remains negligible.\n\n**Comments on server-side resource consumption**\n   \nWe reported the computational complexity in terms of the number of MFLOPs (rather than runtime) in Appendix G.2. The cost of the data generation process is already included in the MFLOPs. In cases where there is no unlabeled dataset on the server, we generated 25,000 images using the generator before main FL stage. Once generated, this dataset is reused for distillation in every communication round. The image generation process takes only 3 seconds and is performed just once per experiment, making its time cost negligible.\n    \n**Comments on extreme data heterogeneity**\n   \nWe first would like to emphasize that our weighting method is proposed to address data heterogeneity, and our theoretical analysis already takes into account the discrepancy between the client data distribution and the generator distribution. To validate the effectiveness of our method in highly heterogeneous setting, we considered Dirichlet parameter $\\alpha = 0.05$, for which it is common for only two or three clients out of 20 clients to possess all the images of a particular class. The data distributions for $\\alpha = 0.1$ and $\\alpha = 0.05$ are visualized in Figures 5 and 6 in the Appendix E.2. Additionally, to account for more complex data scenarios, we conducted experiments using the ImageNet100 dataset. Since ImageNet is sufficiently complex, we believe these experiments offer meaningful insights into FedGO’s performance under realistic and challenging conditions.\n \n**Comments on theoretical guarantees under heterogeneity**\n    \nThe proposed method is designed to address data heterogeneity, and our theorems and their proofs already take into account possible imbalance or heterogeneity in the client data. For “model” heterogeneity, we assumed homogeneous model structures in Theorem 3.2, Corollary 3.3, and throughout all of our experiments. As discussed in Appendix H (Limitations), defining an optimal model ensemble becomes challenging when dealing with multiple hypothesis classes (synonym for multiple client model structures).\n    \n**Comments on experimental completeness**\n    - We appreciate this valuable suggestion. To address this concern, we have implemented and tested the following baseline algorithms targeting data heterogeneity (published in 2024) under the main experimental setting in Section 4.1:\n\n1. FedUV (CVPR 2024)\n2. FedTGP (AAAI 2024)\n        \nAs shown in the table below, FedGO outperforms both baselines across all settings. We will incorporate these results into the final version of our paper.\n\n\\\\begin{array}{|l|cc|cc|cc|}\n\\\\hline\n& \\text{CIFAR-10} && \\text{CIFAR-100} && \\text{ImageNet100} & \\\\\\\\\n& \\\\alpha = 0.1 & \\\\alpha = 0.05& \\\\alpha = 0.1 & \\\\alpha = 0.05& \\\\alpha = 0.1 & \\\\alpha = 0.05  \\\\\\\\\n\\\\hline\n\\text{FedUV} & 62.58 \\\\pm 4.83 & 53.80 \\\\pm 5.68 & 38.84 \\\\pm 0.79 & 36.17 \\\\pm 1.24 & 30.09 \\\\pm 1.09 & 27.32 \\\\pm 0.65 \\\\\\\\\n\\\\hline\n\\text{FedTGP} & 61.16 \\\\pm 6.98 & 61.51 \\\\pm 7.78 & 39.58 \\\\pm 0.10 & 36.56 \\\\pm 0.11 & 29.21 \\\\pm 1.13 & 26.34 \\\\pm 1.02 \\\\\\\\\n\\\\hline\n\\text{FedGO} & \\\\mathbf{79.62} \\\\pm 4.36 & \\\\mathbf{72.35} \\\\pm 9.01 & \\\\mathbf{44.66} \\\\pm 1.27 & \\\\mathbf{41.04} \\\\pm 0.99 & \\\\mathbf{34.20} \\\\pm 0.71 & \\\\mathbf{31.70} \\\\pm 1.55 \\\\\\\\\n\\\\hline\n\\\\end{array}", "claims_and_evidence": "This paper proposes a provably near-optimal weighting method that utilizes client discriminators, which are trained using a server-distributed generator and local datasets. I believe that the claims made in the submission are well supported by evidence, but need further refinement (see comments)\n\nComments：\n\n1．\tThe authors claim that the communication burden of the steps of distributing the generators to the clients and uploading the discriminators from the individual clients to the server side is negligible, and that if the number of generators and discriminators parameterized is large, the communication cost of a single transmission is non-negligible. If the generators need to be federated for training, multiple rounds of communication are required, so there is a big doubt that the introduction of additional communication overhead is negligible.\n\n2．\tHas server-side resource consumption time (GPU hours) been considered in the data generation process?\n\n3．\tIn the case of highly heterogeneous or long-tailed distributions, the generator may not be able to accurately estimate the distribution of data among clients, and the computation of weights relies on the distribution of client data, which may or may not lead to a weight distribution that is biased toward a small number of clients, and it is suggested that the authors use more complex data to validate the validity of the weight distribution method.\n\n4．\tWhen there is extreme imbalance or heterogeneity in the client data, the authors are requested to provide a rigorous mathematical proof and the conditions under which this conclusion holds true regarding that the expected loss of the ensemble model does not exceed the minimum possible loss of the single model, and also whether such a conclusion can still be given in the presence of model heterogeneity.\n\n5．\tTo further enhance the persuasiveness and completeness of the experiment, it is suggested that the authors add 2-3 relevant papers published in 2024 as baseline, on federated learning under data heterogeneity and federated ensemble learning.", "methods_and_evaluation_criteria": "This paper proposes a federated ensemble distillation method (FedGO) based on generative adversarial network (GAN) theory to generate pseudo-labels by dynamically assigning optimal weights through a client-side discriminator to ensure the accuracy of collaborative learning, but further improvements are still required.", "theoretical_claims": "I carefully reviewed the theoretical claims presented in the paper, including the proofs provided in the main text and the appendix. I specifically verified the correctness of Theorems 3.2, 3.4, and C.1, as well as the supporting lemmas and intermediate steps. The authors have also provided clear explanations and references to existing theoretical results, which further support the validity of their claims.", "experimental_designs_or_analyses": "To further enhance the persuasiveness and completeness of the experiment, it is suggested that the authors add 2-3 relevant papers published in 2024 as baseline, on federated learning under data heterogeneity and federated ensemble learning.", "relation_to_broader_scientific_literature": "The paper makes a meaningful contribution to the field of federated learning (FL) and ensemble distillation by addressing client heterogeneity through a theoretically grounded weighting method inspired by GAN theory. It builds upon and extends prior works on federated distillation and ensemble learning, such as FedDF and FedGKD+, by introducing a provably near-optimal weighting strategy. The paper's experimental validation on benchmark datasets, along with its comparison to existing methods, further demonstrates its relevance and contribution to the existing literature.", "essential_references_not_discussed": "The paper provides a thorough and comprehensive review of the relevant literature, citing the necessary and appropriate prior works that form the foundation of the study. The key contributions are contextualized with proper references to prior results in federated learning, ensemble distillation, and relevant theoretical frameworks. I did not identify any missing references that are critical for understanding the paper or its contributions.", "other_strengths_and_weaknesses": "Strength：\n\n1.\tThe FedGO algorithm introduces a novel weighting method that utilizes client-trained discriminators to weight the ensemble models, which are trained based on data generated by the server generator. This approach enables more efficient model integration in the presence of heterogeneous data on the client side, thus improving the overall performance.\n\n2.\tThis paper experimentally demonstrates that the proposed method has significant improvements over existing studies in terms of final performance and convergence speed on multiple image datasets.\nWeakness:\n\nIt is necessary to further discuss the limitations of the proposed method in the case of extreme heterogeneity. It would also be valuable to elaborate on the robustness of the method when the pre-trained generator fails to adequately fit the customer data distribution, and to clarify whether the theoretical guarantees still hold in such cases.", "other_comments_or_suggestions": "Didn't find the obvious typo problem in the paper.", "questions_for_authors": "I'll reorganize all my queries here\n\n1.\tPre-trained Generator Robustness : How robust is the proposed method when a pre-trained generator fails to accurately fit the client data distribution? Under such circumstances, do the theoretical guarantees you provided still hold, or are there modifications needed to account for the discrepancy? Understanding this aspect is crucial for evaluating the method's reliability and generalizability when the underlying assumptions about the generator are not fully met.\n\n2.\tThe authors claim that the communication burden of the steps of distributing the generators to the clients and uploading the discriminators from the individual clients to the server side is negligible, and that if the number of generators and discriminators parameterized is large, the communication cost of a single transmission is non-negligible. If the generators need to be federated for training, multiple rounds of communication are required, so there is a big doubt that the introduction of additional communication overhead is negligible.\n\n3.\tHas server-side resource consumption time (GPU hours) been considered in the data generation process?\n\n4.\tIn the case of highly heterogeneous or long-tailed distributions, the generator may not be able to accurately estimate the distribution of data among clients, and the computation of weights relies on the distribution of client data, which may or may not lead to a weight distribution that is biased toward a small number of clients, and it is suggested that the authors use more complex data to validate the validity of the weight distribution method. Could you provide further insights on the limitations of your proposed method in scenarios with extreme client data heterogeneity? In such settings, what potential pitfalls might arise, and how does the performance of your method degrade? Clarification on this point would help assess the applicability of your method in more challenging, real-world scenarios.\n\n5.\tWhen there is extreme imbalance or heterogeneity in the client data, the authors are requested to provide a rigorous mathematical proof and the conditions under which this conclusion holds true regarding that the expected loss of the ensemble model does not exceed the minimum possible loss of the single model, and also whether such a conclusion can still be given in the presence of model heterogeneity.\n\n6.\tTo further enhance the persuasiveness and completeness of the experiment, it is suggested that the authors add 2-3 relevant papers published in 2024 as baseline, on federated learning under data heterogeneity and federated ensemble learning.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "US7AjGcRUb", "reviewer": "Reviewer_PA82", "summary": "FedGO, a method for federated ensemble distillation (FED) that optimally assigns weights to client predictions using client-trained discriminators, is theoretically justified by GAN principles. It mitigates client data heterogeneity. Experiments on image classification datasets show FedGO outperforms existing approaches in accuracy and convergence speed.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Comments on data-free setting**\n    \n We believe we have thoroughly examined both scenarios, with and without a server dataset.  For data-free setting, we have conducted experiments using both off-the-shelf generator and generator trained via federated learning and analyzed the results in Appendix F.5. Also, we have analyzed the communication, privacy, and computational complexity of data-free FedGO in Appendix G. \n    \n**Comments on security risks**\n    \nIn accordance with the reviewer’s comment, we have conducted additional experiments where 5 and 10 out of 20 clients were Byzantine, outputting only the maximum value for the discriminator. The results showed that while the accuracy on the CIFAR-10 classification task under $\\alpha=0.05$ was initially 72.35$\\pm$9.01, it dropped to 69.75$\\pm$5.05 with 5 Byzantine clients and 66.38$\\pm$4.97 with 10 Byzantine clients. Even in this extreme scenario where half of the participants were Byzantine, our method significantly outperformed all the baselines that did not utilize a discriminator. We will report this result in the final paper. \n\n**Comments on theoretical assumptions**\n    \nAs shown in the experimental results in Appendix F.1 and Appendix F.5, FedGO outperforms existing baselines even when clients have limited data. Specifically, the experiments were conducted with 100 clients, each having an average of only 250 data samples—an amount that is insufficient for training an optimal discriminator. \n    \n**Comments on generator assumptions**\n    \nFor the case (G3) where a generator and a discriminator are trained using an FL approach, we have already addressed the reviewer’s concern by conducting experiments with a small number of training samples per client (250 images per client when there are 100 clients) in Appendix F.5 and by showing that additional client-side computational overhead is negligible compared to FedDF, which does not train a generator and a discriminator. \n    \nFor the case (G2) where an off-the-shelf generator is used, we have also conducted experiments where the distribution of the generator differs from that of the client data. These results are presented in Table 5 of Section 4.2. Specifically, when the off-the-shelf generator was trained on ImageNet and the client datasets were CIFAR-10, CIFAR-100, or ImageNet100, the performance remained comparable to the case where the generator was trained on data matching the client distribution. This indicates that even when the off-the-shelf generator is trained on a different dataset from clients’ data, clients can still train their discriminators effectively.\n    \nWe hope our response addresses your concerns. If we have misunderstood your question, please feel free to clarify.\n\n**Comments on experimental design**\nThe simplest weighting method for ensemble distillation would be the uniform weighting that FedDF incorporates. The paper demonstrates that the improvement from FedAVG to FedDF stems from the benefit of ensemble distillation itself, while the improvement from FedDF to FedGO is attributed to GAN-based weighting. Additionally, extensive experiments were conducted by fixing the ensemble distillation process while varying the weighting method, effectively quantifying the contribution of weighting in Figure 2 of our main paper.\n\n**Comments on deployment challenges**\n- Latency: We provided a comparison of the MFLOP counts between the baseline and FedGO algorithms in Table 16 of Appendix G. The comparison of MFLOP counts can serve as a proxy for latency comparison.\n- Scalability: We have already provided experimental results in Appendix F.1 and F.5 with a large-scale setup of 100 clients under various settings.\n- Potential biases in GAN-based weighting: In accordance with the reviewer’s comment, we have additionally conducted experiments with malicious clients and demonstrated the effectiveness of our weighting method in such a challenging scenario. We have reported the experimental results in the response to the second comment.\n- Generalization to non-image tasks: In accordance with the reviewer’s suggestion, we have additionally conducted experiments with a tabular healthcare dataset, confirming performance improvements over FedAVG and FedDF as shown in the table below. In this experiment, we used a total of four clients, all of whom participated in every communication round. Regarding NLP, FedDF and some GAN-based approaches have demonstrated promising results, indicating the strong generalization potential of our weighting method. This appears to be an interesting direction for future research, and we appreciate the suggestion.\n\n\\\\begin{array}{|l|cc|cc|cc|}\n\\\\hline\n& \\\\alpha = 0.1 & \\\\alpha = 0.05  \\\\\\\\\n\\\\hline\n\\text{Central training} & 36.21\\\\pm0.15 \\\\\\\\\n\\\\hline\n\\text{FedAVG} & 34.20\\\\pm0.56 & 33.82\\\\pm0.86  \\\\\\\\\n\\\\hline\n\\text{FedDF} & 34.66\\\\pm0.22 & 34.21\\\\pm0.46  \\\\\\\\\n\\\\hline\n\\text{FedGO} & \\textbf{34.81}\\\\pm 0.36 & \\textbf{34.64}\\\\pm0.32  \\\\\\\\\n\\\\hline\n\\\\end{array}", "claims_and_evidence": "The majority of the claims in the paper are well-supported by theoretical proofs and empirical results, but there are a few areas where the evidence is either incomplete or lacks robustness. Below, I assess the validity of key claims and identify potential issues.\n- While the paper does test both cases (with and without a server dataset), the results for the data-free setting are not extensively analyzed.\n- If a malicious client manipulates its discriminator outputs, can it bias the ensemble weights? The authors do not address this risk.\n- While Theorem 3.6 is theoretically sound, it assumes the existence of an optimal discriminator for each client. However, in practice, Clients may not have sufficient data to train an optimal discriminator.", "methods_and_evaluation_criteria": "+ FedGO consistently outperforms existing methods (e.g., FedDF, FedGKD+, DaFKD) in accuracy and convergence speed across multiple datasets and data heterogeneity settings.\n\n+ FedGO introduces minimal computational, communication, and privacy costs for clients, which is crucial for federated learning in practical settings.\n\n- The method assumes that a generator can be pretrained or trained collaboratively, but in many real-world FL scenarios, clients might lack sufficient training data or computational power to train discriminators efficiently. The use of off-the-shelf generators is promising, but their effectiveness for out-of-distribution client data needs more validation.\n\n- The experiments are well-structured, but the paper does not isolate the contributions of different aspects of FedGO (e.g., GAN-based weighting vs. ensemble distillation itself). Would a simpler weighting heuristic perform nearly as well?", "theoretical_claims": "The paper provides a strong theoretical foundation, proving that the proposed weighting scheme leads to near-optimal ensemble model performance. The authors use results from GAN discriminator theory to derive optimal weight assignment for ensemble learning.\n\nWhile the theoretical analysis is strong, the paper does not discuss practical deployment challenges, such as latency, scalability for larger client populations, or potential biases in GAN-based weighting. In addition, I am also curious how well this method would generalize to non-image tasks (e.g., NLP, healthcare, or IoT applications).", "experimental_designs_or_analyses": "The method assumes that a generator can be pretrained or trained collaboratively, but in many real-world FL scenarios, clients might lack sufficient training data or computational power to train discriminators efficiently. The use of off-the-shelf generators is promising, but their effectiveness for out-of-distribution client data needs more validation.", "relation_to_broader_scientific_literature": "The literature study is comprehensive.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QH88IPT4jY", "reviewer": "Reviewer_9rmt", "summary": "This paper presents a near-optimal and practical client weighting method that leverages client discriminators trained with a server-distributed generator and local datasets in federated ensemble distillation, which are supported by rigorous theoretical analysis and experimental validation. The work has significant research value and application potential in the fields of federated learning and distributed machine learning.\n\n## update after rebuttal\nAuthors answer my questions, and I increse 1 score after reading the rebuttal.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Comments on theoretical assumptions**\n\nWe believe the reviewer's suggestion regarding $L$-smoothness may arise from a different interpretation of the convexity assumption. The reviewer seems to have interpreted the convexity assumption with respect to the model parameter $\\theta$, whereas our convexity assumption is in fact with respect to the model output $\\hat{y}$.  \n\nWhile deep neural networks like ResNet-18 do not exhibit convexity with respect to $\\theta$, our theoretical results do not rely on such a convexity with respect to $\\theta$. Instead, we assume the convexity of the loss function with respect to $\\hat{y}$, which is standard in the literature. We acknowledge that our explanation may have caused some confusion, and we will revise the statement in our paper more clearly. \n\n**Comments on GAN-based assumptions**\n\nWe agree that, to theoretically guarantee the achievement of optimal single-model performance, the generator and discriminator must be properly trained. However, our experimental results in Appendix F.7 demonstrate that the proposed method is robust to the quality of the generator and discriminator. In particular, in Appendix F.7.1, we showed that FedGO even with a completely untrained generator outperforms FedDF in terms of both server test accuracy and ensemble test accuracy. Also, in Appendix F.7.2, we can see that FedGO still significantly outperforms baselines even when client discriminators were trained at only one-sixth of the main setting.  \n\nNext, regarding privacy leakage due to the provision of the discriminator, we have conducted a comprehensive privacy leakage analysis in Appendix G. Table 15 demonstrates that by incorporating local differential privacy (LDP), FedGO can guarantee a client-side privacy level comparable to FedAVG. Furthermore, as a measure to prevent excessive client distribution leakage, we adopted a simple four-layer CNN for the discriminator, as detailed in Appendix E.2 and F.7.3, and implemented the output activation using a double composite sigmoid function, restricting the discriminator’s output range to [sigmoid(0), sigmoid(1)]. \n\n**Comments on client-side data limitations**\n\nOur experimental results demonstrate that, contrary to the reviewer's concern, FedGO is effective even with low-volume client datasets. We present experimental results not only for 20 clients but also for 100 clients in Appendix F, where each client has an average of only 250 data samples. Figure 7 shows that FedGO achieves significant performance improvements over existing baselines even in such a client data-deficient situation. \n\n**Comments on FedGO and data heterogeneity**\n\nFederated ensemble distillation algorithms leverage additional unlabeled datasets at the server to perform pseudo-labeling and knowledge distillation, thereby enhancing server model performance. However, in client data heterogeneous situations where client distributions are highly diverse, inference quality for server unlabeled data $x$ varies significantly across clients. Using a fixed weighting function (e.g., uniform weighting) for pseudo-labeling can degrade the quality of pseudo-labels in proportion to the average discrepancy between the client average distribution $p$ and each individual client distribution $p_k$ (as summarized in Table 1 of our paper, specifically (1.1)).\n\nThus, research has focused on developing weighting methods that assign higher weights to clients with higher inference quality per data sample $x$. DaFKD is one such method. However, in large-client scenarios, its generalization bound become vacuous, and there has been little research on weighting methods that are theoretically guaranteed and robust to client data heterogeneity.\n\nIn this paper, we demonstrate in Theorem 3.4 that assigning weights in a specific manner ensures that pseudo-label performance remains independent of client data heterogeneity while providing the tightest existing generalization bound (as discussed in Table 1, specifically (1.3), and Definition 3.1). Furthermore, we show that the generalization bound for the server model trained on these pseudo-labels is expressed in terms independent of client data heterogeneity, proving that our ensemble distillation scheme is theoretically robust to client data heterogeneity.\n\nWe modeled this weighting function through client discriminators, as presented in Theorem 3.6, and implemented it via FedGO. Our experimental results across various settings confirm that FedGO is significantly more robust to client data heterogeneity compared to baseline algorithms.\n\nWe hope that this answers your question. We will emphasize this aspect further in the paper to enhance clarity.", "claims_and_evidence": "Yes. This paper provides the theoretical anylysis and experiments.\n\nThe paper provides strict mathematical derivations and theoretical analyses, such as the proofs of Theorem 3.4 and Theorem 3.6, which validate the correctness and optimality of the method. This theoretical rigor lays a solid foundation for practical applications of the method and demonstrates its high credibility and general applicability in addressing real-world problems.\n\nAlthough the paper provides reliable theoretical guarantees for their weighting method, there are still some flaws in theory:\nTheorem 3.4 assumes convex functions, which does not align with the experimental setting(ResNet-18) of the paper. Perhaps the authors should consider a more practical assumption under L-smoothness?", "methods_and_evaluation_criteria": "Yes.\n\nBut the theoretical assumptions are overly idealized; the discriminator in GAN must accurately estimate the proportion of data distributions, requiring effective training of both generator G and discriminator Dk. Furthermore, if the GAN discriminator becomes too accurate, it may lead to unnecessary privacy breaches.", "theoretical_claims": "I've not checked proofs in appendix.", "experimental_designs_or_analyses": "Yes. \n\nThe experimental setting, models and baselines are soundness.", "relation_to_broader_scientific_literature": "This paper has a close relation to federated learning with client heterogeneity.", "essential_references_not_discussed": "Not found yet.", "other_strengths_and_weaknesses": "Weakness:\nThe research in this paper is actually limited by the client-side data situation because training a discriminator to represent a distribution requires an extremely high volume of client data. This is a cross-silo scenario where each client side needs a significant amount of data to accurately represent a distribution.", "other_comments_or_suggestions": "No", "questions_for_authors": "1. \"We proposed the FedGO algorithm, which effectively addresses the challenge of client data heterogeneity.\" Please explain how your proposed method tackles the issue of client data heterogeneity?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposed a method for federated ensemble distillation designed (in client heterogenous environment) in federated learning. This core idea is to use discriminators trained on each client using data generated from a server-distributed generator. The server then aggregates predictions by assigning weights based on these discriminators, thereby generating pseudo-labels for a server-side dataset. Theoretical justifications are provided for the weighting scheme. Also, experiments across several image classification tasks, demonstrating efficiency over existing baselines. 

The reviewers appreciated the paper’s theoretical grounding. They also found the adaptation of GAN results to the federated setting interesting. Als, the experimental setup was found strong with well-chosen benchmarks, comprehensive comparisons, and ablation studies. Reviewers generally agreed that FedGO achieves better accuracy and faster convergence than prior methods, while maintaining low overhead. At the same time, several reviewers raised concerns about the limitations and assumptions of the proposed method. A shared concern was the theoretical analysis being restricted to binary classification, despite all experiments being conducted in multi-class settings. Additionally, there were doubts about the robustness and scalability of the approach in real-world settings, particularly in the presence of extreme heterogeneity or insufficient client data to train effective discriminators. These major concerns were mostly addressed during the rebuttal perior.
Decision: Accept (poster)