Enhancing One-Shot Federated Learning Through Data and Ensemble Co-Boosting
Authors: Rong Dai
Yonggang Zhang
Ang Li
Tongliang Liu
Xun Yang
Bo Han
Venue: ICLR 2024 poster (2024)
Keywords: federated learning;
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: One-shot Federated Learning (OFL) has become a promising learning paradigm, enabling the training of a global server model via a single communication round. In OFL, the server model is aggregated by distilling knowledge from all client models (the ensemble), which are also responsible for synthesizing samples for distillation. In this regard, advanced works show that the performance of the server model is intrinsically related to the quality of the synthesized data and the ensemble model. To promote OFL, we introduce a novel framework, Co-Boosting, in which synthesized data and the ensemble model mutually enhance each other progressively. Specifically, Co-Boosting leverages the current ensemble model to synthesize higher-quality samples in an adversarial attack manner. These hard samples are then employed to promote the quality of the ensemble model by adjusting the ensembling weights for each client model. Consequently, Co-Boosting periodically achieves high-quality data and ensemble models. Extensive experiments demonstrate that Co-Boosting can substantially outperform existing baselines under various settings. Moreover, Co-Boosting eliminates the need for adjustments to the client's local training, requires no additional data or model transmission, and allows client models to have heterogeneous architectures.

Published as a conference paper at ICLR 2024
ENHANCING ONE-SHOT FEDERATED LEARNING THROUGH DATA AND ENSEMBLE CO-BOOSTING
Rong Dai Yonggang Zhang?, Ang Li Tongliang Liu", Xun Yang' Bo Han
Universityy of Science and Technology of China, "TMLR Group, Hong Kong Baptist University &ECE Department, University of Maryland College Park, "Sydney Al Centre, The University of Sydney rongdai @mail 1. ustc. edu. {csygzhang, bhanml| J deop. hkbu. edu. .kk angliece umd. edu tongliang 1 iuu sydneyyy...... edu. au
edu. cn
ABSTRACT
One-shot Federated Learning (OFL) has become a promising learning paradigm, enabling the training of a global server model via a single communication round. In OFI,, the server model is aggregated by distilling knowledge from all client models (the ensemble), which are also responsible for synthesizing samples for distillation. In this regard, advanced works show that the performance the server model is intrinsically related to the quality of the synthesized data and the ensem- ble model. To promote OFL, introduce novel framework, Co-Boosting, in which synthesized data and the ensemble model mutually enhance each other pro- gressively. Specifically, Co-Boosting leverages the current ensemble model synthesize higher-quality samples in an adversarial manner. These hard samples are then employed to promote the quality of the ensemble model by adjusting the ensembling weights for each client model. Consequently, Co-Boosting period- ically achieves high-quality data and ensemble models. Extensive experiments demonstrate that Co- Co-Boosting der various settings. Moreover, Co- Co-Boosting eliminates the need for adjustments to the client's local training, requires additional data model transmission, and allows client models to have heterogeneous architectures.
substantially outperform existing baselines
INTRODUCTION
Federated learning (FL) (McMahan et al. 2017) has emerged a prominent distributed machine learning framework to train global server model via collaboration among users without sharing their dataset. Though the multi-round parameter-sere
communication paradigm offers the benefit of effectively exchanging information among clients and the central server, it might not be feasible in the real world. This paradigm brings forth significant challenges: 1) heavy communication burden and the risk of connection drop errors between clients and the server (Li et al. 2020aa Kairouz et al.,
2021 Dai et al. 2022). and 2) potential risk for man-in-the-mieee
attacks (Wang et al., 2021 and
various other privacy security concerns (Mothukuri et al. 2021 Yin et al. 2021).
One-shot FL (OFL) (Guaa et al., 2019) has emerged as a solution to these issues by restricting communication rounds to single iteration, thereby mitigating errors arising from multi-round com- munication and concurrently diminishing the vulnerability to malicious interception. Furthermore, OFL is more practical, particularly within contemporary model market scenarios (Vartak et al. (016) where clients predominantly offer pre-trained models. In OFL, the server model is aggregated by distilling knowledge from all client models, commonly using the ensemble, while the ensemble is also responsible for synthesizing data samples for knowledge distillation. Consequently, as illus- trated in Guha et al. (2019) and Zhang al. (2022a), the server model's performance is intricately linked to both the quality of synthesized data and the ensemble. Thus, the primary challenge in improving performance lies in the process of improving the data and the ensemble.
Existing approaches tend to tackle this challenge by exclusively concentrating either enhancing the quality of the ensemble or improving the quality of synthetic data. For instance, to bolster en- semble, prior works including Dennis et al. (202T). Heinbaugh et al. (2023). and Diao et al. (2023)
"Coresponding author. Work done during Rong's visit to TMLR Group HKBU.

---

Published as a conference paper at ICLR 2024
Epoch t-1
Egoch
(%0
70
Arg
Eas
Ensemble
65
Hard
Eas- Halr
Eee Real
Ees-byn
FelENS:
49
DENSE
50
Eascutiic
(Oand
46
Fettrg
Server
Epoch
a
(b)
(c)
(d)
Figure 1: Co- Co-Boosting Framework and Experimental Comparison. (a) illustrates the core concept of our approach. In each epoch, high-quality samples are first generated based on last epoch's ensemble and server, which are then used to adjust client weights giving a better ensemble. Based on the enriched data and refined ensemble, server model is updated by distilling knowledge from them. (b) shows test accuracy of ensemble with averaged weights, learned weights on real data, and learned weights on hard samples. (c) shows test accuracy of server obtained through distillation on real, synthetic, and hard samples with same ensemble. (d) presents an overall comparison. DENSE (Zhang et al. 2022ab signifies the current state-of-the-art, FedENS denotes the averaged ensemble. Experiments in (d) are all done on CIFAR-10 10 with a 10-client Dir (ir(0.1) >-pattd setting.
modify the local training phase and require additional transmissions. terms of improving synthetic data, Li et al. (2021) utilizes auxiliary public datasets, Zhou et al. (2020) proposes transmitting dis- tilled datasets to the server, Yang et al. (2023) proposes to use auxiliary diffusion model and Zhang et al. (202aa) employs data-free data generation methods to synthesize data directly from averaged ensemble models. While the distilled server may improve through the above methods, it is note- worthy that these approaches typically follow sequential process, which means the enhancement of data or the ensemble is prerequisite step before the server model can benefit, omitting the cru- cial relationship between them. What's more, contemporary model market scenarios where only well-pre well-ppe-ttran pre-trained models with diverse architectural possibilities are accessible, any modifications to local training or additional data or model transmissions are discouraged and often disallowed.
To address these challenges, we propose Co-Boosting, a novel one-shot federated learning algorithm as in Fig. aa), in which the synthesized data and the ensemble model mutually boost each other progressively. More specifically, in each training epoch, higher-quality hard samples are generated based on the previous epoch's 's ensemble and server model. Based on these hard samples, the aggre- gation weight for each client model is adjusted, forming better ensemble. Subsequently, the server model is updated by distilling knowledge from both the enriched data and the refined ensemble. As result, with the continuous enhancement of both data and the ensemble, the final server model naturally improves. As depicted in Fig. [bb. (c), and (d), with a better weighted ensemble model and better-quality hard samples, Co-boosting naturally achieves state-of-art performance.
Thorough experiments on multiple benchmark datasets demonstrate the superiority of the proposed Co- Co-Boosting. What's more, due to its inherent nature, our proposed Co- oo-Boosting is more practical to today's ' model market scenarios. In summary, our main contributions can be summarized as follows: 1) We demonstrate that it is possible to simultaneously improve the quality of the synthesized data and the ensemble, which are two key elements in OFL. This discovery could spur progress in OFL methods, highlighting the need optimize their interaction. 2) Within an adversarial paradigm, we introduce Co- Co-Boosting, novel one-shot federated learning method. Periodically, in Co- Co-Boosting, hard samples are generated from the current ensemble, which, in turn, are used to reweight clients, forming an improved ensemble. This mutual enhancement of synthetic data quality and the ensem- ble collectively contributes to the natural emergence of high-performing distilled server model. 3) Our proposed method Co-Boosting, is highly practical to the contemporary model market scenarios as it eliminates the necessity for client-side training adjustments, entails no extra data or model trans- missions, and accommodates diverse client model architectures. 4) Extensive experiments confirm the effectiveness of Co- Co-Boosting, consistently outperforming other basclines thanks to the improved quality of both the synthetic data and ensemble.
2

---

Published as a conference paper at ICLR 2024
2 RELATED WORKS
2.1 ONE-SHOT FEDERATED LEARNING
Guha et al. (2019) originally proposes OFL which collects local models ensemble for the final prediction and further proposes to use knowledge distillation (KD) on such ensemble with public data. This paradigm, which is followed by most works, inherently relates the performance of the server model to the data and ensemble used in the KD stage. [i et al. (201)) proposes to improve the ensemble on the public data. Instead of using public data, Zhou et al. (2020) proposes transmit the distilled local dataset for the server, Yang et al. (203)) proposes to use auxiliary pre-trained diffusion model, while Zhang et al. (022a) generates fake data scouring from the direct ensemble. Regarding the improvement of the ensemble, Dennis et al. (202I) utilizes a cluster-based method and requires uploading the cluster means. Diao et al. al.] (2023) and Heinbaugh et al. (2023) modify the local training phase of each client by introducing placeholders, or conditional variation auto-encoders. However, none of the aforementioned methods simultaneously address improvements in both data and the ensemble. Moreover, few works can be practically applied, especially in contemporary model- market scenarios (Vartak et al. 2016) where only well-pretrained models are provided to the server. This situation implies constraints such as alterations to the client's local training, no additional transmissions, and the possibility of client model heterogeneity.
2.2 KNOWLEDGE DISTILLATION
Knowledge distillation (KD) (Hinton et al. 2015) is proposed to transfer knowledge from or more networks (teacher) to another (student). Taking the same spirit, KD in federated learning focuses transferring knowledge from multiple local clients to the global server model. Lin et al. (2020) initially introduced to utilize KD at the server side based on an unlabeled auxiliary dataset. In effort reduce reliance proxy datasets, generators that are locally updated and globally aggregated are used in Zhu et al. (202T) and Zhang et al. (2022b) to synthesize distillation samples. Wang et al. (2023) further enhances the basic ensemble distillation by using weighted averaging based on locally trained discriminators. However, in the context of OFL, conducting multiple rounds of training or transmitting generators and discriminators is not practical. Additionally, the need for an additional local client component violates the constraints in modern model-market OFL settings. More seriously, the generator trained locally has direct access to the training samples, potentially leading to privacy leakage through its ability to remember all the training data (Liu et al. 2019). On the other hand, the generator OFL trained without access to even one single raw data.
METHODOLOGY
In this section, we first introduce the general process of one-shot federated learning (OFL). Then we detail the proposed method, Co-Boosting, in how we generate high-quality data, high-quality ensemble, and how to link and make them boost each other as illustrated in Fig. aa).
3.1 ONE-SHOT FEDERATED LEARNING
Suppose we have a set of clients C, with n = |C| clients in total. Each client Cs E C has a local private dataset + {xx,, 34) 1 1 where 7ıg JD*| is the number of local data samples x, with the corresponding label y/+ OFL's goal is to train a good machine learning model with parameter 0g over D A DD D with the help of a server in only one communication, as in
min C(ss)
Σ
(1)
IID
(xD
where (c.)):)) is the cross-entropy function, (x;; Os) is the prediction function of the server that outputs the logits (i.e., outputs of the last fully connected layer) of x; given parameter 0s.
Noticeably, in one-shot federated learning, the original training set ID* cannot be accessed, and only well-pretrained models parameterized by Oz. are provided. Here, we define the ensemble as:
Au (x: (0x 17 A (x 0x),
(2)
3

---

Published as a conference paper at ICLR 2024
where (x; 0x)) denotes the prediction function that output the logits of x given 0₁. while w [wi, W2, Wo] adjusts the weights of each local client logits. When w = 1/n, the ensemble the same the averaged ensemble, while when Wa n&/ _k=1 Ra. the ensemble becomes weighted according to the data amount. For simplicity, in the rest paper, we use Aw to denote the ensemble and A (x) to denote Ax (0& (-=)) which means the output logits of the ensemble given x.
When aggregating pre-trained models (0x 1k=1 into one server model 0g, existing works mostly follow two-stage framework. The first is synthesize data Dy based on the ensemble output. In particular, giving random noise 2 sampled from a standard Gaussian distribution and a random uniformly sampled label ys. the generator Gi-) with fg is responsible for generating the data x5 G(z), forming the synthetic dataset Dg. Typically, to make sure the synthetic data can be classified correctly with a high probability by the ensemble Aw, the following loss is adopted:
L(Og)
A
Σ
(CE(Aw (x.),
(3)
Dsl
[x..2.
After getting the synthetic dataset Ds based on the generator in Eq (3, OFL intends to distill the ensemble Aw into the final server model Os with the help of these synthetic data, as in:
min C(ss) 0g
A
Σ
(4)
Dss
where (x)(-:) denotes the Kullback-Leibler (KI.) divergence.
Existing works illustrate that the performance of the server model is intrinsically related to the synthetic data Dss and the ensemble Aw. which can also be concluded according to Eq.(4.
3.2 BOOSTING THE DATA QUALITY
Synthesizing data Ds is used to distill the ensemble model into the final server model as in Eq.44) The quality these synthesized data has been demonstrated vital to the distillation stage Lin et al. 2220)) Moreover, since these data are also generated soureing from the ensemble in Eq. is of great importance to make these data embed as much the knowledge of the ensemble as possible and make them transferable to the final server model.
However, as hinted in Wang et al. 2020 and Zhang et al. 2022a), by utilizing only the CE loss, the synthesized data be easily fitted by the server model, resulting in poor performance in the knowledge distillation stage. Therefore, to improve the quality of the generated data and make them focus more on transferable components, taking inspiration from Dong (023a).. we increase the importance of hard samples while suppressing the importance of easy-to- fit samples in the generation stage. More specifically, given a prediction function which output logits, we employ the GHM introduced in Li et al. (2019) to measure the sample difficulty d of x:
(202)) and Li et al.
d(x, f) = 1 - o (f((x;)) f));;
where (x;(x; 0)), is the probability on label y predicted by the function f(-) with Built upon the sample difficulty, we propose hard-sample-enhanced loss Lu synthesize data:
(x:
Us:
Σ
d(x,, ( vx)).
(6)
Dsl
(X.:BL)EDs
Moreover, to make synthesized samples hard for the server model to fit, adversarial loss (Zhang ct al. 2022c) is also introduced to generate hard samples. We try to maximize the differences in predictions between the ensemble model and the server model when generating data as follows:
0s: 0s: A
Σ
(kL((A fs(x;:
Dsl
(x..).) JEDs
By combining the above losses, we can obtain the loss used to train the generator as follows:
((Og) A Lu ys 0a) + BC Os; Oz),
8
where B is the scaling factor for the losses, which is set in the implementations.
4

---

Published as a conference paper at ICLR 2024
Though samples synthesized using Eq.48) are hard to fit for the current ensemble model, their diffi- culties for the server model are still lacking. This stems from the fact that the server model can easily fit these limited unchanged data during multiple distillation steps. To further promote the sample difficulty and diversity on the fly, we draw inspiration from adversarial learning (Goodfellow et al.) 2014 Tashiro et al. 2020) generate hard and diverse samples for the server model to learn.
More specifically, we diverse and increase the sample difficulty d(x,, Aw) on the fly to make the synthetic samples hard to fit by introducing perturbation ò; for each Xs:
d arg max d(x, + 8', Aw),
(9)
18 Sc
where || oleo represents the Loo -norm and 6 controls the strength of perturbation. To simplify the computation and enhance the diversity, instead of the iterative adversaris attacks, take only one step of loss backward to seek the direction in the input space that maximizes the similarity between the model output and randomly sampled vector. The hard samples constructed as follows:
Vx, (n An (xx))
(10)
|| Xx, (u Aw (x,)))l2
where (ni,([-1, is a randomly sample vector with dimension d. Following these, the originally generated hard samples are further harder and more diverse due to the randomness in u.
By replacing each sample x, in Ds into x.. we can achieve a more hard and diverse synthetic dataset Ds. Utilizing these hard samples, the knowledge of the ensemble model is transferred to the server model with parameter 05 by knowledge distillation the same as in Eq.(4.
Overall, with the hard sample technique embedding in the data synthesizing stage (replacing the generator loss in (q.(3) with (8. and making them diverse in the distillation stage (reconstruct the synthetic dataset D' according to Eq. (I)). the quality of data generated and used for distillation becomes better, naturally boosting the performance of the server model.
3.3 BOOSTING THE ENSEMBLE QUALITY
The ensemble model takes the role of aggregating knowledge from all pre-trained models (0x)
)
and forms virtually best-performance teacher. A straightforward method is to obtain the global model by averaging the parameters of all client models (e.g. FedAvg (MeMahan al. 2017)) ). However, FedAvg may fail to deliver a good performance when data among clients are non IID (Karimireddy et al.] [2200; Acar et al. [02T]) and cannot handle the challenge of client model het- erogeneity. Recent works (Heinbaugh et al. 2023 Diao et al.) 2023) intend to construct better ensemble model by altering the local training phase of each client, which may unreliable, espe- cially in today's 'ss model market scenarios. To tackle the client model heterogeneity and make the ensemble more practical, Guha et al. (2019); Zhang et al. (2022a) utilizes the direct ensemble Aw with wy I/n as the teacher, which means averaging the logits weighted averaging them ac- cording to the number of the client data (w), = n&/ Σx=1 n4)) However, as suggested by Zhang et al. 2023) and Wang al. (2023), the simple averaging weighted averaging based on the client data amount may not be effective, especially in non-IID settings. There exists a better weighted combination of each client's contribution. Yet, their methods either need alter the local training or transmit additional information, therefore their methods cannot be applied to one-shot FL.
To this end, we propose to boost the ensemble quality by searching for more effective weighted ensemble of logits. As demonstrated by our experimental results in Fig. [b), given high-quality data (validation data), we can achieve a better ensemble with weights different from simple averaging or data amount based averaging. Fortunately, instead of using auxiliary data, we actually can acquire high-quality generated data from the hard synthesized samples set Dg. Therefore, to get the best weights [W], W2, wN] on Ds. we need to solve the following optimization problem:
min Lu (w)
Σ
ÉCE
Wk (x.; 0z), va).
(11)
Ds
(X..V.EEDg
where y, is the corresponding label to each synthesized hard samples x,. Exploring the optimal w requires multiple inner steps, leading to the training time to increase exponentially. Also inspired by
5

---

Published as a conference paper at ICLR 2024
Algorithm 1 Co-Boosting
1: Input: Clients' local models {01.... )n server model 0s. synthetic dataset Ds M, ensem- ble Aur. generator 0g. perturbation strength step size learning rate of generator and server IJG and 7/s, generation iterations Ta. global model training epochs T, and batch size b
2: Output: Global server model 0
3: for epoch T do
4: Generate hard synthetic samples
5: Sample batch of noises and labels {zz, V/ Wilim
do 7: Generate fx, Jim1 with (2i)/_1 and OG
6: for ty =0 0 to Ta
8: Update the generator: AG e- 0g - nJ Vog ((OG), where L(Og) is defined Eq.
9:
end for
10:
Ds f D3s {x.}i=
11:
Diverse each sample {x.} in Bs to { according Eq.10
12:
// Obtain a better ensemble
13:
Update the mixing weights with Ds according to Eq 12
14:
Construct an updated ensemble Am with updated wr according to Eq.
// Obtain the final server model for sampling batch {x.} in D do
16:
Update the server model: 05 t 0s 11s ve, C(Os), where C(0s) is defined in Eq 18: end for
end for
methods of adversarial attacks (Goodfellow et al., 2014), we use the gradient's direction and fixed step size y to update uo every time after getting each batch of the synthesized data DDs:
w Normalizo(ui) --1 usign( V (w)
(12)
where Normalize denotes bounding each wy into [0, 1] and sign(-) means the sign function.
The reweighting of each client's logit results in a superior ensemble model, which will naturally benefit the server model. Morcover, since the operations are done on the logit layer, this reweighting technique can be easily applied to both heterogeneous and homogeneous client model settings.
3.4 CO-BOOSTING THE DATA AND THE ENSEMBLE
As aforementioned, we introduce how to boost the data quality by utilizing hard sample techniques with fixed ensemble and how to boost the ensemble with fixed synthetic dataset. Actually, these two stages are inherently entangled and can boost each other at the same time. To get high-quality ensemble A, and synthesized data Ds, we in fact trying to solve the following problem:
min
Σ (X..I-)EDg
max ECE BES
wafa(x, + 6; 0x0x)..),
(13)
Dsl
where ys the label of sample X,, is the perturbation constrained in S. This problem can be addressed adversarially, which means the improvement of the data and the ensemble can be done si- multaneously. With better quality data, the weighted ensemble can reach higher performance, while with this better ensemble, the data synthesized sourcing from this ensemble can further embed more knowledge. Therefore, by mutually boosting the quality of the synthesized data and the ensemble, we can naturally get a better-performance server model through the distillation in Eq. (
The overall algorithm summarized in Algorithm| 1 In each epoch, we first generate hard samples based on the current ensemble model and the last epoch server model. With these generated data, an enhanced ensemble model is cultivated by searching for the optimal ensembling weights of each client's logits. Utilizing the generated data and the upgraded ensemble, the final server model is trained by distilling the ensemble on these data. As illustrated in Sec. 3.2 and Sec. 3.3] with either one fixed, one benefit from the other, thus by mutually boosting each other in the proposed Co-Bor Beossottng we can achieve both better quality data and the ensemble periodically. Therefore, the global server model trained on them will inherently become better than other methods.

---

Published as a conference paper at ICLR 2024
Table I: Test accuracy of the server model of different methods over five datasets and across three levels of statistical heterogeneity (lower is more heterogeneous).
Method
a
FedAvg
FedDF
F-ADI
F-DAFI.
DENSE
Co-Boosting
0.05
46.35:10.98
80.7311.08
80.12:1..76
78.4911.36
81.06.1.112
93.9
MNIST
0.1
75.68
87.91
85.92.10.22
87.4410.61
87.8311.388
9..4411.02
0.3
78.9710.82
97.66.10.10
96.3410.83
96.36110.98
96.96.10.53
0.05
20.07.11.98
44.7311040
42.25
41.66.10.83
44.77.11.87
50.62.114
FMNIST
0.1
16.61=1.94
68.40+1.8 1.80
63. 13.199
67.81=0.93
69,43+1.94
74.86=1.70
0.3
60.13+2.62
83.14=0.477
7..80+1.72
78.68=0.49
81.31+0..3
83.11
0.05
39.41#2.119
60.79=0.52
56.58+1.05
59.38=1.19
6..24+1.31
0..40=0.86
SVHN
0.1
1..22:1.99
68.98
1..33+1.69
67.77=0.34
1.030
7.18
0.3
72.61:12.06
79.78
16.75:11.47
1.00
78.73:10.84
81,31 1.09
0.05
17.494-2.51
37.63++0.67
36,94++1.70
1..8
38.37++.088
47.20
CIFAR-
0.1
27.54+
49.63+0080
47.19+00.97
46.32-00.7
47.80+1.21
57.09- 0.94
10
0.3
46.39+2..77
67.18+0060
6..60+1.32
65.89+1.69
66.77+
70.24
0.05
6.45+
16.07+00554
13.75+1.01
15.79+0.21
1..1
19.24+1.44 1.42
CIFAR-
0.1
10.2811.70
22.07:10.43
19.4411.66
20.99=1.17
22.21
23.59
100
0.3
15.2212..08
30.71:10.53
26.14:11.37
28.7911.25
30.3311.24
31.30
EXPERIMENTS
4.1 EXPERIMENTAL DETAILS
Datasets and partitions. We conduct experiments on five real-world image datasets that are stan- dard in the FL literature: MNIST (LeCun et al. 1998), FMNIST (Xiao et al. 2017). SVHN (Netzer et al. 1) CIFAR 10, and CIFAR 100 (Krizhevsky et al. 2009). To simulate statistical heterogene- ity. we use Dirichlet distribution to generate disjoint non IID client client training datasets as in inZhang et al. sample pk Dir(a) and allocate a Pa pro- portion of the data of class 7 to client k: The parameter a controls the level of statistical imbalance, with smaller inducing more skewed label distributions among clients.
2022a) and Heinbaugh et al. (2023). In partieular,
Baselines. Within the contemporary model-market scenarios, we compare the performance of Co- Boosting lggainst two existing methods: FedAvg (MeMahan et al. 2017)) and DENSE (Zhang al. 2022ab. Similar in (Zhang et al. 2022a), we also introduce two prevailing data-free KD methods DAFL Chen et al., (2019) and ADI (inn et al.
2020) and apply these one-shot FL, giving F-DAFL
and F-ADI. We also include FedDF (Lin et al.
2020) using the real validation dataset as the baseline.
Configurations. Mollaann al. (001), we use CNN with 5 layers for SVHN, CIFARIO, and CIFARIOU LeNet-5 (LeCun et al.) 1998) for MNIST and FMNIST. All available test data is used to evaluate the final server model (or ensemble). Unless otherwise stated, experiments are done with 10 clients and (ir Dir(O.1)-parted Results are reported averaged across at least 3 random seeds.
4.2 GENERAL RESULTS
Overall Comparison. To evaluate the effectiveness of our method, we conduct experiments un- der various non-IlI settings by varying a = (0.05, 0.1, 0.3) and report the performance across different datasets and methods in Table [ Notice, that we use the validation set in FedDF, which not practical in the real world. From the table, we can conclude that Co-Boosting consistently outperforms all other baselines in all settings. Notably, in many settings, Co-Boosting achieves over a 5% accuracy improvement compared to the best baseline, DENSE. In cases of extreme statisti- cal heterogeneity, such as when a 0.05. Co-Boosting surpasses the best baseline by substantial margins with 12.87%. 5.85%, 5. 16%,, 8. 8.83%, and 3.07% on MNIST, FMNIST, SVHN CIFAR- 10, and CIFAR- 100, respectively. We also compare the performance of the ensemble used in different methods on SVHN and CIFAR-IO in Table others please refer to the Appendix. FedENS denotes averaged ensemble. The superior performance of the server model can be attributed to the enhanced
"Code available http://ggghaa...ooooooooaiiiiiio

---

Published as a conference paper at ICLR 2024
Table 2: Test accuracy of the ensemble in SVHN, and CIFAR-10 in Dir-parted setting.
Dataset
SVHN
CIFAR-10
Method
c=0.05
x=0.1
0=
<=0.05
a=0.1
<=0.3
FedENS
61.62+1.61
69.71
80.54:0.57
42.34=0.67
49.99=0.85
69.61=0.50
Co- Co-Boosting
65.69:1.448
13.5
82.90=
48.75
59.86 1.76
72.67:1.277
quality of synthesized data and the ensemble. While all compared methods, except FedAvg, utilize the direct logits averaging ensemble as the ensemble and aim to distill knowledge from it in data- free manner, Co-Boosting, with its co-enhancing technique, results a superior ensemble teacher, surpassing FedENS significantly. In word, the superiority of our proposed method can be owed to the enhanced data and ensemble quality, which naturally translates into better server model.
Adaptation to Client Model Heterogeneity. To evaluate proposed method in potential client heterogeneity setting, we apply five different models under a CIFAR- 10, (ir(0.1) --pated setting. The heterogeneous models include CNNI in McMahan al. (2017), CNN2 in the pytorch tutorial (Paszke et al. 2019). ResNet (He et al. 2016), MobileNet (Howard et al. 2019), and ShuffleNet (Ma) al. 2018). Table demonstrates the results of comparable methods, where Local denotes directly taking the pre-trained model for testing. We take ResNet as the server architecture and omit FedAvg as it does not support this setting. We use the same optimization hyperparameter for all methods and across all model architectures. We remark that as suggested in ZZang et al. (2022a) and Diao et al. (2021) FL under both non-llD data and different model architecture is a quite challenging task. Even under this setting, our proposed Co- Boosting still consistently outperforms other baselines by a large margin thanks to the benefits of making the ensemble and data improve together.
Table 3: Test accuracy of the server model architectured in RestNet in CIFAR- 10 across three levels of statistical heterogeneity under a heterogeneous client model setting.
Dir(-)
Local
FedDF
F-ADI
F-DAFI
DENSE
Co-Boosting
o=0.05
46.49+1.97
+6.51
55.64+11.96
55. 15.14
1..79+
58.64+1.83 62.30 75.02++1.37
o=0.1
51.70+1.14
59.49+1.06
60.30+1.70
58.58+1.49
58.67+-0.8 1.698+1.61
a=0.3
1.56
70.53+1.19
71.61
1..9
4.3 IN-DEPTH STUDY
Different Local Data Amount. To further assess the
effectiveness of Co-Boosting, which involves a better-
Table 4: Test accuracy of ensemble.
weighted ensemble, we conduct experiments in an unbal-
Method
d-40.4
a-088
d=1.2
anced local data setting. Similar to Acar al. (202I), we sample data amounts for each client from lognor- mal distribution. Higher values of result in more un- equal data distribution. Table and Fig. display the per- formance of the ensemble and the server, where the pre- fix 'DW'' signifies weighted averaging based on the local data amount. As observed, reweighting clients based on their data amount yields some benefits, but it falls short of achieving the best ensemble. Furthermore, when using the averaged ensemble FedENS as the teacher, all baseline methods perform poorly due to the suboptimal teacher. In contrast, benefiting from simultaneously boosting data and ensemble, the ensemble we get consistently outper- forms FedENS. This leads to substantial performance gain of the server model achieved in Co-Boosting over all baselines, with a margin of at least 10%.
FedENS
46.771102
41 1.20
37,88+1.38 1.38
47.00+121
53.25
47.52-00.00
Co-Roosting
58.9500.0
57.41-112
55.27=1.72
F-ADI
DESSE Boottig
DW-Feddig
FedDy
Figure 2: Test accuracy of server
Different Data Distribution Shift. Following Diao et al. 2023 we also conduct experiments on a CLols partition setting, which means each client only possesses data of C' out of all classes. Results in Table further demonstrate the superiority of our proposed method. With better data quality and better ensemble, method consistently achieves the best server model.
8

---

Published as a conference paper at ICLR 2024
Table Test accuracy of the server of different methods in CIFAR-I0 under C_cls- cds-parted setting.
Clrls
FedAvg
FedDF
F-ADI
F-DAFL
DENSE
Co- Co-Boosting
2
16.15112.61
23.07.10.83
23.16.11.12
24.53.10.57
23.85.10.93
36.37.11.8
26.47-11.46
38.39110.64
36.06.11.93
38.13.11.17
38.14
53.91.1
33.78+2.42
5..51+1.12
51.04+1.40
52.53+0.97
51.53+1.79
28.00+1..59
5
1.99
58,3441.58
55.27+2.006
54.67+1..26
56.79+1.03
6..52:+1.75
Different Number of Clients. We evaluate the performance of these methods by varying the number of clients participating in OFL in Table From the table, the final sever model still achieves the best accuracy when increasing the number of clients. This again validates that the increment quality of the ensemble model and data naturally brings better server model.
Table 6: Test accuracy of the server model in CIFAR-10 across different numbers of clients.
FedAvg
FedDF
F-ADI
F-DAFL
DENSE
Co-Boosting
5
1.79
50.62:0.98
48.76:10.78
1..76=t1.42
50.53=1.02
54.29 57.09+
10
2.854++..80
09.63++0.80
47.19+-0.97
46.32+0.97
47.80+1.21
20
1..3
38.98-+0.99
38.93++0.64
36.28+1.399
38.86+0.42
49.56.+0.98
23.01-+0.94
0.652+0062
1.14
29.41
28.51+0054
42.29-+1.43
Table 7: our method. "GHS" for hard sample genera- tion in the generator loss, "DHS" for on-the- fly diverse hard sample creation, and "EE" for ensemble enhancement through reweighting.
Ablations on different components of
Effects the proposed components. We further study the effectiveness of our proposed hard sample generation loss Sec 3.2 on-the-fly sample diffi- culty promotion in Sec 3.2 and ensemble enhancing in S.c Table 7 shows the experimental results on SVHN and CIFAR-10 in 10-client Dir(0.05) parted setting. The results in the table illustrate that individually improving either data ensem- ble leads to noticeable enhancements in the final server model performance. However, the most re- markable results are achieved when both data qual- ity and ensemble capability are improved simulta- neously. This finding strongly aligns with the un- derlying motivation of our study.
GHS
DHS
EE
SVHN
CIFAR-I0
58.46
39.72
61.18
42.85
61.38
43.75
62.67
41.45
63.42
45,81
62.46
44.36
64.40
46.74
65.40
47.20
More facets. For a thorough and comprehensive understanding. we operate sensitivity analyses of hyperparameters, compare with multi-round federated learning, and conduct experiments with heavier local models. Please refer to the Appendix for the results. Moreover, since our Co- co-Boosting needs alternation of the local training, it be combined with advanced local training. The results attached the Appendix further demonstrate the superiority of our proposed Co-Boosting.
Limitation. The mixing weights determined using synthetic samples. Though promising, there still some disparity when compared to mixing-weighttd ensemble trained on real training data, as in Fig. bb). One possible way is introduce virtual data (Tang al. (022). The exploration of methods to generate data capable of bridging this gap remains an avenue for further research.
5 CONCLUSION
In this paper, we seek to tackle the inherent bottleneck one-shot federated learning, where the performance of the server model is inextricably linked with the quality of the generated data and the ensemble. We propose Co-Boosting, a novel method that facilitates a mutually beneficial relation- ship between data generation and ensemble improvement. By iteratively generating hard samples from the ensemble and enhancing the ensemble based on these data, Co- Co-Boosting adversarially improves the quality of both the data and the ensemble, leading to the natural refinement of the server model. Extensive experiments across various settings validate the efficacy of our method and demonstrate that method can be practically applied to contemporary model-market scenarios.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
RD, YGZ and BH were supported by the NSFC General Program No. 62376235, Guangdong Basic and Applied Basic Research Foundation No. 2022A1515011650 CCF--Baidu Open Fund, HKBU Faculty Niche Research Areas No. RC-FNRA--G//2-22SSCCII Incentive Scheme. RD and XY were supported by National Natural Science Foundation of China (NSFC) under Grant U22A2094. TL is partially supported by the following Australian Research Council projects: FT220100318, DP220102121, LP220100527, LP220200999, IC190100031.
and HKBU CSD Departmental
REFERENCES
Durmus Alp Emre Acar, Yue Zhao, Ramon Matas, Matthew Mattina, Paul Whatmough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. In International Con- ference on Learning Representations, 2021. URL
wwtpeeewwwwwwweeeeeeeeeee formm?
id Bd-B7440M6629
Debora Caldarola, Barbara Caputo, and Marco Ciccone. Improving generalization in federated learning by seeking flat minima. In European Conference on Computer Vision, pp. 654-672. Springer, 2022.
Tianyu Chang, Xun Yang, Xin Luo, Wei Ji, and Meng Wang. Learning style-invariant robust repre- sentation for generalizable visual instance retrieval. In Proceedings of the 31 ACM International Conference on Multimedia, pp. 6171-6180, 2023a.
Tianyu Chang, Xun Yang, Tianzhu Zhang, and Meng Wang. Domain generalized stereo matching via hierarchical visual transformation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 9559-9568, June 2023b.
Hanting Chen, Yunhe Wang, Chang Xu, Zhaohui Yang, Chuanjian Liu, Boxin Shi, Chunjing Xu, Chao Xu, and Qi Tian. Data-free learning of student networks. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 3514-3522, 2019.
Hong- You Chen, Cheng- Hao Tu, Ziwei Li, Han Wei Shen, and Wei-Lun Chao. On the importance and applicability of pre-training for federated learning. In The Eleventh International Conference on Learning Reprexentations, 2022.
Rong Dai, Li Shen, Fengxiang He, Xinmei Tian, and Dacheng Tao. Dispfl: Towards communication- efficient personalized federated learning via decentralized sparse training. In International Con- ference on Machine Learning, pp. 4587-466 PMLR, 2022.
Rong Dai, Xun Yang, Yan Sun, Li Shen, Xinmei Tian, Meng Wang, and Yongdong Zhang Fedgamma: Federated learning with global sharpness- aware minimization. JEEE Transactions on Neural Networks and Learning Systems, pp. 1-14, 2023. doi: 10. 1..1109/TNLLS2222333332
Don Kurian Dennis, Tian Li, and Virginia Smith. Heterogeneity for the win: One-shot federated clustering. In International Conference on Machine Learning, pp. 2611 2611-2622. PMLR, 2021.
Enmao Diao, Jie Ding, and Vahid Tarokh. Heterofl: Computation and communication efficient federated learning for heterogeneous clients. In International Conference Learning Represen- tations, 2021. URL htRLihttts::::peeeeeeee
net/ forum ? 2id -idd-iiddie TNK2BByFKXg
Yiqun Diao, Qinbin Li, and Bingsheng He. Towards addressing label skews in one-shot federated learning. In The Eleventh International Conference Learning Representations, 2023. URL
https://openrevii opwnreviewwweeeeeoo net/ wtwp:/rmmmiirrghhhee0 oog.r//ww........eeooooooaaaaaaa
https: /
Jiahua Dong, Yang Cong, Gan Sun, Bineng Zhong, and Xiaowei Xu. What can be transferred:
Unsupervised domain adaptation for endoscopic lesions segmentation. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pp.
2020.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversund training of neural net- works. The journal of machine learning research, 17(1):2096-200 2016.
10

---

Published as a conference paper at ICLR 2024
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
Neel Guha, Ameet Talwalkar, and Virginia Smith. One-shot federated learning. arXiv preprint 17X 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Clare Elizabeth Heinbaugh, Emilio Luz-Ricca, and Huajie Shao. Data-free one-shot feder- ated learning under very high statistical heterogeneity. In The Eleventh International Confer- ence on Learning Representations, 2023. URL httph:tt/////....... /tt.httt:://oppeeeeei ew. net./ / forum? ?id i
hb4 jspB.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:/503.022 2015.
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1314-1324, 2019.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Ad- vances and open problems in federated learning. Foundations and Trends@ in Machine Learning, 14(1-2):1-215 2021.
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning, pp. 5132-5143. PMLR, 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Master thesis, 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (6(11))2278-2333 1998.
Buyu Li, Yu Liu, and Xiaogang Wang. Gradient harmonized single-stage detector, In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 8577-8584, 2019.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision, pp- 5542-5550, 2017.
Huantong Li, Xiangmiao Wu, Fanbing Lv, Daihai Liao, Thomas H Li, Yonggang Zhang, Bo Han, and Mingkui Tan. Hard sample matters a lot in zero-shot quantization. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pp. 24417-24426, 2023a.
Qinbin Li, Bingsheng He, and Dawn Song. Practical one-shot federated learning for cross-silo setting. In Zhi-Hua Zhou (ed.), Proceedings of the Thirtieth International Joint Conference Artificial Intelligence, IJCAI-21, pp. 1484-1490. International Joint Conferences on Artificial Intelligence Organization, 8 2021. doi: 0224963/jjcui.202212222 URL https: orq/
10.2499 10.24963/1jca1. http:////jcaa..222200033 2021/205 Main Track.
Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE signal processing magazine, (7 60, 2020a.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Sys- tems, 45429-450, 2020b.
11

---

Published as a conference paper at ICLR 2024
Ying Li, Xingwei Wang, Rongfei Zeng, Praveen Kumar Donta, IIr Murturi, Min Huang, and Schahram Dustdar. Federated domain generalization: survey. arXiv preprint arXiv:2306.01334 2023b.
Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning, Advances in Neural Information Processing Systems. 33:2351-2366 2020.
Yi Liu, Jialiang Peng, JQ James, and Yi Wu. Ppgan: Privacy-preserving generative adversarial net- work. In 2019 IEEE 25Th international conference on parallel and distributed systems (ICPADS), pp. 285-989. IEEE, 2019.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient cnn architecture design. In Proceedings of the European conference on computer vision (ECCV), pp. 116- 131, 2018.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficii learning of deep networks from decentralized data. In Artificial Intelli- gence and statistics, pp. 1273-1282. PMLR, 2017.
Viraaji Mothukuri, Reza M Parizi, Seyedamin Pouriyeh, Yan Huang, Ali Dehghantanha, and Gautam Srivastava. A survey on security and privacy of federated learning. Future Generation Computer Systems, 115:619-640 2021.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Yixing Peng, Quan Wang, Zhendong Mao, and Yongdong Zhang. Sade: self-adaptive expert for multi-dataset question answering. In ICASSP 2023-2023 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Siyuan Qiao, Wei Shen, Zhishuai Zhang, Bo Wang, and Alan Yuille. Deep co-training for semi- supervised image recognition. In Proceedings of the european conference on computer vision (eccv), pp. 135-152, 2018
Zhe Qu, Xingyu Li, Rui Duan, Yao Liu, Bo Tang, and Zhuo Lu. Generalized federated learning via sharpness aware minimization. In International Conference on Machine Learning, pp. 18250- 18280. PMLR, 2022.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
Peipei Song, Dan Guo, Xun Yang, Shengeng Tang, Erkun Yang, and Meng Wang. Emotion-prio awareness network for emotional video captioning. Proceedings of the 31 ACM International Conference on Multimedia, pp. 589-600, 2023.
Zhenheng Tang, Yonggang Zhang, Shaohuai Shi, Xin He, Bo Han, and Xiaowen Chu. Virtual ho- mogeneity learning: Defending against data heterogeneity in federated learning. In International Conference on Machine Learning, pp. 21111-21133 132. PMLR, 2022.
Yusuke Tashrro, Yang Song, and Stefano Ermon. Diversity can transferred: Output diversification for white-and black-box attacks. Advances in neural information processing systems, 33:4536 4548, 2020.
Manasi Vartak, Harihar Subramanyam. Wei-En Lee, Srinidhi Viswanathan, Saadiyah Husnoo, Samuel Madden, and Matei Zaharia. Modeldb: a system for machine learning model manage- ment. In Proceedings of the Workshop Human-In-the-Loogg Data Analytics, pp. 1-3, 2016.
12

---

Published as a conference paper at ICLR 2024
Derui Wang. Chaoran Li, Sheng Wen, Surya Nepal, and Yang Xiang. Man-in-the-middle attacks against machine learning classifiers via malicious generative models. IEEE Transactions on De- pendable and Secure Compuring, 18(5)22074 2087, 2021. doi: 10.1 10.1109TDSSC22200:00100000
Haozhao Wang. Yichen Li, Wenchao Xu, Ruixuan Li, Yufeng Zhan, and Zhigang Zeng. Dafkd: Domain-aware federated knowledge distillation. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pp. 20412-20422 2023.
Jianyu Wang. Qinghua Liu, Hao Liang Gauri Joshi, and Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 33:7611 1-7623,,, 2020.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint arXiv: / 170.007747 2017.
Mingzhao Yang, Shangchao Su, Bin Li, and Xiangyang Xue. Exploring one-shot semi-supervised federated learning with a pre-trained diffusion model. arXiv preprint arXiv:2305.0406, 2023.
Xun Yang, Fuli Feng, Wei Ji, Meng Wang, and Tat-Seng Chua. Deconfounded video moment re- trieval with causal intervention. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1-10, 2021.
Xun Yang, Shanshan Wang, Jian Dong, Jianfeng Dong, Meng Wang, and Tat-Seng Chua. Video moment retrieval with cross-modal neural architecture search. IEEE Transactions Image Pro- cessing, 31: 11:1224-1 2022.
Hongxu Yin, Pavlo Molchanov, Jose M Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K Jha, and Jan Kautz. Dreaming distill: Data-free knowledge transfer via deepinversion. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pp. 8715- 8724, 2020.
Hongxu Yin, Arun Mallya, Arash Vahdat, Jose M Alvarez, Jan Kautz, and Pavlo Molchanov. See through gradients: Image batch recovery via gradinversion. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pp. 16337-16346, 2021.
Jie Zhang, Chen Chen, Bo Li, Lingjuan Lyu, Shuang Wu, Shouhong Ding, Chunhua Shen, and Chao Wu. Dense: Data-free one-shot federated learning. Advances in Neural Information Processing Systems, 35:2 14 14-2244,, 2022a.
Lin Zhang, Li Shen, Liang Ding, Dacheng Tao, and Ling- Yu Duan. Fine-tuning global model via data-free knowledge distillation for non-iid federated learning. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pp. 10174-10183, 2022b
Ruipeng Zhang, Qinwei Xu, Jiangehao Yao, Ya Zhang, Qi Tian, and Yanfeng Wang. Federated do- main generalization with generalization adjustment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3954-3963, 2023.
Yonggang Zhang, Mingming Gong, Tongliang Liu, Gang Niu, Xinmei Tian, Bo Han, Bernhard Schölkopf, and Kun Zhang. Adversarial robustness through the lens of causality. In International Conference on Learning Representations, 2022c.
Sheng Zhou, Dan Guo, Jia Li, Xun Yang, and Meng Wang. Exploring sparse spatial relation in graph inference for text-based vqa. IEEE Transactions on Image Processing, 32:5060-515 5074, 2023. doi: 10.1109/T1P.203.3333 333332...
Yanlin Zhou, George Pu, Xiyao Ma, Xiaolin Li, and Dapeng Wu. Distilled one-shot federated learning. arXiv preprint arXiv:2009.0799) 2020.
Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou. Date-free knowledge distillation for heterogencous federated learning. In International conference machine learning, pp. 12878-12889, PMLR, 2021.
Weiming Zhuang, Chen Chen, and Lingjuan Lyu. When foundation model meets federated learning: Motivations, challenges, and future directions. arXiv preprint arXiv:2306. 15546, 2023.
13

---

Published as a conference paper at ICLR 2024
A MORE DISCUSSIONS ABOUT RELATED WORKS
A.1 ONE-SHOT FEDERATED LEARNING
Large-scale data is beneficial to service providers in areas such as computer vision (Chang et al. 2023b a). natural language processing (Peng et al. 2023), or video processing (Yang et al. 2021 2022 Zhou et al. 2023 Song et al. 2023) among many others who can improve Al-based products or user experience via deep learning techniques with large data requirements. With the growing con- cern of data privacy, federated learning (FL) (McMahan et al.) [017] allows multiple decentralized clients to collectively train a machine learning model without the need to gather all clients' data. Most FL algorithms follow the communication prototype of FedAvg (McMahan et al., (0177). which requires many communication rounds to train an effective global. To minimize communication costs and mitigate potential security risks associated with multi-round communications, one-shot federated learning (OFL. has emerged as a promising research direction.
Compared to the straightforward baseline of parameter averaging, in FedAvg (McMahan al., 2017). Guha et al. (2019) is the first to propose OFL and utilizes the ensemble of each client's pre- trained models as the ensemble teacher. Subsequently, an auxiliary dataset employed to distill knowledge from this ensemble and consolidate into a unified server model. This paradigm, which the mainstream of OFL, inherently relates the performance of the server to the data and ensemble used in the knowledge distillation stage.
With the purpose of using high-quality data, Li et (021) proposes two-tier knowledge distilla- tion stage with the public dataset. Instead of using public data, (Zhou et al. (2020) proposes to distill the local dataset on the client side and then send it to the server, which may help the aggregation. Yang et al. (2023) proposes to transmit class prototype and utilize auxiliary pre-trained diffusion models. To avoid transmitting additional information, Zhang al. (2022a) proposes to generate fake data scouring from the direct ensemble and then these fake data to distill the server model.
Regarding the improvement the ensemble, Dennis et al. (02T reforms the OFL task by utilizing cluster-based method and requires uploading the cluster means, Diao et al. (2023) modify the local training phase of each client by introducing the placeholders each client model. Heinbaugh et al. (2023) alters the local model into conditional variation auto-encoders and uses these to generate samples for ensemble and knowledge distillation. Nevertheless, it's worth noting that none of the previously mentioned approaches tackle enhancements in both data and ensemble simultaneously. Furthermore, only a few methods are applicable in real-world scenarios, particularly in the context of modern model-market scenarios (Vartak al., 2016). where the server supplied with exclusively models. In this context, there are constraints that include maintaining the integrity of the client's ' local training process, avoiding additional data transmissions, and accommodating potential variations in client model heterogeneity.
We compare these OFL algorithms in Table Noticeably, we are the first to simultaneously enhance both data and ensemble, and our approach is adaptable to contemporary model-market scenarios.
Table 8: Comparison among existing one-shot FL algorithms.
Method
Purpose
Limitations of contemporary model market scenarios
Improve data
Improve ensemble
No support of
No alternations of local training
No additional transmission
Adapt to model heterogencity
extra data
Guha et al.
2019
Zhou al.
2020
Li al.
2021
X
Dennis al
2021
Zhang et al.
2022a
Diao al.
2023
Heinbaugh et al. 2023
Yang al
2023
Co-Boosting (nurs)
14

---

Published as a conference paper at ICLR 2024
A.2 KNOWLEDGE DISTILLATION
Knowledge distillation (KD) (Hinton et al. 2013) is proposed to transfer knowledge from one or more networks (teacher) to another (student). Typically, the student model is trained by minimizing the discrepancy between student and teacher logits generated using a suitable auxiliary dataset; KL- divergence is often chosen the measure of the discrepancy.
In the same vein, Knowledge Distillation (KD) in federated learning aims to transfer knowledge from multiple local clients to global server model. This concept was initially introduced by Lin ct al. (2020) which proposed using KD at the server side based on an unlabeled auxiliary dataset. To reduce dependency on proxy datasets, researchers have turned to data-free knowledge distillation methods used in centralized settings, such as those proposed in Chen et al. (2019) and Yin et al. 2020). In the centralized setting, generator noise is optimized to mimic the behavior of the teacher model. These synthetic data are then used for knowledge distillation. Inspired by these centralized data-free KD works, generators are updated locally and aggregated globally to synthesize distillation samples, as demonstrated in the works Zhu et al. (202T) and Zhang et al. (2022b). Wang al. (2023) further improved upon basic ensemble distillation by implementing weighted averaging based on locally trained discriminators.
However, in one-shot federated learning (OFL), conducting multiple rounds of training or transmit- ting generators and discriminators is not practical. Moreover, the requirement for an additional local client component contradicts the constraints in modern model-market OFL, settings. A more serious concern is that locally trained generator has direct access to the training samples, which could potentially lead to privacy leakage as it has the ability to remember all the training data Liu et al. (019).. In contrast, the generator in OFL is trained without access to even single data point.
A.3 MORE DISCUSSIONS
The parallels and contrasts with centralized co-learning techniques, such as those discussed in Qiao iet al. (2018), are indeed intriguing to explore. It's important to note that centralized co-learning typically operates under the premise of learning from data that presents different views. However, in the one-shot federated learning context, the divergence in data distributions among clients is significantly more pronounced. Moreover, in OFL, models provided to the server are already pre- trained, which precludes the possibility of retraining local models. This distinction is crucial for understanding the unique challenges and approaches in OFL.
Additionally, viewing OFL through the lens of today's pre-trained foundation model era, as high- lighted in works like Chen et al. (2022); /Zhuang et al. (2023) opens up new avenues of exploration. Our work positions itself as a pioneering effort in adapting OFL to the era of foundation models. The key strength of our method is its independence from alterations in local training and its architecture- agnostic nature. This flexibility is particularly relevant in the context of foundation models, which are becoming increasingly central in various domains.
B MORE EXPERIMENTS
B. MORE DETAILS ABOUT THE EXPERIMENT CONFIGURATION
Dataset. We use five datasets in our experiments. MNIST LeCun et al. 1998) is a large database of 10-class handwritten digits. It consists of 60,000 training images and 10,000 testing images, each of which is a 28x28 grayscale image. FMNIST (Xiao et al. 2017) is a dataset of Zalando's 's 10-class article images. It consists of a training set of 60,000 examples and a test set of 10,000 examples, each of which is a 28x28 grayscale image. SVHN (Netzer et al. (011T) is a real-world image dataset in 10 classes with a size of 32x32 used for developing machine learning and object recognition algorithms, 72.357 for train, and 26,032 for test. CIFAR-10 (Krizhevsky et al., 2209) consists of 60,000 32x32 color images in 10 classes. 50,000 for train, and 10,000 for test, while CIF dataset is similar to the CIFAR- 10 dataset but it has 100 classes containing 600 images each. Each image is also a 32x32 color image. For the FedDF method, we use 20% of the training set as a validation set for distillation. While all of the test data is used for testing.
15

---

Published as a conference paper at ICLR 2024
Baselines. To accommodate the contemporary model-market scenarios as seen in Table 8 we com- pare the performance of Co- Co-Boosting against two existing methods: FedAvg (McMahan et al. 2017) p and DENSE (Zhang et al. 2022a). To ensure fair comparisons, that require the use of auxiliary public datasets, such as Li et al. (202T), or the modification of the local training phases of each client, as seen in [Dioo al. (2023) and Heinbaugg et al.. (2023). Both of these approaches may not be applicable in real-world scenarios. Moreover, since there is only one communication round, standard FL algorithms based on regularization terms including FedProx (Li et al. 2020b), SCAFFOLD (Karimireddy et al. (020), and FedDyn (Acar et al., (021) have no effect. Furthermore, since data-free one-shot FL can be addressed simply by operating data-free KD methods on the direct ensemble model of all models, similar in (Zhang et al., (022a). we also introduce two prevalent data-free KD methods DAFL. (Chen et al. 2019) and ADI (Yin et 2020)) resulting in F-DAFL and F-ADI. More specifically, F-DAFL. and F-ADI aim to either optimize generator or a noise to mimic the performance of the ensemble teacher model. The data generated from this process is then utilized to distill knowledge into the student (server) model. Additionally, we include FedDF (Lin et al. 2020) using the validation dataset.
omit comparisons with methods
Hyper-parameters. Unless otherwise stated, we conduct experiments with 10 clients and Dir(0.1) (high heterogeneous)) partition. Results are reported averaged across at least 3 random seeds. For each client's local training. we use the SGD optimizer with momentum=0.9 9 and learning rate=0.01. We set the batch size to 1 128 and the local epoch to 300. Following McMahan et al. (2017). we use a simple CNN with 5 layers for SVHN, CIFARIO, and CIFARI00, LeNet-5 (LeCun et al., 1998) for MNIST and FMNIST. All available test data is used to evaluate the final server model (or ensemble). The generator we use is the same as in Zhang et al. (2022a): Chen al. (2019) and is trained by Adam optimizer with a learning rate fg = 1e3 over To = 30 rounds. The distillation temperature in the knowledge distillation stage used in the server model stage is set to 4, while the temperature used in the KL loss in the generator loss is set to 1. The perturbation strength is set to e = 8/25t and the step size y is set to 0.1/n. For the training of the server model (s((), we use the SGD optimizer with learning rate 7s 0.01 and momentum=0.9. The number of total epochs T is set to 500. We maintain the same hyperparameters for all baseline methods.
B.2 COMPARISON OF THE ENSEMBLE MODEL
Our Co-Boosting method, in addition to improving the capabilities of the final server model, also yields an enhanced ensemble model through iterative enhancement of synthetic data and ensemble learning, as illustrated in Algorithm [] The server model obtained through Co-Boosting can be considered as distilled from the progressively improved ensemble and the generated hard samples. Therefore, investigating the performance of this ensemble also highly meaningful. In this section, present the final ensembles obtained by Co-Boosting under various experimental settings corre- sponding to the main paper and compare them with the ensembles used by other methods, FedENS (average logit output). Results on Dir-parted setting, C'olss-arted setting, and different numbers of client settings are demonstrated in Table I0 and 11 respectively
Table 9: Test accuracy of the ensemble in MNIST, FMNIST, and CIFAR- 100 in Dir-parted setting.
Dataset
MNIST
FMNIST
CIFAR-100 100
Method
ce00.0
0=0O.1
0m0.33
om0.05
g=0.11
e=0.3
am0.05
e=0.1
e=0.3
FedENS
81.52+0.89
88. 16+ 1.27
97.75+0.15
45.51++1.05
71 71.3440.66
83.5 83.56-+1.24
21 21.39-0.52
26.56-
35.56+1.00
Co Boosting
14,59
94,4710.74
97.7510.27
1.93
76.52.1.08
8
22.97
27.71 1.68
1.
Table 10: Test accuracy of the ensemble in CIFAR- 10 in the C'els partition setting.
Method
C=2
C=3
C=4
C=5
FedENS
24.43 37.77=1.13
39.67
56.37 1.40 61.80=1.34
60.01 - 1.46
Co- Co-Boosting
56.25+1.22
64.96=1.23
Table 11: Test accuracy of the ensemble in CIFAR-10 across different numbers of clients.
Method
n=5
n=10
n=20
n=50
FedENS
53.27=0.66 58.80+1.15
49.99-10.85
32.60=1.22 43.47=1.42 01.14+0.86
Co- fo-Boosting
59.86+1.776
45.05- 1.08
16

---

Published as a conference paper at ICLR 2024
As can be easily concluded from the above tables, the final ensemble model we get in our proposed method consistently outperforms the baseline FedENS to a large extent. This demonstrates that by finding optimal weight based the hard samples ensemble, which will naturally in turn become a better teacher in the server distillation age. It is noteworthy that in cases where the data partition exhibits higher degree of difficulty (resulting in greater distribution shift among clients), the weighted ensemble we obtain can outperform the baseline weighted average by a more substantial margin. This phenomenon may be attributed the fact that when clients share more similar distribution of data, their knowledge tends to converge, making simple averaging somewhat sufficient encapsulate their collective knowledge. Neverthe- less, this is still not optimal. Our weighted ensemble, learned based on the hard samples, excels in determining more effective weights for aggregating all clients across all settings.
generate, can form much better
B.3 MORE EXPERIMENTS ON MORE DATASETS
To comprehensively evaluate method's ' performance, conduct experiments using widely used Tiny-ImageNet dataset (Le Yang 2015) (200 classes, 500 training samples each) with ResNet18 serving as the client architecture backbone. Using the same settings as other experiments, explore server models based on both CNN and ResNet18 architectures. The results, presented in Table [2] not only confirm the effectiveness of our method but also highlight its sechitecture-annosst feature.
Table 12: Test accuracy of the server model of different methods in Tiny-Imagenet dataset over two architectures and across three levels of statistical heterogeneity (lower is more heterogeneous).
Server
FedAvg
FedDF
F-ADI
F-DAFL
DENSE
Co Co-Boosting
0.05
0.78+
6.44
6.90+-0.08
6.71++0.1
7.38++0.08
CNN
0.1
9.71+-0.20
9.57++0.71
9.77++0.48
9.09--0.43
10.10-00.69
0.3
12.62:10:45
11.94-10.8
12.65=0.54
12.71#10.22
13.83:10.44
0.05
0.70++.20
5.65+-0.30
5.70++0.38
5.68-
5.92+-0.18
8.21--0.12
ResNet
0.1
1.1940.84
8.63
7.01
7.81
8.88+1023
10.29=:043
0.3
2.09++0.52
11.93+0.38
11.58-10.84
02.30+0.36
13.05++0.36
14.35-+0.93
is also of great importance evaluate our proposed Co- Co-Boosting under scenarios involving feature shifts. To this end, extend experiments to include domain generalization datasets, specifically MNIST-M (Ganin et al. (0166) and PACS Li et al. (0177). Experiments follow the settings outlined in Li et al. (023b); Zhang al. (2023), which focus federated domain generalization. In these experiments, each domain is allocated to a different client, and we employ the wwweee-on--dmii--ooe testing technique. We utilized CNN for MNIST-M and ResNet18 for PACS. To further evaluate method's versatility, we tested with both CNN and ResNet 18 as server architectures. Results in Table 13 and Table 14 demonstrate the superior performance of proposed method, which we attribute to the mutual enhancement principle inherent in our approach. The results not only validates our method's model-agnostic feature but also its effectiveness in settings with feature distribution shifts.
Table 13: Test accuracy of the server model of different methods in MNIST-M dataset over tWO
architectures on the unseen domain using leave-one-or test mechanism.
Server
Domain
FedAvg
FedDF
F-ADI
F-DAFL
DENSE
Co-Boosting
MNIST
25.82+1 140
6..38+11.77
66.27+11.48
66.75+1.13
70.81+1.12
82.15:10.83
CNN
MNIST-M
16.40=1.66 41.18:10.34
39,3441.50
39.85+1102
41.35++1.19
42.85
SVHN
18.8 1.791
1..38
45.05:10.3
46.20+1.10
1..23
SYN
45.97+1.96 78.91+0029
78.70
79.83+0.55
0..9
78.67+0.69
AVG
26.77=1.67
58.59++1.12
1..39+
57.87+0.88
59.57++0.93
64.23++100
MNIST
1..6
7..91
69.15112.6
13.25.11.99
84.39
ResNet
MNIST-M M SVHN
43.42:10.54
4..40
41.69+1.34
13.71+1.56
1..2
47.914-0.88
48.02-10.70
18.46+1148
47.27+1.119
13.75
SYN
80.48-10.51
1..4
80.55
80.03+0044
1..93
AVG
61.12.0..8
60.2111.17
19.96111.44
61.07+1.20
65.33
17

---

Published as a conference paper at ICLR 2024
Table 14: Test accuracy of the server model different methods in PACS dataset over two architec- tures on the unseen domain using leave-one-out test mechanism.
Server
Domain
FedAvg
FedDF
F-ADI
F-DAFL
DENSE
Co-Boosting
33.80.10:45
32.71
34.25-2.12
35.68.1.83
50.77.1..78
CNN
A
23.36-+0.43
21.17+0.52
21.14=0.78
22.41=00339
2..54:::118
C
29.65+0.30
29.32+0.83
29.61=0.86
29.01=0.99
37,59=11441
S
30.45+0.49
29.85=0.266
28.36=0.76
30.17=0.34
30.67:22.77
AVG
29.12
28.26++0.79
28.34-1.13
29.34++0.89
35.64--
11.32:1..33
37.09110.45
36.5011.04
37.55_1.78
37.19
1..4311.72
ResNet
A
18.5042.88
23.76.10885
22.95:1.82
22.92=1.89
24.83:11.49
26.76:11.22
C
16.6011.22
29.6210.48
28.5310.89
29.35=1.68
31.781:1.00
36.73
19.6512..3
30.2011.42
29.7711.85
29.67=0.76
32.00
35.35 1.74
AVG
16.52+2.18
30.16+0.55
29.31
29.87=1.53
31.45=1447
37,56+ 1.48
B.4 MORE EXPERIMENTS LOCAL MODEL HETEROGENEITY SETTING
Similar to the main paper, evaluated the performance of the proposed Co-Boosting against other baselines under the same client model heterogeneity setting, while varying the server model archi- tecture. Results are shown in Table 15] all experiments done in 5-client Dir(0.1) parted setting. While it natural to observe variations performance among different architectures, this can be at- tributed the of the same hyperparameters across all architectures and the inherent differences in representation capacity among them. However, is noteworthy that Co-Boosting consistently outperforms other baselines regardless of the server architecture. This further underscores the effec- tiveness of the proposed Co-Boosting technique for enhancing both data and ensemble.
Table 15: Test accuracy of server model in different architectures in CIFAR-10 across three levels of statistical heterogeneity under a heterogeneous client model setting.
Architecture
Local
FedDF
F-ADI
F-DAFL
DENSE
Co-Boosting
CNN1
41.06++1.4
52.96+ 1.66
1.39
51.73+0.90
52.66+1.43
56.10+-1.64
CNN2
30.03+00.69
44.09
43.38-1.73
43.13+1.61
43.82+1.49
45.82++1.36
0.05
MobileNet
37.32:11.52
19.47
50.55-1.31
49.48.10.66
50.35111.3
12.97
ShuffleNet
38.0911.77
51,05.1..23
48.75-0.68
49.95
50.01
1.94
CNN1
18.05++1.68
07.66+00.7
56.18=1.67
1..04+1.554
58.31+1.21
61.87=1.86
CNN2
39.90.10.25
16.2111..3
43.03=1.56
45.56-11.94
45.88
18.31
0.1
MobileNet
44.72::1.39
50.25+1.04
49.36=1.44
51.91=10.98
51.47
54.75=
ShuffleNet
47.12::1.71
53.15+1.68
52.88=1.41
12.04:11.36
52.25+11000
55.72:10.77
CNNI
60.16+0..5
6..0
68.94-0.92
67.55+1.22
67.89+1.08
70.06+0.92
CNN2
51.7910.775
18.5211.41
57.49-0.86
17.43
58.15
62.45
0.3
MobileNet 55.54:11.11
60.6010.38
60.42-1.07
60.11:11.12
61.1810.84
6..2011.07
ShuffleNet
55.49++0.80
61.82+0.89
60.30-0.73
59.98+1.93
61.19+1.24
63.22++1.19
B.5 COMBINATION WITH ADVANCED LOCAL TRAINING
To alleviate the non-III data problem in the Federated Learning (FL) setting, recent works (Qu al. 2022 Caldarola et al. 2022; Dai et 2023) have shown that the use of the Sharpness Aware Minimization (SAM) technique can be beneficial. It is suggested that incorporating this technique into local training can lead to improved server aggregation. In this section, to investigate the per- formance of methods utilizing SAM for local training in the one-shot federated learning setup, we conduct experiments on SVHN and CIFAR-10 datasets. The experiments are done with clients under Dir(0.05) partition scenario. From Table [6] is evident that introducing SAM-based tech- niques during the local training phase, as compared to the baseline in Table T indeed enhances the performance of various methods. This improvement is attributed to the potential alignment achieved during local training, which ultimately leads to better server performance upon aggregation. While, proposed method, Co- Co-Boosting, continues outperform all the baselines, highlighting the
18

---

Published as a conference paper at ICLR 2024
periority of our approach. Moreover, this shows that our method is not dependent the specifies of the local training phase. With a better local training method, our approach can further improve.
Table 16: Test accuracy of the server of different methods when combining advanced local training.
Dataset
FedAvg
FedDF
F-ADI
F-DAFL
DENSE
Co-Boosting
SVHN
43.46:1.11 23.3943.93
6.167
1..9411.35 37.88.1883
1..1811.59 38.53:10.60
62.0011..47
6..3411..9 49.04.10.73
CIFAR-10
39,54 1.69
11.14::..65
B.6 MORE EXPERIMENTS WITH HEAVIER MODELS
In this section, we conduct further experiments using larger models, VGGII (Simonyan & Zisser- man 2014) and ResNet18 (He et al. 2016), on CIFAR-10. The experiments were carried out with I0 clients federated setting partitioned according to a Dir(0.05) distribution, which represents a sig- nificantly large difference in data distribution. Results in Table 17 show that our method continues to perform well with heavier models. Our proposed Co-Boosting outperforms the existing baselines by significant margin, approximately 10% in both cases of VGG and ResNet architectures.
Table 17: Test accuracy of server model with VGG and ResNet in CIFAR-10.
Model
FedAvg
FedDF
F-ADI
R-DAFL
DENSE
Co-Boosting
VGG
34.90=12.70
52.92
12.28+1.80
1..84+1.29
53.32
63.37
ResNet
33.55:3.559
52.62+0.93
1..5
52.65
52.98+1811
62.18 1.92
B.7 SENSITIVITY ANALYSIS OF HYPERPARAMETERS
Step size μ. a is the step size used when adjusting the ensemble weights of each client. It controls the range of weight changes based the synthetic data. As be from Table 18 when j is very small, the performance will decline because the range of weight exploration is not large. When is of moderate size, it does not have significant impact the final ensemble and the final server model. This is because after each weight adjustment, a normalization operation is performed to limit within certain range. Overall, we recommend using 0.1 0.1/n as the step size.
Table 18: Analysis of under 10-client Dir(0.1) Pir(O..)-parted setting in CIFAR-10.
0.005
0.01
0.05
0.1
Ensemble
19.13+1..38
19.86+1.776
59. 19.18-11.9
59.20 1.04
Server
56.46:11.79
57.0910.94
57.4412.01
17.52
Perturbation strength c. c is the parameter that adjusts the difficulty of synthesizing data on the fly during data distillation. It controls the range of pixel-level changes on the image plane. As can be seen from Table 19, when is small, it cannot generate sufficiently diverse hard samples, thus not maximizing the extraction of information from the samples. Conversely, when the is large, causes significant damage at the image level, potentially destroying semantic information. Therefore, recommend using a moderate epsilon value. In this paper, we use 8/255 as the default.
Table 19: Analysis of e under 10-client (1r(00005--prtee setting in SVHN.
1/255
2/255
4/255
8/255
16/255
32/255
1.29
63.53
64.71+1.20
65.40=0.86
65.58+11.20
64.25:t1.32
19

---

Published as a conference paper at ICLR 2024
B.8 COMPARISON WITH MULTI-ROUN FEDERATED LEARNING
In this section, we run FedAvg (cMMaan et al. 2017). and Scaffold (Karimireddy et al. 2020). two popular and efficient algorithms for 200 communication rounds. There 10 clients with 10 local epochs in each round. Fig. shows that our approach can outperform the multi-round FedAvg algorithm across various settings. Furthermore. our method remains competitive with the results of the Scaffold algorithm, which undergoes 200 rounds of communication. However, it's crucial note that the performance improvement multi-round algorithms is result of multiple rounds of information exchange and model alignment. The multi-round paradigm entails the possibility of client dropouts and vulnerability to potential attacks. contrast, our algorithm necessitates only single communication round, making it exceptionally suitable for real-world applications.
FodAvg
Scaffold
Co-Boosting
80
80
Text Accumey (86
Text Accuncyy (%)
60
60
60
Accunccy
40
60
40
20
20
20
50
100
150
200
100
200
100
150
200
Communication Rounds
Commonication Rounds
Comomnication
CIFARIO Dir(0.05)
CIFARIO Dir(0.1)
CIFARIO Dir(0.3)
Figure 3: Comparison with multi-round federated learning methods.
B.9 VISUALIZATION SYNTHETIC DATA
We conduct data visualization for the synthetic data of our proposed Co-Boosting on the MNIST and CIFAR-10 datasets Fig. 4 and Fig. [}) where digits 0 represent 10 different classes. There are a total of five rows corresponding to images generated at the 100th. 200th, 300th, 400th, and 500th global model training epochs. For each epoch, we randomly select images for display. From the figures, it can be observed that the generated data does not hold significant practical value to the human eye. However, is sufficient for machine learning models learn robust classifier.
2
3
4
6
7
9
Figure 4: Visualization of synthetic data on MNIST dataset.
20

---

Published as a conference paper at ICLR 2024
2
3
4
5
6
7
8
9
Figure Visualization of synthetic data on CIFAR-10 10 dataset.
21


Reviews:
[{"review_id": "v35Nk9H9RT", "reviewer": "Reviewer_WMX4", "summary": "This paper proposes Co-Boosting to improve the performance of one-shot federated learning. The existing one-shot federated learning algorithm requires training a synthesized-sample generator on the server using the ensemble model of client models. The core ideas of this paper are twofold: 1. Fine-tuning the weights of the ensemble model using synthesized samples to improve its ability to classify these synthesized samples. 2. When training the generator, utilizing the output of the ensemble model to calculate the difficulty of samples, assigning higher loss function weights to hard samples. These two points can be iteratively applied, mutually enhancing each other. The experiments conducted on datasets like SVHN and CIFAR10 demonstrated that Co-Boosting has a certain superiority over existing approaches.", "questions": "See Weaknesses.\nIn Algorithm 1 Co-Boosting, should Lines 16 to 18 be placed after Line 19?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper pays attention to the relationship between generator training and ensemble model, which is an interesting perspective.\n2. The overall paper is well organized and easy to follow.", "weaknesses": "1. The novelty of the proposed method is relatively insufficient. The overall method is composed of several existing tricks, lacking a comprehensive core framework or a more in-depth analysis.\n2. The one-shot federated learning encompasses various types of methods, including training generators [DENSE, FedDF], uploading distilled datasets [1], utilizing existing pre-trained models [2], and so on. However, this paper specifically focuses on methods related to training generators. This point should be clearly discussed in the introduction and related work sections.\n3. The experiments are insufficient. The most complex model architecture discussed in this paper is a 5-layer CNN, and the most complex dataset is CIFAR100. This raises doubts about the practical value of this one-shot method.\n4. In the experimental section, this paper focuses on class distribution. However, this alone may not be sufficient to demonstrate the effectiveness of the proposed approach. This is because when there are differences in class distribution, the weights of the ensemble model will naturally play a crucial role. For instance, if a model does not support category 'a', it would be necessary to reduce the weight assigned to this model. I would recommend the authors to incorporate experiments involving feature distribution differences, for instance, by utilizing the DomainNet dataset.\n\n[1] Zhou, Yanlin, et al. \"Distilled one-shot federated learning.\" arXiv preprint arXiv:2009.07999 (2020).\n\n[2] Yang, Mingzhao, et al. \"Exploring One-shot Semi-supervised Federated Learning with A Pre-trained Diffusion Model.\" arXiv preprint arXiv:2305.04063 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QOaIdBrO9z", "reviewer": "Reviewer_qWAT", "summary": "This work focuses on one-shot federated learning, following the common practice that uses data-free distillation with local models. Co-Boosting is proposed as an algorithm in which the synthesized data and the ensemble model collaboratively enhance one another in a progressive manner. To elaborate, the algorithm produces challenging samples by training a data generator using a loss function that is re-weighted for the samples and includes an adversarial loss component.", "questions": "Recent works in multi-round FL have suggested to use of a pre-trained model for initialization [A, B]. A pre-trained model is typically publically available on both the server side and the user side and might significantly help the convergence speed given limited rounds or distillation. Could the authors provide any insights or discussions about this case?\n\n[A] When foundation model meets federated learning: Motivations, challenges, and future directions. ArXiv 2023\n[B] On the Importance and Applicability of Pre-Training for Federated Learning. ICLR 2023", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The idea is quite straightforward and sound.\n* Nice and clean writing\n* Experiments are complete with comparison to baselines and ablations, across common datasets and FL settings", "weaknesses": "* The related works could be improved. Specifically, \n1) FL works about distillation and ensemble with or without a server dataset: there are many follow-up works after Lin et al. (2020). More works about model ensemble or aggregation would be better. \n2) Since the idea is about co-learning, it will help if a discussion about centralized co-learning techniques can be added; such as\nDeep co-training for semi-supervised image recognition. CVPR 2018\n\n* The experiments are a bit lack of insights. The paper could be stronger if deeper analysis beyond accuracy can be provided. For example, what examples are considered hard? What are the training dynamics in terms of model weighting or sample re-weighting?  Any failure cases? \n\n=== Minor issues ===\n* Suggestions on fig 1: no label for the y-axis\n* Experiment tables: It is better to include the upper bound such as multi-round FedAvg or centralized training\n* \"round\" terminology: I understand the paper is about one-shot FL but sometimes the term \"round\" is used in the text causing me slight confusion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "58soGtUnRU", "reviewer": "Reviewer_vHVL", "summary": "This paper developed Co-Boosting, a new framework in which synthesized data and the ensemble model mutually enhance each other progressively. Extensive experiments demonstrated the good performance of the proposed method.", "questions": "How to evaluate whether the labels generated in the synthetic data are accurate and correct? Whether the pseudo label affects the results a lot?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Develop a new framework that can enhance synthesized data and the ensemble model mutually\n2. The generation of hard samples seems interesting and helpful.\n3. Extensive experiments have shown the proposed method outperforms the baselines.", "weaknesses": "1. Fig 1 (a) needs to be elaborated to help readers better understand the 3 steps. Please elaborate on these steps in the caption.\n\n2. Missing one-shot FL baselines. The introduction discussed two methods, Heinbaugh et al. (2023), and Diao et al. (2023). However, the authors did not compare them in the experiments.\n\n3. How to evaluate whether the labels generated in the synthetic data are accurate and correct?\n\n4. The technical contribution is not very significant. It seems to combine some existing works. For instance, generating hard samples motivated by existing work like Dong et al. (2020) and Li et al. (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Qm8ZQcWzh8", "reviewer": "Reviewer_rouU", "summary": "The paper proposes a method for one-shot federated learning named \"co-boosting\" that aims to improve ensemble and data quality for training the server model. The main contribution lies in the utilization of adversarial samples for training the server-side model.", "questions": "\"Though samples synthesized using Eq.(8) are hard to fit for the current ensemble model, their difficulties for the server model are still lacking\". If the synthesized samples are hard to fit by the ensemble, why make it even more harder for the server model to learn by introducing adversarial samples? \n\nWhat kind of issues arise when one has multi-round FL? This question is related to the following statement: \"mitigating errors arising from multi-round.\" Is OFL a silver bullet? when it can fail ? Please explain in Limitation section to better contextualize when using OFL is useful. \n\nWhere does the generator G(·) come from? What is the architecture, and how is it trained? It is completely unclear how data is generated for knowledge distillation eq. 4 to train the server model. I suggest the authors write a separate section explaining it in detail since it is a main component of the proposed method. Is it pure random noise? If yes, why not use [1]?\n\nHow does Co-Boosting not require model transmission? From eq. 2, it seems like the ensemble is created with FedAvg in a single round. But as per section 3.3, the ensemble is not created by weighted averaging of client models? Is that correct? So logits are averaged on synthetic data, which acts as an ensemble for the server model, what is correct?\n\nWhat is GHM? Please provide the full form and a minimal description to improve the readability of the paper. There are already too many references/links to prior work and moving back and forth makes it hard to follow the method.  In eq. 5, does $\\bf{x}$ represent a synthetic sample?\n\nRegarding the statement \"searching for the optimal ensembling weights of each client's logits,\" it is unclear how the search is performed.\n\n[1] Baradad Jurjo, Manel, et al. \"Learning to see by looking at noise.\" Advances in Neural Information Processing Systems 34 (2021): 2556-2569.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "On a high level, the idea of OFL and co-boosting is understandable if one wants to avoid multi-round FL then the method is useful. The evaluation is thorough across datasets, architectures, baselines and varying federated settings. From results it looks like method performs exceptionally well compare to the baselines at least on MNIST, FMNIST and SVHN.", "weaknesses": "The important details of synthetic data generation and ensemble creation are vague and overly complicated, which makes the paper very hard to follow and identify the key components of the method. As a result, it is difficult to judge if the method is indeed OFL. The evaluation is thorough, but if it is not conducted when the clients' models are trained only once, then I have serious concerns about what the paper is claiming and how the experiments are conducted.\n\nSection 3.2 is extremely hard to follow and lacks structure. It is unclear whether the data is generated on the fly during the training of the server model or if it is pregenerated.\n\nAlgorithm 1 does not provide any useful information beyond what has already been described in earlier sections. Please highlight what happens on the server and what happens on the client device. If it is truly OFL, what does the epoch correspond to? Is it the server model training? This question also relates to Figure 1 (a) and (d). If there are multiple epochs of both the client and server, how is it considered OFL?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)