Data-Free One-Shot Federated Learning Under Very High Statistical Heterogeneity
Authors: Clare Elizabeth Heinbaugh
Emilio Luz-Ricca
Huajie Shao
Venue: ICLR 2023 poster (2023)
Keywords: One-Shot Federated Learning
Statistical Heterogeneity
Model Heterogeneity
Variational Autoencoder

Abstract: Federated learning (FL) is an emerging distributed learning framework that collaboratively trains a shared model without transferring the local clients' data to a centralized server. Motivated by concerns stemming from extended communication and potential attacks, one-shot FL limits communication to a single round while attempting to retain performance. However, one-shot FL methods often degrade under high statistical heterogeneity, fail to promote pipeline security, or require an auxiliary public dataset. To address these limitations, we propose two novel data-free one-shot FL methods: FedCVAE-Ens and its extension FedCVAE-KD. Both approaches reframe the local learning task using a conditional variational autoencoder (CVAE) to address high statistical heterogeneity. Furthermore, FedCVAE-KD leverages knowledge distillation to compress the ensemble of client decoders into a single decoder. We propose a method that shifts the center of the CVAE prior distribution and experimentally demonstrate that this promotes security, and show how either method can incorporate heterogeneous local models. We confirm the efficacy of the proposed methods over baselines under high statistical heterogeneity using multiple benchmark datasets. In particular, at the highest levels of statistical heterogeneity, both FedCVAE-Ens and FedCVAE-KD typically more than double the accuracy of the baselines.

Published as a conference paper at ICLR 2023
- ONE-SHOT FEDERATED LEARNING UNDER VERY HIGH STATISTICAL HETEROGENEITY
Clare Heinbaugh", Emilio Luz-Rieca', Huajie Shaot Department of Computer Science, William & Mary, VA, USA {cehei nbaugh, e eluzri cca, hshao) @wm edu 'Equal contribution, Corresponding author
ABSTRACT
Federated learning (FL) is an emerging distributed learning framework that collab- oratively trains shared model without transferring the local clients' data to cen- tralized server. Motivated by concerns stemming from extended communication and potential attacks, one-shot FL limits communication to a single round while attempting to retain performance. However, one-shot FL methods often degrade under high statistical heterogeneity, fail promote pipeline security, require an auxiliary public dataset. To address these limitations, we propose two novel and its extension FEDCVAE- KD. Both approaches reframe the local learning task using a conditional vari- ational autoencoder (CVAE) to address high statistical heterogeneity. Further- more, FEDCVAE-KE leverages knowledge distillation to compress the ensemble client decoders into single decoder. We propose method that shifts the center of the CVAE prior distribution and experimentally demonstrate that this promotes security, and show how either method can incorporate heterogeneous local mod- els. We confirm the efficacy of the proposed methods over baselines under high statistical heterogeneity using multiple benchmark datasets. In particular, at the highest levels of statistical heterogeneity, both FEDCVAE-EL and FEDCVAE- KD typically more than double the accuracy of the baselines.
data-free one-shot FL methods: FEDCVAE-
INTRODUCTION
Traditional federated learning (FL) achieves privacy protection by sharing learned model parameters with central server, circumventing the need for centralized dataset and thus allowing potentially sensitive data to remain local to client devices (McMahan et al.) 2017). FL has shown promise in several practical application domains with privacy concerns, such as health care, mobile phones, and industrial engincering (Li et al. 2020a). However, most existing FL methods depend sub- stantial iterative communication (Guha et al. 2019 Li et al., (2020b), introducing a vulnerability to eavesdropping attacks, among other privacy and security concerns (Mothukuri et al. 2021).
One-shot FL has emerged to address issues associated with communication and security in standard FL (Guha et al. 2019). One-shot FL limits communication in scenarios like model markets, where models trained to convergence are sold with no possibility for iterative communication during local client training (Li al. b02 In high impact settings, like health care, data could be highly heterogeneous and computation capabilities could be varied; for example, health care institutions could have different prevalence rates of particular diseases or no data on disease and substantially different computing abilities depending on funding (Li et al.) 2020a). Furthermore, fewer communications rounds means fewer opportunities for eavesdropping attacks. While results in one-shot FL. are promising, existing methods struggle under high statistical heterogeneity, non-indepenndnnlly and
single round, which is more practical
(non-lID) data, (i.e., Zhou et al. (2020)
Zhang et al. (2021)) or do not fully consider statistical heterogeneity (i.e., Guha et al. (2019) Shin
et al. 22020)
Li et al. ) Additionally, most do not consider pipeline security (i.e., Shin Li et al. 2021b) Zhang et al. (202T).. Furthermore, Gu.c.,Guhaa often required to achieve satisfactory performance one-shot FL (i.c., Guha et al. (2019), Li et al. (002b)). which may be difficult obtain in practice (Zhu et al. 2021
et al. 2020
auxiliary public dataset is

---

Published as a conference paper at ICLR 2023
Conditioning class
0
3
4
5
7
8
9
7
7
7
4
4
/
7
7 '
*
7
4
2
9
4
7
4
7
4
8
7
7
7
9
4
7
7
7
H
4
Class label
Client
5
6
3
3
0
3
6
6
3
6
3
3
6
3
6
0
6
3
3
6
5
3
6
6
6
3
1
6
Client 4
1
0
0
2
4
5
7
8
9 9
0
2
3
4
5
6
8
0
1 2 4
5
6
7
0
2
3
4
5
6
7 8
9
Client ID
Server decoder
Figure 1: Motivating our proposed methods, FEDCVAE-ENS and FEDCVAE-KD, using the MNIST dataset an example. In cases of very high statistical heterogeneity, each client will only observe one or two the ten available classes, as seen on the left where the size of each dot is proportional to the number of samples. For example, client 2 only observes 4's and 7's, resulting in a client decoder that can expertly generate these digits. Note that columns are shown in order of conditioning class (digits 0-9). Similarly, client 4 is expert in 3's and 6's. In FEDCVAE-R -.D, our lightweight knowledge distillation training procedure compacts local learning into a single server decoder, evidenced by the high-quality samples from all available classes (digits 0-9). This server decoder can then be used for any downstream task, e.g., classification.
To address these issues, we jointly propose FEDCVAE-ENT and FEDCVAE-KD. KD, two novel data- free one-shot FL models that reframe the local learning task using conditional variational autoen- coders (CVAE). Because CVAEs can easily learn a simplified data distribution, both methods train CVAES locally capture the narrow conditional data distributions that arise in the high statisti- cal heterogeneity setting. Figure T shows how client decoders become experts in the few classes that they observed. These decoders are ensembled (FEDCVAE-EL) (FEDCVAEKKD) More specifically, FEDCVAE-KD aggregates the models using a lightweight knowledge distillation procedure; client decoders are teachers, and the server decoder is the student. Figure shows shows images generated by the server decoder.
or compactly aggregated
Thorough experiments on multiple benchmark datasets (MNIST, FashionMNIST, SVHN) demon- strate the superiority of FEDCVAF-ENS and FEDCVAE-KI over other relevant one-shot FI. meth- ods in the high statistical heterogeneity setting. particular, FEDCVAF E. and FEDCVAF-KD obtain more than 1.75x the accuracy of the best baseline method for MNIST, more than 2x the accuracy for FashionMNIST, and more than 2. 75x the accuracy for SVHN under extreme statistical heterogeneity (i.e., clients only observe uploaded to the server, we propose a method to shift the center of the CVAE prior distribution. We show that without knowing the center of the prior, eavesdropping attacker cannot train perfor- mant classifier, thus promoting pipeline security.
or two classes). Furthermore, to protect the decoders
one-shot FL methods targeted the high statistical heterogeneity setting that: (1) perform substantially better than other baseline methods in this setting, (2) demon- strate invariance to the number clients, (3) are data-free and can be applied to any downstream task requiring a labeled dataset, (4) allow for heterogeneous local model architectures, and (5) extend to promote pipeline security. To the best of our knowledge, we are the first thoroughly address very high statistical heterogeneity in one-shot FL.
In sum, contributions
2 PRELIMINARIES
Conditional Variational Autoencoders. A variational autoencoder (VAE) is a probabilistic gen- erative model that attempts to learn the distribution of data samples (Kingma Welling 2014).
2

---

Published as a conference paper at ICLR 2023
A VAE is latent variable method that models the joint distribution po z) of data sample x E X C RD and latent variable z E 2 C R", where usually d << D. This joint can be factorized as (o(x. z) Po (x)))z) where the prior p(z) is usually chosen to be a multivariate standard normal distribution, i.e., p(z) = (0, 1). The posterior po (2|x) is approximated via an inference model q (z x) called the encoder and the model for the conditional likelihood px(x)z) is called the decoder. In our case, both the encoder and decoder are deep neural networks parameterized by Φ and 0. respectively. Conditional VAES (CVAEs) extend the basic VAE by conditioning the encoder and decoder on the one-hot encoding of the class label y E y = (1.2. ...., K} corresponding to data sample x, resulting in the conditional encoder 9 (z) x. y) and conditional decoder po px(x A CVAE is trained by maximizing the variational lower bound:
[( 0: x. y) = -Kx. (1 (z)x,y) p(z)) + (x (x
(I)
which bounds the conditional marginal likelihood of the data logy()ly). Here, (xL(-) represents KL-divergence.
Knowledge Distillation. Knowledge distillation (KD) aims extract information from a trained teacher model to train a separate lightweight student model (Buciluă et al.) 2006 |inntnn et al. 2013). Typically, the student model is trained by minimizing the discrepancy between student and teacher logits generated using suitable auxiliary dataset (Hinton et al. 2015); KL-divergence is often chosen as the measure of discrepancy. Some works ensemble teacher models, using the average of teacher logits in an attempt to compact ensemble knowledge into single student model (Anil jet al. 2018; Dvornik et al., 2019: Furlanello et al.] 2018). Several works have integrated KD into FL to mitigate privacy risks, reduce upload costs, or regularize local learning using an ensemble-of- teachers approach (Lin et al. 2020; Zhu et al. [222]] Guha et al. (019). However, KD approaches in FL usually require auxiliary public dataset with similar properties the distributed dataset, which may be difficult to obtain in practice (Zhu et al. (202I).
One-shot Federated Learning. In the federated setting, we have set of clients C, with m |C| clients in total. Each client k has a local private dataset D* = {(x;, yt) with nu = |D repre- senting the number of data samples belonging to user kt Traditional FL methods assume each client has local differentiable model fw (-), usually a deep neural network parameterized by w It is typically assumed that the server and clients can communicate over multiple rounds. However, in the one-shot FL. setting communication is restricted to single round, which severely limits com- munication costs but also increases the difficulty of the distributed learning task (Guha et al.. 2019). Notably, existing one-shot FL methods either ignore the issue of statistical heterogeneity (i.e.. Guha cct al. (2019)), fail to comprehensively explore the effect of statistical heterogeneity on performance (i.e., (Shin al. (2020) and Li et al. (2021b)), or degrade substantially at even moderate levels of statistical heterogeneity (i.e., ZZhou et al. (2020) and Zhang et al. (2021).
3 TACKLING VERY HIGH STATISTICAL HETEROGENEITY IN ONE- FL
We jointly propose FEDCVAE-EN and FEDCVAE-KD, one-shot FL methods that do not require an auxiliary public dataset for server-side training (they are fata-free). They address issues caused by high statistical heterogeneity by reframing the learning task using CVAES and account for model heterogeneity by allowing different CVAE architectures across clients. FEDCVAE ENS is described in Al Algorithm 2 (Appendix A and visualized in Figure 6 (Appendix A). with FEDCVAF- KD de- scribed in Algorithm [ and visualized in Figure 2 We discuss privacy- and
extensions in Appendix
B and Section 3.3 respectively.
3.1 OVERVIEW
Figure illustrates the overall framework of the proposed one-shot method FEDCVAE-KD and Figure (Appendix A shows FEDCVAB-ENS. Specifically, clients first train CVAES locally on their private data. Next, each client's trained decoder parameters and local label distributions are uploaded to the server once; this is the only communication round. Then, the server generates sam- ples from the client decoders according to the client's 's local label distribution. Generating samples based on client's label distribution ensures that each client presents samples from the classes that they know best. In REDCVAE-ENG these generated samples are directly used to perform a down- stream task, e.g., train a classifier. In FEDCVAE these samples are used to train single server
3

---

Published as a conference paper at ICLR 2023
Communication
clients
Server
O E
0
|
Train server decoder via KD
decoder weights
O
-ل
D
O
D
Train classifier decoder samples
4
EAD
D 4
The full pipeline for one of our proposed methods, FEDCVAE-KD. Here, E, D, and represent "encoder," "decoder," and "classifier" models, respectively. First, clients train CVAEs on their private local datasets. Then, the server uses uploaded client decoder parameters and local label distributions to train server decoder using knowledge distillation (KD). Finally, synthetic labeled samples from the server decoder are used to train a classifier.
Figure 2:
decoder via KD. Thus, the single conditional decoder can be used as a compact labeled dataset to Because FEDCVAB-R extends
perform task like training classifier, as depicted Figure 2
EODC,, we leave the description of FEDCVAE-ER ENS to Appendix A
3.2 FRDCVAF-KD. DECODER AGGREGATION USING KNOWLEDGE DISTILLATION
FEDCVAE-KD trains a CVAE fwi (-) for every client & E C convergence on their private local dataset by solving Equation | this CVAE is parameterized by w = [Ø 0*), with encoder Esr (-) and decoder Dæ (-). Then, clients communicate their decoder weights 0 and label distributions (y) to the server; in practice, this simply requires upload of client label counts as Zhu et al. 202T). This completes the single communication round.
Now we move to the server. Intuitively, if a CVAE observes samples primarily from only a few of the K total classes (as is likely when data is highly heterogencous), the CVAE will become "expert" in the simplified data distribution over those few classes, To ensure each client only presents its highest-quality samples,
generate conditioning classes y by sampling from the client's local label distribution, i.e., y ~ (y). Then, to sample from client decoders, we sample a latent vector from the prior (i.e., z (0, I and obtain synthetic data sample from client & using x Dox (z)
The trained client decoders act as the teacher models, conveying their aggregate knowledge of how to map from latent space to data space To achieve this, we generate np total KD training samples, with DEns defined as the combination of client subsets D {&R,yf
single student server decoder, parameterized by Ø
2/)) [=//m] Then, train the server match the teacher's mapping of (z), y.) to & by minimizing reconstruction loss:
yf 2
(0: z² x*x = g (Dos (z":
which penalizes the dissimilarity g(-) in data space between the synthetic data sample generated by the server decoder Das ( and the client decoder sample &*. To facilitate comparison with existing works in one-shot FL, we use the trained server decoder to generate an IID labeled dataset De of ne samples train the server classifier /wg (.), parameterized by S
In practice, find it useful focus the highest density region of the prior and instead sample from truncated standard normal distribution with tight symmetric bounds.
"If classification is the downstream task, note that rather than train auxiliary classifier in the server, the server decoder's conditional likelihood model p(x z. y) could be used directly in the generative classifier
p(y[x) pe)()d p(y). We leave further exploration of this modeling direction future work.
p(0)
4

---

Published as a conference paper at ICLR 2023
Algorithm 1 FEDCVAE-KD in the one-shot FL setting, Tr. represents the number of local training epochs. The server decoder parameters are 05, with KD training epochs TKD, number of KD training samples p. KD loss (<D(.), and KD learning rate 7KD. The server classifier parameters are w with training epochs Tc, number of training samples nc, classification loss (c(-), and learning rate TJC. C is the set of clients.
1: procedure SERVER
2: for each client & E C in parallel do CilentLoccalUpaarl
3:
0 (y)
T2)
See Algorithm in Appendix A
4: Generate samples from client Din = {(*,. )] ! and label distribution p² (y)
|np/ml using client decoder Dø (+)
5:
Combine client subsets into KD dataset DEns == D U U for server decoder epoch i to TKD do for mini-batch & Fas do
Ens U... U D
6:
7:
8:
0 f- 0 TKD Vas (KD(0";))
9:
Generate IID labeled dataset De b by sampling from trained server
decoder ae (+)
10:
for classifier epoch i to Tc do for mini- batch 6 C De do
11:
12:
g t w nc V wg (
Because FEDCVAE-KO ensembles client decoders create labeled dataset DEas to train the server decoder, each client can have a unique CVAE model architecture, accommodating each client's computational limitations. Furthermore, the decision of the classifier architecture can be deferred until after FL is finished and will not affect the learning procedure. FEDCVAE-KI can be applied any task that requires labeled dataset, which is more general than classification; there is no commitment to a particular terminal task before learning occurs. While we do not explore the extended communication setting, we note that FEDCVAE-KI extends naturally by communi- cating the server decoder parameters obtained through KD to all clients and repeating the outlined procedure for non-terminal communication rounds.
3.3 SECURITY-PROMOTIE EXTENSION
We define a secure pipeline as one where an outside attacker who obtains transferred data cannot train a performant classifier (Zhou al. 2020). In the of FEDCVAE-ENS and FEDCVAE- KD, an attacker who intercepts all client decoders and local label distributions should not be able to generate the high quality samples necessary to train high quality classifier.
CVAEs use a prior distribution over latent space to train the encoder and decoder models. While multivariate standard normal distribution is typically used for convenience, any normal distribution is acceptable. To promote security, we propose to shift the center of the prior distribution μ to random position in real space (i.e., μ IR,), which can be communicated offline or via encryp- tion methods between server and clients Zhou et al. 2020). As shown in Figure 7 (Appendix B). sampling latent vectors too far from the center of the normal prior produces qualitatively poor data samples, deterring cavesdropping attackers who have knowledge of μ. We conduct experiments to verify the effectiveness of this extension.
4 EXPERIMENTAL EVALUATION
4.1 SETUP
Benchmark Datasets. To validate FEDCVAE-ENS and FEDCVAE-KD, we conduct experiments on three image datasets that are standard in the FL literature: MNIST (Lecun et al. 1998), Fashion-
MNIST
Xiao et al. 2017), and SVHN (Nezeer et al. 2011
Datasets are described in Appendix
The code used to implement our proposed methods and carry out all experiments is included in the follow- ing public repository: https:: htttps://github..c
fed_cvae
5

---

Published as a conference paper at ICLR 2023
0.001
0.05
Cass labe!
Class label
Cass
Client
Client ID
Client
Figure 3: Example distributions class labels for MNIST with m = 10 clients over multiple levels of statistical heterogeneity The size of each dot is proportional to the number of samples.
To simulate statistical heterogeneity, we the Dirichlet distribution generate disjoint IID client training datasets as Hsu et al. (2019) and Lin et al. (2020). Specifically, we sample pa Dir(a) and allocate P proportion of class to client k. The parameter controls the level of non-IID-ness, with lower inducing more skewed label distributions across clients, To illustrate the effect of on dataset partitions across clients, we visualize the distribution of labels across m = 10 clients for a {0.001, 0.01, 0.05} in Figure
Baseline Methods. We compare the performance of FEDCVAE-ENT and FEDCVAE-KI KD in the one-shot data-free FL setting against two existing methods: FEDAVG (McMahan et al. 2017) and a method proposed in Guha et al. (2019), which we call FEDONESHOT. HEDÓNESHOT ensembles the predictions of select uploaded client classifiers using sampling procedure; because we consider substantially less clients than Guha et al. (2019). we disregard sampling and use all clients in the ensemble. There are recent FL methods that are not appropriate in our proposed setting. The one- shot methods proposed in et al. (2021b) and Shin et al. (2020) are not applicable because their reliance on public auxiliary data for server-side training methods are not appropriate because they depend on an auxiliary dataset (i.e., FEDDF (Lin et al, 2020h) focus regularization (i.e., FEDGEN (Zhu et al. 2021). FEDPROX (Li al. (020c)), SCAFFOLDD (Karimireddy et al.) 2020), FEDNOVA [Wang et al. 2020)) which is incompatible with the one-shot setting.
fine-tuning Similarly, many standard FL
Configurations. Unless otherwise stated, we use m 10 clients, a = 0.01 (very heterogeneous), and report average accuracy across seeded parameter intializations one standard deviation. The data partition is fixed unless otherwise stated. Following Zhu et al. (022)). we distribute 50% of the available training data to clients for MNIST and FashionMNIST. and 100% for SVHN. All available test data is used to evaluate the final server classifier (or ensemble for FEDONESHOT), We adopt the same convolutional classifier architecture as McMahan al. (2017) for all methods. We base our CVAE architecture on Higgins et al. (2017). The server decoder for FEDCVAF-KD has the same architecture as the client CVAEs by default, although this is not strictly necessary because FEDCVAE BECVEE-DD KD supports heterogeneous CVAE architectures.
For each method, hyperparameters were obtained through tuning, with the bounds extended until the best perffrrrfonnng value appeared in the middle of the grid. Full hyperparameter settings can be found Table and Table T (Appendix(C) All classifiers use a cross-entropy objec- tive. CVAE training the objective for grayscale and RGB images, respectively; we use the same reconstruction objective for the KD loss (g()) in Equation
search grid
binary
entropy and mean squared error for the reconstruction term of
4.2 GENERAL RESULTS
Statistical Heterogeneity. To demonstrate the efficacy of FEDCVAE-ENS and FEDCVAE-KD in the difficult setting of high statistical heterogeneity, we test varying levels of from high (a = 0.05), to very high (a = 0.01), to extreme (a = 0.001) statistical heterogeneity. FEDCVAE- ENS consistently outperforms all other methods, and SEDCVAE-KI outperforms the baselines in nearly all datasets and levels of a (the only exception is MNIST at the lowest level of statistical heterogeneity) shown in Table At 0.001, FEDCVAF-KD obtains more than 1.75x the

---

Published as a conference paper at ICLR 2023
Table 1: Performance four data-free one-shot FL methods over three datasets and across three levels of statistical heterogeneity (lower is more heterogeneous). Best results for each dataset and each level of are in purple with second best results in yellow
Heterogeneity
FEDAVG
FED
FEDCVAE- KD (ours)
FEDCVAB- (ours)
0.001
45.12 5.87
11.90 0.40
82.24 1.09
95.37 0.52
MNIST
0.01
58.29
41.15 + 0.70
82.01 1.61
93.83 1.53
0.05
80.10
82.95 = 0.49
79.57 1.17
91.86 - 0.75
a = 0.001
32.77 4.52
10.00 0.00
69.53 1.70
76.04 0.93
FashionMNIST
0.01
45.85 + 2.95
37.63 0.53
69.97
76.62
0.05
46.86 1: 2.37
64.53 1.73
67.24 1.96
71.95
a 0.001
20.31 1: 4.36
15.94 + 0.00
55.48 1: 2.12
65.32 0.66
SVHN
0.01
25.12 + 2.07
31.91 + 1.26
55.97 0.38
65.50 2.28
0.05
33.66 4.12
49.60 + 1.59
54.24 2.08
66.61 2.09
Table 2: Performance of four one-shot FL methods over three datasets and across four numbers of clients Best results for each dataset and each level of in purple with second best results
in yellow
of Clients
FEDAVG
FEDONESHOT
FEDCVAE KD (ours)
FEDCVAE ENS (ours)
m =
42.34 + 1.75
42.97 1.54
80.19 = 1.76
92.81 + 2.16
m 10
58.29 + 2.83
41.15 0.70
82.01 = 1.61
93.83 + 1.53
MNIST
m 20
37.14 + 3.87
32.22 0.71
81.93 = 1.75
93.52 = 0.86
m 50
28.26 + 7.98
35.62 + 0.66
77.68 1.39
87.56 = 1.44
38.51 1: 2.40
32.23 1.68
68.01 1.56
73.70 +: 1.93
10
45.85
+: 2.95
37.63 37.63 0.33
69.97 1.63
76.62 - 1.61
FashionMNIST
M 20
27.49
4.77
24.72 1.20
71.28 1.17
76.51 + 1.38
50
31.91 +: 2.20
41.01 2.49
69.25 1.13
76.13 + 2.00
m
40.61 + 0.64
42.05
1.31
57.17 2.56
65.72 2.17
SVHN
m
25.12 2.07
31.91
1.26
55.97 0.38
65.50 2.28
m 20
26.99
+ 3.15
10.68
0.70
51.80 2.75
65.54 + 1.89
m 50
19.59 + 0.00
30.20 0.64
40.03 2.62
64.81 2.44
accuracy of the best baseline method for MNIST, more than 2 the accuracy for FashionMNIST, and nearly 2.75 2.75x the accuracy for SVHN (Table D. While the baselines FEDAVG and FEDONESHOT very sensitive to the level of statistical heterogeneity, both FEDCVAE-ENS and FEDCVAE-KD demonstrate consistent performance across levels of a (Table
Number of Clients. Because FL applications often include many participating clients Lii et al. 2020a), we evaluate several values for number of clients (m = (5. 10, 20, 50}). While both FEDAvG and FEDONESHOT struggle when many clients present, Table T shows that FRDCVAE-ENS and FEDCVAE-KD perform consistently across the number clients, with the only exception being FEDCVAE-KD for SVHN. Even though is typical to observe FL methods' accuracy degrade with increasing numbers of clients (Zhang et al.) 2021). FEDAVG and FEDONESHIOT are unstable with no clear decrease in accuracy, which we ascribe to the highly variable partitions generated high levels of statistical heterogeneity. Experiments varying the dataset partitions reveal high variation in accuracy for FEDAVG and FEDONESHOT, whereas both FEDCVAE ENS and FEDCVAE-K exhibit consistent accuracy (Table in Appendix C
Decoder Aggregation. The proposed KD aggregation method used in FEDCVAE-KD substantially improves on aggregation via parameter averaging, generating qualitatively more realistic samples (Figure g in Appendix C) and achieving higher server classifier accuracy (Table 6 in Appendix C).
As expected, FEDCVAE-EN bounds the performance of FEDCVAE-KD, but KD-based aggrega- tion still preserves client ensemble performance well across levels of a for all datasets (Table I The upper bound accuracy established by FEDCVAE-ENT is promising, but successful decoder ag- gregation may require a stronger KD procedure (e.g., feature- based KD as described in Gou et al. 202T))

---

Published as a conference paper at ICLR 2023
FedCVAE-KD
FedCVAE-Ens
100
Pateropenecus Homagareous
100
Heterogemecua Homoganesus
80
80
Accuracy (%)
Accuracy
60
60
20
20
0
MNIST
FashionMNIST Dataset
SVHN
MNIST
Mashion Dataset
SVHN
Figure 4: Results with heterogeneous local models. "Homogeneous" uses the same CVAE architec- ture for clients, whereas "heterogeneous" uses two architectures with similar generative capabil-
ities.
MNIST
FashionMNIST
SVHN
60
80
Accuracy (96)
Accuracy (96)
60
Accuracy (%6)
60
40
40
20
20
20
+1
+10
+100
1000
#1
+10
+100
+100
+1
#10
+100
#1000
Sampling Bounds
Sampling Bounds
Sampling Bounds
Figure 5: Results for our proposed pecurity-promotiin extension at high heterogeneity (a 0.05). We show the accuracy for classifier trained on samples from the intercepted client decoders and using client label distributions when the center of the normal prior is unknown. The sampling bounds represent the parameters of the uniform distribution used for latent vector sampling, i.c., +100 represents the distribution ((0-000, 100). The prior is multivariate standard normal (0, I).
4.3 EXTENSIONS
Heterogeneous Local Models. To simulate clients with diverse computational resources, we train both FEDCVAE-ENS and FEDCVAF. using two local CVAE architectures: the first as de- scribed in Section 4.1 4.1 and the second with one convoluiooal/decoovlluuii encoder/decoder, respectively. The server decoder matches that which described Section 4. but can be chosen arbitrarily. Because the two architectures demonstrate similar generative capabilities, final server classifier accuracy is very similar when comparing homogeneous against heterogeneous local architectures (Figure Notably, KD procedure for FEDCVAE-KO KD still performs well using heterogeneous models, indicating diverse architectures can organize latent space similarly enough to successfully translate this knowledge single decoder.
block removed for the
Promoting Security, We verify the effectiveness of our proposed distribution shift extension for securing the uploaded information. Suppose an eavesdropping attacker is able intercept the label distributions, decoder weights, and decoder architectures from all clients during upload. Without knowledge of the shared center of the multivariate normal prior μ4, we show that training per- formant classifier is infeasible because it is difficult to extract high-quality samples from the client decoders. In particular, even when the attacker samples latent vectors z from broad region which overlaps with the high-density region of the prior (i.e., a uniform distribution centered on μu), the accuracy of the classifier trained on the resulting samples degrades sharply as the sampling region
'We choose not to baseline against FEDONESHOT, which supports heterogeneous local models, because our notion of "neterogeneous model" is different; FEDONESHOT supports heterogeneous local classifiers while our two methods defer the choice of classifier after FL is complete and instead support heterogencous local CVAFs.
8

---

Published as a conference paper at ICLR 2023
grows (Figure 5). Even good guess of of 10) for normal prior of (O, I) results in a 30% point decrease in accuracy depending on the dataset. A more realistic guess of (000000, 1000) re- sults in 33 67% point decrease. Therefore, this simple FEDCVAE-ENT and FEDCVAE-KI extension reduces eavesdropping attackers' capacity to extract high-quality samples from uploaded decoders or train performant downstream model, reducing communication risks.
RELATED WORK
FL Under High Statistical Heterogeneity. FL. was originally proposed by McMahan et al. (2017) a paradigm for decentralized distributed learning. Statistical heterogeneity quickly emerged as a core issue within FL, with many studies focusing on maintaining high performance under very non-IID data. Many approaches have experimented with augmenting FEDAVG by adding proximal terms to the local objective as an attempt to restrain local updates el. 2020e Karimireddy et al,) 2020 Wang et al. 2020 Acar et al., [021 Li et 2021a). Other works have used KD to circumvent
issues associated with parameter averaging under non-IID data, focusing on leveraging auxiliary 202T)
generative model (Zhu et al. 2021 Zhang et al., 2022) to compactly capture client ensemble learning. Recent approaches focus on improving client selection strategy (Tang et al., 2022), reducing catastrophic forgetting balance global knowledge against local learning (Huang et al. 2022), or improving the generality of client models (Mendieta al.) 2022). However, all of these methods are designed for standard FL and rely heavily on local regularization through substantial iterative communication, which is not feasible in the one-shot FL. setting where communication limited to a single round.
data (Lin et al. 2020; Sattler al.
One-Shot FL. Methods in one-shot FL have demonstrated strong performance under substantial communication constraint. Guha et al. (2019) originally proposed one-shot FL and introduced two methods, one based on heuristic selection methods for client inclusion in the final ensemble and another that used KD with an auxiliary dataset for ensemble aggregation. [i et al. (2021b) extended variety of local model
the of KD with hierarchical KD procedure with wide applicability types. Although these methods obtain high accuracy, they rely on an auxiliary public dataset to support KD, which is inapplicable in data-free FL. Secure dataset transfer has been applied in sev- eral studies: Zhou et al. (2020) achieved this through dataset distillation, which requires a shared model architecture, Shin al. (2020) provided limited experiments using XOR-based data aug- mentation techniques. Zhang et al. (2021) proposed a data-free KD procedure based on generator network trained using the ensemble of client classifiers, showing promising performance even under heterogeneous local models. However, existing approaches in one-shot FI. either do not experiment with high statistical heterogeneity or degrade even under moderate levels of heterogeneity, unlike proposed methods.
VA in FL. A few studies have experimented with using VAEs FL., Kasturi et al. (2022) proposed distributed learning framework based on VAEs, but required an auxiliary pre-trained classifier to generate sample labels and did not specify how to obtain this classifier. Wen et al. (2020) and Gu & Yang (202l) used CVAEs to protect against malicious clients, but only included limited experimental results with respect to statistical heterogeneity, relied on multiple communication rounds, and, in the case of Wen et al. (2020), did not use KD for server-side aggregation. We are the first to apply CVAES to one-shot FL with a focus on high statistical heterogeneity.
CONCLUSION
In this paper, we proposed FEDCVAE-ENS and FEDCVAE-KD. data-free one-shot FL. methods that reframe the local learning task using CVAEs. Both methods performed well given high statis- tical heterogeneity, demonstrated consistent performance with increasing numbers of clients, allow for model heterogeneity across clients, and can be extended to promote security. Extensive exper- imental results showed that -ACCAE-EENS and FEDCVAE KD substantially extended the state- oh-the--rt in one-shot FL under very high statistical heterogeneity, making way for more nuanced development in this difficult environment for distributed learning.

---

Published as a conference paper at ICLR 2023
REFERENCES
Durmus Alp Emre Acar, Yue Zhao, Ramon Matas, Matthew Mattina, Paul Whatmough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. In International Con- ference on Learning Representations, 2021. URL https:// www.:nee./eeee formm?
1d=B7v4QMR629w
Rohan Anil, Gabriel Pereyra, Alexandre Passos, Robert Ormandi, George Dahl, and Geoffrey Hinton. Large scale distributed neural network training through online distillation, arXiv preprint arXiv:180..03255 2018.
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil Model compression. KDD '06, pp. 535-541, New York, NY, USA, 2006. Association for Computing Machinery. ISBN 1595933395.
doi: 00.1145/1104002. 155444
URL latpss///cl.orr//00..
(150402.115084
Nikita Dvornik, Cordelia Schmid, and Julien Mairal. Diversity with cooperation: Ensemble methods for few-shot classification. Proceedings of the IEEE/CVF international conference on computer vision, pp. 3723-3731, 2019.
Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In International Conference on Machine Learning, pp. 1607-1616.
PMLR, 2018.
Robin Geyer, Tassilo Klein, and Moin Nabi. Differentially private federated learning: A client level perspective. arXiv preprint arXiv:1712.0755: 2017.
Jianping Gou, Baosheng Yu, Stephen Maybank, and Dacheng Tao. Knowledge distillation: A survey, International Journal of Computer Vision, 17866)178--1819, 2021.
Zhipin Gu and Yuexiang Yang. Detecting malicious model updates from federated learning on con- ditional variational autoencoder. In 202/ IEEE International Parallel and Distributed Processing
Symposium (IPDPS), pp. 671-680, 2021. doi: 10. 10.1109/PDDS449936222 0tw.:/4499966222..000000 000055.
Neel Guha, Ameet Talwalkar, and Virginia Smith. One-shot federated learning. abs/1902. 1175, 2019. IRI.hhtt:///axxii
CoRR,
abs 111/11902.1111
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 3-VAE: Learning basic visual concepts with a con- strained variational framework. In International Conference on Learning Representations, 2017.
www.eeeweeeeeeeeeeeeeeeeeeeeeeee wttp:ewww..eeeeerteeeeeme net. ftrrmm SttpuummiddSy22222900
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in a neural network. arXiv preprint arXiv: 1503.02333,, 2(7), 2015.
Tzu-Ming Harry Hsu, Qi, and Matthew Brown. Measuring the effects of non-identical data distri- 2019.
bution for federated visual classification. ArXiv,
Wenke Huang, Mang Ye, and Bo Du. Learn from others and be yourself in heterogeneous fed- crated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10143-10153, 2022.
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning, pp- 5132-5143. PMIRR 2020.
Anirudh Kasturi, S Vishwamithra, and Chittaranjan Hota. Communication efficient distributed learning using variational auto encoders. In 2022 14th International Conference on COMmunica- tion Systems & NETworkS (COMSNETS), pp. 744-748, 2022. (COnOSNE
doi: 10. 10.100//0MSETTS3330..
2022.9668564.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 4242269800 2014.
10

---

Published as a conference paper at ICLR 2023
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. CoRR, abs/1312.6114, 2014.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recog- nition. Proceedings of the IEEE, 86(1 (6(11))2278-22) 2324, 1998. doi: 10.1 10.1509//.7279..
Li Li, Yuxi Fan, Mike Tse, and Kuo-Yi Yi Lin. A review of applications in federated learning. Computers & Industrial Engineering, 149:106854, 2020a. ISSN 0360-8352. doi: https://doi.
org/10. 110.10011f...ee222000666666 106854. URL httpp::/www..t.. http://wwwwcccieeeerrrrcc.tccee. /
Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2021a.
Qinbin Li, Bingsheng He, and Dawn Xiaodong Song. Practical one-shot federated learning for cross-silo setting. In I/CAI, 2021b.
Tian Li. Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine, 37(3):50 60, 2020b.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Sys- tems, 2:429-450,, 2020c.
Tao Lin, Lingjing Kong. Sebastian U. Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. Advances in Neural Information Processing Systems, 33: 2351-2363 2020.
H. B. McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüier y Arcas. Communication-effiienn learning of deep networks from decentralized data. In AISTATS, 2017.
Matias Mendieta, Taojiannan Yang, Pu Wang, Minwoo Lee, Zhengming Ding, and Chen Chen. Local learning matters: Rethinking data heterogeneity in federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8397-8406, 2022.
Viraaji Mothukuri, Reza M Parizi, Seyedamin Pouriyeh, Yan Huang, Ali Dehghantanha, and Gautam Srivastava. A survey security and privacy of federated learning. Future Generation Computer Systems, 15::19-640, 2021.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 201 2011. URL http::/ufldl. wtw.dl..daaadddeeooooooooo hoosenumberr//miss2200 hhwooloooouuuuuuuuuuossee .pdf
Felix Sattler, Tim Korjakow, Roman Rischke, and Wojciech Samek. Fedaux: Leveraging unla- beled auxiliary data in federated learning. IEEE Transactions on Neural Networks and Learning Systems, 2021.
MyungJae Shin, Chihoon Hwang, Joongheon Kim, Jihong Park, Mehdi Bennis, and Seong-Lyun Kim. Xor mixup: Privacy-preserving data augmentation for one-shot federated learning. arXiv preprint arXiv:2006.051 05148, 2020.
Minxue Tang, Xuefei Ning, Yitu Wang, Jingwei Sun, Yu Wang, Hai Li, and Yiran Chen. Fedcor: Correlation-based active client selection strategy for heterogeneous federated learning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10102- 10111, 2022.
Gerrit van den Burg and Chris Williams. On memorization in probabilistic deep generative mod- els. M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems. volume 34, pp- 27918-27928. Curran As- sociates, Inc., 2021. URL Feee

---

Published as a conference paper at ICLR 2023
Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 33:7611-7623, 2020.
Hui Wen, Yue Wu, Chenming Yang. Hancong Duan, and Shui Yu. A unified federated learning framework for wireless communications: towards privacy, efficiency, and security. In IEEE IN- FOCOM 2020 IEEE Conference on Computer Communicattonn Workshops (INFOCOM WK- SHPS), pp. 653-658, 2020. doi: 10.1 LLL10//OOOOOWSSSPSS6600000002220
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmark- ing machine learning algorithms. ArXiv, abs/1708.07747) 2017.
Jie Zhang, Chen Chen, Bo Li, Lingjuan Lyu, Shuang Wu, Jianghe Xu, Shouhong Ding, and Chao Wu. practical data-free approach to one-shot federated learning with heterogeneity. arXiv preprint 1rXiv:2112. 2021.
Lin Zhang, Li Shen, Liang Ding, Dacheng Tao, and Ling- Yu Duan. Fine-tuning global model via data-free knowledge distillation for non-iid federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10174-10183, June 2022.
Yanlin Zhou, George Pu, Xiyao Ma. Xiaolin Li, and Dapeng Wu. Distilled one-shot federated learning. arXiv preprint arXiv:2009.0799 2020.
Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou. Data-free knowledge distillation for heterogeneous federated learning. Proceedings of machine learning research, 100e12878-2888 2021.
12

---

Published as a conference paper at ICLR 2023
A ADDITIONAL DESCRIPTION OF METHODS
FEDCVAE-ENS Description. FEDCVAE-ENT follows the same procedure as FEDCVAB-KD,
but alternatively defines Eas as the combination of client subsets D { y, Fiz1 uses Das to directly train the server classifier. Figure O visualizes the full pipeline for FEDCVAE- ENS and Algorithm 2 details the full procedure. Algorithm shows the local client training pro- cedure, which is the same for FEDCVAE- EAN-ESS and SEDCVAE-KID Note that while we represent parameter optimization as using stochastic gradient descent all algorithms, any optimizer can be used; for our experiments, we uniformly use Adam (Kingma & Ba 2014).
['c/"] and
Communication
Server
clients
ببlل
Generate ensemble dataset
decoder weights
0
0
O
E
D
label distributions
4
4
D 4
Train classifier on ensemble dataset
... 0
Figure 6: The full pipeline for one of our proposed methods, FEDCVAE-ENS Here, E, D, and C represent "encoder," "decoder," and "classifier" models, respectively. First, clients train CVAEs on their private local datasets. Then, the server uses the ensemble of uploaded client decoders and corresponding local label distributions generate labeled dataset of synthetic samples to train classifier.
Algorithm FEDCVAE-ENS the one-shot FL setting. Ty represents the number of local training epochs. The server classifier parameters w with training epochs To. number of train- ing samples ne, classification loss (c),), and learning rate nJ. C is the set of clients.
1: procedure SERVER
2:
for each client & E C in parallel do
3: 0k, 1 (y) + LolentLocaaUpdateee Tr.) {(8] w.t v/))!"//m/ using client decoder Dør (-)
4: Generate samples from each client D
and label distribution ø* (y) Combine client subsets into an IID labeled dataset DEns = Dim U D Ens U U Dm for classifier epoch i for mini-batch 6 Deas do
5:
6:
To do
7:
8:
g < w nc Vwg (
B DETAILS ON THE PRIVACY- AND RECURITY-PROMOTEE EXTENSIONS
Privacy. A private pipeline one that does not leak private client data to other participating clients or the server. van den Burg & Williams (2021) define probabilistic generative model's propen- sity to reproduce samples observed in the training data as memorization, and prove that ensuring particular level of differential privacy (DP) can bound memorization in probabilistic generative models (including CVAEs). Thus, beyond the normal privacy guarantees attributable to DP, employ- ing FI. Deessimmation techniques (i.e., Geyer et al. (007)) also ensures low memorization across
13

---

Published as a conference paper at ICLR 2023
Algorithm Local training procedure. DD represents the client's local dataset. The local learn- ing rate is 17 and the local loss function is E(-)
1: procedure CLIENTLOCALUPDATEE TT)
2: Initialize local CVAE parameters w == 100 0K
3: for local epoch in i
Tz do
4:
for mini-batch 6 C D do
5:
Optimize based on Equation return Decoder parameters 0 and local label distribution j (y) to the server
- < w 7) V ww+.((**;
the FEDCVAE-ENS and FEDCVAE-KI pipelines, further enhancing the privacy of our proposed methods. We leave further exploration of privacy-preserving extensions future work.
Quality of Trained Decoder Samples. To generate high-quality samples from a trained CVAE, it typical sample latent vectors z, either directly from the prior (usually multivariate standard normal, i.e., Z; (0 I)) from some other distribution with tight bounds around the prior distribution's mean (e.g., a truncated standard normal or a uniform distribution). During training. the CVAE will largely observe latent vectors in the highest density region of the prior distribution; for a standard normal distribution, this near the center = Latent vectors distant from the center will not generate high-quality samples when used with the trained decoder. As a demonstration, train a centralized CVAE and sample both close to the center of the prior (i.e., 2; ~ (1(11, 1)) and distant from the center of the prior (i.e., 2, (5, 20)). The resulting image samples are shown in Figure
MNIST
FashionMNIST
SVHN
2
2 J
2
O
/
7
8
9
0
1
2
6
7
8
9
0
1
2
3
4
5
6
7
9
9
15
0
1
7
3
4
5
6
7
8
9
0
1
2
3
4
5
6
7
8 3
Figure 7: CVAE samples generated using latent vectors distant from the center of the multivariate normal prior (top row) and close the center (bottom row).
C ADDITIONAL EXPERIMENTAL DETAILS AND RESULTS
Benchmark Datasets. MNIST and FashionMNIST contain 28 x 28 grayscale images of handwritten digits and clothing/accessoriis, respectively, with 60, 000 train samples and 10, 000 test samples. SVHN contains 32 x 32 RGB image crops of street-view house numbers, with 73, 257 train samples and 26, 032 test samples.
Hyperparameter Settings. Tables Jand contain fixed and variable hyperparameters, respectively. While the number of local epochs (Tr,) may seem low for FEDAVG, we observed substantially reduced accuracy at higher numbers of local epochs, which is consistent with Lin et al. (2020) and references therein.
Stability With Respect to Dataset Partition. To complement the results in Table 2 we test the stability of each model across varying dataset partitions, controlled by a random seed (Table 5). When is very low, as in study, the dataset client splits generated by sampling from the Dirichlet distribution are diverse. This exacerbated when more clients are used (higher m), potentially While FEDAVG and SED very
explaining of the unstable results in Table 2
14

---

Published as a conference paper at ICLR 2023
Table The default hyperparameter settings, which used in experiments unless otherwise men- tioned. These values are held consistent across datasets.
Hyperparameter
Value
Shared
Local learning rate (1)
0.001
Classifier optimizer
Adam
Batch size (all)
32
FEDCVAB-ENS & FEDCVAE-KD
Server decoder optimizer CVAE optimizer
Adam
Adam
Server classifier train samples (nc)
5000
Server decoder train samples (nzp)
5000
Server classifier learning rate (MC
0.001
Server decoder learning rate (1))
0.01
Table 4: Dataset-speciftt hyperparameter settings, where applicable. "Truncated normal width" denotes the truncation bounds for the truncated standard normal distribution used for sampling from client decoders. The truncated normal width values are in terms of number of standard deviations.
Hyperparameter
MNIST
FashionMNIST
SVHN
FEDAVG
Local epochs (T')
10
10
FEDONESHOT
Local epochs (Tg)
15
15
10
FEDCVAE-ENS
Local epochs (Ty)
15
25
50
CVAE latent dimension (d)
10
10
10
Server classifier epochs (Tc)
10
5
5
Truncated normal width
#:3
+3
#3
FEDCVAE-KD
Local epochs (Tt)
15
25
50
CVAE latent dimension (d)
10
100
10
Server classifier epochs (Tc)
10
5
Server decoder epochs (Tb)
7
10
80
Truncated normal width
#1
#2
13
sensitive to dataset split (high standard deviation), the results in Table 5 for both FEDCVAE-ENS and FEDCVAE-KD are not only consistent with Table but also exhibit reasonable stability (low standard deviation).
Table 5: Performance of four one-shot FL. methods over three datasets and across four numbers of clients m. Results show the average test accuracy across 5 random dataset partitions + one standard deviation. Parameter initialization remains constant. Best results for each dataset and each level of m are in purple. with second best results yellow
# of Clients
FEDAVG
FEDONESHOT
FEDCVAE-K] (ours)
FEDCVAE-ENS (ours)
5
50.58 + 8.12
38.81 + 6.449
80.84 + 1.03
93.97 + 1.11
10
52.67 + 12.67
34.90 10.79
82.67 + 0.85
94.01 + 1.54
MNIST
20
38.20 +: 4.71
38.85 +: 9.04
82.08 + 0.89
93.29 +: 2.16
50
30.24 +: 7.24
45.65 1 14.79
79.49 + 1.60
89.68 +: 2.21
44.87 7.02
36.28
65.60 = 3.06
71.80 3.59
FashionMNIST
10
37.54 13.00
6.00
69.43 + 2.26
7 2.47
20
34.34 4: 12.70
33.57 8.59
69.55 :: 1.47
76.66 1.79
50
28.89 + 8.43
36.49 + 7.65
68.91 = 0.30
74.70 +
m =
30.27 + 7.88
35.01 + 6.02
57.43 - 2.20
66.01 1.30
m 10
10.46
3.45
65.34
SVHN
m 20
20.63 + 2.16
29.92
12.12
49.18 - 2.84
65.82
77 50
19.59 + 0.01
30.22 + 2.83
38. + 2.97
64.91
15

---

Published as a conference paper at ICLR 2023
Comparing KD With Parameter Averaging. FEDAVG McMahan et al. 2017) introduced the no- tion of parameter averaging to FL. While parameter averaging may seem like reasonable approach for client decoder aggregation in FEDCVÁE-KI it generates qualitatively poor samples (Figure and fails to train a high-aceuracy classifier (Table 6). The KD approach we propose to aggregate client decoders generates substantially better samples (Figure ( while also obtaining more than 2x classifier accuracy on MNIST, more than 3.25x accuracy on FashionMIIST, and nearly 5x accuracy for SVHN at a 0.001 (Table 6).
MNIST
FashionMNIST
SVHN
9
4
0
9
0
2
0
9
0
0
0
9
9
7
0
0
0
3
2
2
9
9
8
9
8
e
e
e
e
9
9
0
1
2
6
7
8
9
0
2
4
6
7
8
9
0
1
2
4
6
7
8
9
0
1
2
3
4
5
6
7
8
9
0
1
2
3
4
5
6
7
8 9
Figure 8: Samples from the aggregated server decoder obtained through parameter averaging (top row) versus knowledge distillation (bottom row) high statistical heterogeneity (a = 0.00)..
Table 6: Comparing the knowledge distillation aggregation methods of FEDCVAE-KD simple parameter averaging. Best results for each dataset and each level of a are in purple
Heterogeneity
Parameter Averaging
Knowledge Distillation
a 0.001
38.59 + 3.29
82.24 + 1.09
MNIST
0.01
39.50 = 6.76
82.01 + 1.61
a 0.05
50.20 4.63
79.37 = 1.17
0.001
20.88 + 4.78
69.53 + 1.70
MNSST
a 0.01
27.66 7.08
69.97 + 1.63
0.05
39.35 + 6.27
67.24 + 1.96
Q 0.001
11.26 1 0.77
55.48 1 2.12
SVHN
0.01
15.22 + 4.29
55.97 = 0.38
0.05
10.61 2.23
54.24 + 2.08
Performance With Less Data Per Client. To gauge our proposed methods' performance relative the size of each client's local dataset, we vary the percent of the benchmark training data distributed clients (Table 7). While accuracy for both FEDCVAE-ENS and FEDCVAE-K degrade with less data per client, they both consistently perform better than FEDAVG and FEDONESHOOT across all tested percent subsets of the training data.
Uploading client label distributions ş may generate additional privacy concerns. One potential solution mask the precise label counts for each client by adding noise before upload as in Zhang et al. (2022); to achieve this, we draw noise from a normal distribution 6, ~ A'0 7 ng) such that the "strength" (variance) of the noise applied to class e is in proportion 7 to the total number of training samples for client k. Noise is applied to the training sample count for each class for a given client before uploading this information to the server. We visualize the effect of noise client label distributions for several levels of 7 in Figure 9 As 7 - oc, information from the uploaded label distribution disappears; when y = 0, communicated. When adding modest amount of noise the client label distributions (i.c., 7 < 0.1), accuracy for both FEDCVAE- E-ENS and FEDCVAE-KO is barely affected compared upload with noise; Table 8 for and refer to the 0.01
Adding Noise to Uploaded Label Distributions.
the exact local label distributions
16

---

Published as a conference paper at ICLR 2023
Table Performance of four data-free one-shot FL methods over three datasets and across multiple percent subsets of each dataset. Best results for each dataset and percent subset are in purple with second best results yellow
% Subset
FEDAVG
PEDONESHOT
FEDCVAE-KD (ours)
FEDCVAE-ENS (ours)
5%
40.65 + 2.65
34.24 1.51
74.98 + 4.82
76.47 + 2.58
10%
38.35 + 4.43
36.86 1.14
81.24 1.45
89.95 + 1.27
MNIST
25%
56.17 3.82
47.50
80.00 + 1.52
92.41 + 0.79
50%
58.29 2.83
41.15 0.70
82.01 + 1.61
93.83 = 1.53
5%
14.11 + 2.33
26.96 0.74
66.75 + 1.80
69.78 +: 1.26
10%
26.53 + 6.28
27.26 + 1.69
69.07 + 1.53
72.10 = 1.66
MNIST
25%
28.29 + 1.69
29.97 + 2.53
71.24 + 1.48
75.31 + 1.42
50%
45.85 + 2.95
37.63 + 0.53
69.97 + 1.63
76.62 + 1.61
5%
17.19 + 5.38
11.08 +
0.38 0.38
41.74 + 3.67
46.06 = 1.29
10%
22.97 + 4.37
16.16 0.73
47.65 + 2.88
54.36 = 1.76
SVHN
25%
28. + 3.15
29.10 + 0.26
52.15 + 1.23
59.96 = 2.68
50%
20.38 + 1.03
32.39 + 2.85
37.56 + 1.09
64.82 + 1.27
1005
25.12 + 2.07
31.91 + 1.26
55.97 + 0.38
65.50 + 0.28
Table 8: Performance of our proposed methods with noise added to the uploaded client label distri- butions. Results show the average test accuracy across 5 seeds for the random noise = one standard deviation.
Noise Proportion
FEDCVAB-KD
FEDCVAE-ENS
7 = 0.01
81.10 + 0.69
93.69 0.82
MNIST
= 0.05
80.41 + 1.81
92.65 0.77
7 0.1
80.56 1 0.36
92.65 1.33
0.01
69.43 +: 1.61
76.44 1.11
PashionMNIST
? 0.05
68.57 +: 1.74
74.44 2.14
7 = 0.1
69.07 1.97
72.88 3.03
T 0.01
57.18 + 2.24
64.03 1.78
SVHN
7 0.05
56.93 + 2.43
62.68 2.64
0.1
55.85 + 2.29
61.47 1.83
in Table for y 0. The notion of uploading and harnessing client label distributions in FL is Zhu et al. 2021 Zhang et al. 2022) and quantifying the privacy risks that label distributions might induce is an open problem which could benefit from focused development.
0.01
y 0.1
Cass label
Cass label 2
Class label
Cass label
N f d U1
5
Client
Client ID
Client ID
Client
Figure 9: The effect of the noise proportion y on an example data partition at a 0.01. m 10 clients, and on MNIST. The original partition is in yellow (7 0). The size of each dot is proportional the number of samples.
17


Reviews:
[{"review_id": "jsFHgZMwyG", "reviewer": "Reviewer_feTx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper targets to one-shot federated learning under high statistical heterogeneity. The authors designed two different solutions: FEDCVAE-ENS and FEDCVAE-KD, both of which construct a VAE on the server to generate samples. The generated samples are utilised to train a classifier on the server. The difference between FEDCVAE-ENS and FEDCVAE-KD is that FEDCVAE_KD create a single server VAE decoder while FEDCVAE-ENS stores a set of VAEs that is trained on local clients.", "summary_of_the_review": "This work focuses on one-shot FL with high data heterogeneity. Two solutions are proposed in this work, which has been tested under extensive experiments. The writing is clear and well-organized. There might be some minor issues in this draft. I look forward to the feedback from the authors in the rebuttal.", "strength_and_weaknesses": "Strength:\n\n-- The proposed solutions seem effective under the one-shot FL setting. The efficacy and efficiency are both demonstrated in the extensive experiments.\n-- The writing is clear and easy to follow. The tables and figures are well-organized.\n\nWeaknesses:\n\n-- Constructing a generative model to generate samples for FL model training is not new. There are a few works utilizing a similar strategy in FL, for example:\nData-Free Knowledge Distillation for Heterogeneous Federated Learning, ICML 2021\nFine-tuning global model via data-free knowledge distillation for non-iid federated learning, CVPR 2022\nPlease make a comparison with those works\n\n-- Some details are not clear enough for me. For example, in Eq 2, what is the formulation for g(.)?  Is this dissimilarity measured in the data-space or feature space?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SB6URURlVmz", "reviewer": "Reviewer_TDXe", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes an alternative way to perform federated learning; instead of training a discriminative model, the authors propose to train a conditional (on the label) VAE model $p(x, z| y) = p(z)p(x|z, y)$, to convergence on each client and then communicate (once), provided that the clients agree on a $p(z)$, the decoder along with the marginal label distribution of the client, back at the server. The server can then use the marginal $p(y)$ on each client along with its decoder to generate training data $(x, y)$ in order to train centrally a discriminative model. Having access to all of the decoders, the server can also train a global decoder to “distill” all of the client specific ones to reduce storage requirements. The authors further propose the use of a randomly centred prior mean for $p(z)$ in order to improve the security of the system against external entities. Better performance in cross-silo settings and high non-iid settings is demonstrated. \n", "summary_of_the_review": "While the concept of FedCVAE is straightforward, it is novel in the federated setting. Having said that, given my aforementioned comments, around the auxiliary classifier, security / privacy and experimental evaluation, I cannot recommend acceptance of this work.", "strength_and_weaknesses": "Strengths\n- Relatively simple procedure that requires a single round of communication\n- As far as I know, using VAEs in this specific way for FL is novel\n\nWeaknesses\n- Simple datasets for generative models; how about CIFAR10, CIFAR100 or even Shakespeare?\n- Experimental evaluation against baselines is not entirely apples to apples\n- Some statements and explanations are not entirely accurate (see below)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7l3T4OTAii", "reviewer": "Reviewer_mEpo", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces two methods, FEDCVAE-ENS and FEDCVAE-KD, for one-shot federated learning with extreme data heterogeneity among clients. Once client models are trained using conditional variational auto-encoder (CVAE) architectures, the decoders, alongside label distributions of all clients, are uploaded to the server. FEDCVAE-ENS creates an ensemble of synthetic data samples created via all client decoders to create a dataset that can be used to train a centralized classifier at the server. FEDCVAE-KD, on the contrary, trains a centralized decoder at the server using knowledge distilled from the client decoders. The trained decoder is then used to create a dataset of synthetic samples, which is used to train the centralized classifier. Simulation results show that i ) the proposed methods considerably outperform baselines when the data samples are very heterogeneous, and ii) using the CVAE architecture enables a security-promoting extension of the proposed methods that prevents an eavesdropper from training a well-performing centralized classifier.", "summary_of_the_review": "I think this is a borderline paper with clear positive aspects, but one that needs some content streamlining and restructuring, as well as stronger motivation so that the contributions are better appreciated.", "strength_and_weaknesses": "The paper is very well written and easy to follow. The numerical experiments are well executed and show promising performance achieved by the proposed methods.\n\nThat being said, there are some downsides that, in my opinion, slightly outweigh the strengths of the paper. On the problem formulation side, I believe the one-shot federated learning (FL) problem is not very well motivated; the point of FL is to enable training models at different clients by leveraging each others' data through communication without explicitly sharing them. In a one-shot scenario, the problem essentially reduces to having a set of models, which need to be aggregated into a single, universal model, so I am still not convinced how the \"federated\" aspect of the problem is of the essence.\n\nOn the contributions side, the main issue I have with the paper is a lack of a central, key novelty. In its current form, the paper is mostly a collection of different ideas, namely two algorithms that are almost independent of each other, as well as a security aspect, which is again mostly orthogonal to the proposed methods. To me, what would make this paper much stronger is to focus on only one of these algorithms/aspects throughout the paper. For example, since the performance of FEDCVAE-KD is better than baselines, I believe it would be better to focus on this algorithm in the main body of the paper, and add the ensemble version of the algorithm (which, as the authors allude to, requires a higher amount of memory that scales with the number of clients) as an extension, in a separate discussion section toward the end of the paper and/or in the appendix. Moreover, the security and privacy considerations are definitely important, but including them in the main body of the paper stretches it too thin, in my opinion.\n\nFurther comments:\n\n- Aside from the solid empirical performance, are there any intuitive/theoretical explanations as to why the proposed methods are well suited for i) *one-shot* FL, and ii) *high heterogeneity* FL? As presented, I cannot see a clear motivation for why the proposed methods should perform better in the considered problem formulation.\n- Please discuss whether or not sharing local label distributions with the server violates the privacy of clients' data. How do the proposed methods perform if this information is unavailable at the server?\n- Presenting the flowchart figure for only one of the proposed methods (FEDCVAE-KD) in Figure 2 and not including any of the algorithms for the two proposed methods makes the paper slightly incoherent, without switching back and forth between the body of the paper and the appendix. I suggest rearranging the contents to remedy this issue as best as possible by at least having both the figure and algorithm for one of the methods or having the figures for both methods in the main body of the paper.\n- In the first footnote on page 4, why is a uniform distribution used for sampling $\\mathbf{z}$? Why not, for example, use a truncated Gaussian distribution?\n- In my opinion, including FedAVG is slightly unfair in the experiments. Under FedAVG, a set of client model parameters $\\\\{\\mathbf{w}^k\\\\}_{k=1}^m$ are aggregated using a simple average, i.e., $\\mathbf{w}=\\frac{1}{m} \\sum_\\{k=1\\}^m\\mathbf{w}^k$. In a very heterogeneous case, each of these model parameters is the minimizer of a very different objective function over an entirely different optimization landscape, so it is completely expected that a linear averaging of these model parameters should lead to inferior performance, especially if the averaging operation is done only once.\n- Following the above comment, I believe your primary competing baseline is FedONESHOT, so I suggest adding more details on this algorithm's main idea.\n- Do you have any explanation as to why the standard deviation of the performance of the proposed methods is, on average, larger under SVHN in Table 2?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "3ZtCaGvhH5-", "reviewer": "Reviewer_nh59", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers one-shot federated learning under high statistical heterogeneity. They propose a data-free FL mothods FEDCVAE-ENS and its extension FEDCVAEKD. Conditional variantional autoencoders(CVAE) are trained locally and then the decoders are sent to the server. At the server, either ensemble or knowledge distillation is used to generate an ensemble dataset which is later used to train the final model. The paper conducts experiments on three datasets to demenstrate its effectiveness under high statistical heterogeneity.", "summary_of_the_review": "The idea of the paper is novel and interesting. Although in real applications with larger images and limited data samples, it may not work very well, it provides a novel perspective of doing one-shot FL.", "strength_and_weaknesses": "### Strength\n\n* The proposed method of one-shot FL is very interesting and novel.\n* The proposed method doesn't require auxiliary public dataset.\n* Both security and privacy extensions are considered in the paper.\n* It can be applied to heterogeneous local models with similar generative capabilities.\n\n### Weaknesses\n* One potential issue of the method is training a CVAE locally. In the paper, the images used in the dataset are very small, and the amount of data each client has is large. However, in the real-world application of FL like medical imagings, with very large images and very limited number of data samples, training a CVAE locally can be difficult even just for one class of data.\n* The training task of local models is different from the original task. This limits the re-use of pre-trained local model from original task. I think one of the reasons that people want to do one-shot FL is to prevent any re-training locally. But the proposed method still requires to train some new model.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster