Revisiting Ensembling in One-Shot Federated Learning
Authors: Youssef Allouah
Akash Dhasade
Rachid Guerraoui
Nirupam Gupta
Anne-Marie Kermarrec
Rafael Pinot
Rafael Pires
Rishi Sharma
Venue: NeurIPS 2024 poster (2024)
Keywords: One-shot Federated Learning
Communication Efficiency
Federated Ensembles
Primary area: other

Abstract: Federated Learning (FL) is an appealing approach to training machine learning models without sharing raw data.  However, standard FL algorithms are iterative and thus induce a significant communication cost. One-Shot FL (OFL) trades the iterative exchange of models between clients and the server with a single round of communication, thereby saving substantially on communication costs. Not surprisingly, OFL exhibits a performance gap in terms of accuracy with respect to FL, especially under high data heterogeneity. We introduce Fens, a novel federated ensembling scheme that approaches the accuracy of FL with the communication efficiency of OFL. Learning in Fens proceeds in two phases: first, clients train models locally and send them to the server, similar to OFL; second, clients collaboratively train a lightweight prediction aggregator model using FL. We showcase the effectiveness of Fens through exhaustive experiments spanning several datasets and heterogeneity levels. In the particular case of heterogeneously distributed CIFAR-10 dataset, Fens achieves up to a $26.9$% higher accuracy over SOTA OFL, being only $3.1$%  lower than FL. At the same time, Fens incurs at most $4.3\times$ more communication than OFL, whereas FL is at least $10.9\times$ more communication-intensive than Fens.

Revisiting Ensembling in One-Shot Federated Learning
Youssef Allouah'
Akash Dhasade
Rachid Guerraoui!
Nirupam Gupta?
Anne-Marie Kermarrec
Rafael Pinot
Rafael Pires
Rishi Sharma'
EPFL
University of Copenhagen
Sorbonne Université and Université Paris Cité, CNRS, LPSM
Abstract
Federated learning (FL) is an appealing approach to training machine learning models without sharing data. However, standard FL algorithms iterative and thus induce a significant communication cost. One-shot federated learning (OFL) trades the iterative exchange of models between clients and the server with a single round of communication, thereby saving substantially on communication costs. Not surprisingly, OFL exhibits performance gap in terms of accuracy with respect to FL. especially under high data heterogeneity. We introduce FENS, a novel federated ensembling scheme that approaches the accuracy FL with the communication efficiency of OFL. Learning in FENS proceeds in two phases: first, clients train models locally and send them the server, similar to OFL, second, clients collaboratively train lightweight prediction aggregator model using FL. We showcase the effectiveness of FENS through exhaustive experiments spanning several datasets and heterogeneity levels. In the particular case of heterogeneously distributed CIFAR-10 dataset, FENS achieves up to a 26.9% higher accuracy over state-of-the-art (SOTA) OFL, being only 3.1% lower than FL. At the same time, FENS incurs at most 4.3x more communication than OFL, whereas FL is at least 10.9x more commuuicaationniteeaaee than FENS.
Introduction
FL is a widely adopted distributed machine learning (ML) approach, enabling clients to collabo- ratively train common model over their collective data without sharing raw data with central server 27 Clients in FL engage in iterative parameter exchanges with the server over several com- munication rounds train model. While providing high accuracy, this process incurs substantial communication cost [19]. One-shot federated learning (OFL) [I] has been introduced to address the communication challenges in FL by reducing the exchange of models to a single round. Not surprisingly, this came with loss of accuracy with respect to FL.
Typical OFL methods execute local training at the clients up completion and form ensemble of locally trained models at the server 7 10, 46 [T]. The ensemble is distilled into a single model, through means of either auxiliary public dataset [T] synthetic data generated at the server 7 14 46). While these OFL methods address communication challenges by reducing model exchanges single round, they often achieve lower accuracy compared to iterative FL.. This is especially true when data distribution across clients highly heterogeneous as OFI. methods typically
*Corresponding author takksh.dhasadee www.ddda@@@@ll.cchh....... epfl.ch>
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Federated Learning
One-shot FL
FENS
Iterative
One-shot
One-shot
Iterative
Partially trained model
Fully-trained model
Aggregator
Figure 1: FENS comparison iterative and one-shot federated learning.
rely simple prediction aggregation schemes such averaging or voting g
46
weighted averaging 7 10
We introduce FENS, hybrid of OFL and standard FL. FENS aims to approach both the accuracy of iterative FL as well as the communication cost OFL. Learning in FENS proceeds in two phases. In the first phase, similar to OFL, clients upload their locally -trinne models to the server. Instead of using the traditional OFL aggregation, FENS employs second phase of FL: the server constructs an ensemble with a prediction aggregator model stacked top of the locally trained models. This advanced aggregation function is then trained by the clients in a lightweight FL training phase. The overall learning procedure is illustrated in Figure] T alongside iterative and one-shot FL.
1.1 Our Contributions
We show for the first time, to the best of our knowledge, that shallow neural network for the aggregator model suffices to satisfactorily bridge the gap between OFL and FL. Leveraging shallow aggregator model enables two major benefits: first, it induces significantly lower commu- nication cost in the iterative phase, and second, the iterative refinement of this aggregator model significantly improves accuracy over existing OFL methods. By utilizing clements from both OFL and FL in this novel ensembling scheme, FENS achieves the best of both worlds: accuracy of Fl. and communication efficiency of OFI.
Co-boosting (one-shot)
FENS
FedAdam (iterative)
Test accuracy [%]]
60
20
Client comm. [GiB]
0.01
0.05
0.1
Through extensive evaluations on several benchmark datasets (CIFAR- 100, CIFAR-10, SVHN, and AG-News) across different heterogeneity levels, we demonstrate the efficacy of FENS in achieving FL-like accuracy at OFL-like communication cost. We extend our empirical evaluations to the FLamby benchmark [32], a realistic cross-silo FL dataset for health- care upplications. Our results show that in heterogeneous settings where iterative FL algorithms struggle, FENS remains a strong competitor. We then conduct an extensive study of different aggregator models and highlight the accuracy communication trade-off. Lastly, show that FENS maintains high accuracy even with comparable memory footprint.
100
4.3
42
3.8
0.01
0.05
0.1
Data heterogeneily
Bar labels indicate the normal- ized comm. cost OFL
To showcase FENS'S performance, we compare its accuracy and commu- nication costs against Co-Boosting 77.. a state-of-the-art OFL method, and FEDADAM 333 popular iterative FL algorithm, shown in Fig- These evaluations are performed on the CIFAR-10 dataset with 20 clients across three heterogeneity levels: 0.01 (very high), 0.05 (high), and a = 0.1 (moderate). Co-Boosting exhibits an accuracy gap of 13.7 26.9% compared to FEDADAM. FENS closes this accuracy gap, being only 0- 3.1% lower than FEDADAM. To achieve this, FENS incurs only 3.8 4.3x more communication than Co- Co-Boosting whereas FEDADAM is 10.9 22.1 x more expensive than FENS.
Figure 2: Test accuracy and communication cost of OFL, FENS and FL on CIFAR-10 dataset under high data heterogeneity.
ure 2
1.2 Related Work
One-shot Federated Learning, Guha et al. introduced one-shot FL, which limits communication to single round. They proposed two main methods: (i) heuristic selection for final ensemble clients, and (ii) knowledge distillation (KD) for ensemble aggregation into single model the server using an auxiliary dataset. Subsequent methods based on KD [10] 22 require large, publicly available

---

datasets similar to local client data for good performance, which are often difficult to obtain [50]. To address this, synthetic data generation using generative adversarial networks (GAN)s has been proposed 77 46). The SOTA Co- Co-Boosting algorithm 7 iteratively generates and refines synthetic data and the ensemble model. In FEDCVAE-ENS [14], clients train variational autoencoders (VAHs) locally and upload decoders to the server, which generates synthetic samples for classifier training. FEDOV [8] trains an open-set classifier at each client to predict "unknown" classes, with the server ensembling these models and using open-set voting for label prediction. Other OFL or face difficulties under high
approaches either do not fully consider data heterogeneity P2 37
data heterogeneity [48]
Another line of research in OFL focuses on aggregating fully trained client model parameters 42 PFNM [45] matches neurons across client models for fully-connecte networks, while FEDMA 42
45
extends this to convolutional neural networks (CNNs) and LSTMs. However, the performance of these methods drops with more complex models. Few theoretical works exist, such as [18] which analyze global model loss for overparameterized ReLU networks. Despite the advances, OFL still exhibits accuracy gap with iterative FL. We show that FENS narrows this accuracy gap while preserving communication efficiency.
Ensembles in Federated Learning. Ensembles have been previously studied in FL for a variety of different goals. FEDDF 24] performs robust model fusion of client ensembles to support model heterogeneity. The FEDBE algorithm [5] uses Bayesian Model Ensemble to aggregate parameters in each global round, improving over traditional parameter averaging. Hamer et al. propose FEDBOOST 12] that constructs the ensemble using simple weighted averaging and analyze its optimality for density estimation tasks. However, these works are designed for standard FL. and rely on substantial iterative communication. In the decentralized edge setting, [8) show that collaborative inference via neighbor averaging can achieve higher accuracy over local inference alone. However, they assume a setting where clients can exchange query data during inference and consider only IID data replicated on all edge devices. The idea of learning an aggregator model closely resembles late fusion techniques in multimodal deep learning [25]. The key difference is that FENS focuses on fusing single modality models trained on heterogeneous data under the communication constraints of federated settings.
2 Description of FENS
We consider a classification task represented by a pair of input and output spaces X' and V. respectively,
The system comprises M clients, represented by [M] = {1, (l..... M) and central server. Each client i holds local dataset DD; C X x y. For a model he X - Z parameterized by 0 E θ S R4, each data point (z, y) E X' x y incurs a loss of {(ha(x), v) where é x y + R. Denoting by D == Use[M] D; the union of all local datasets, the objective is to solve the empirical risk minimization (ERM) problem: mingee D E(aw)eD (h)()).v)
2.1 Federated Learning (FL) and One-shot FL (OFL)
FL algorithms, such as FedAvg 27. are iterative methods that enable the clients to solve the above ERM problem, without having to share their local data. In each iteration t, the server broadcasts the current model parameter 0, to a subset of clients S. C [M]. Each client E S, updates the parameter locally over its respective dataset D, using an optimization method, typically stochastic gradient descent (SGD). Clients send back to the server their locally updated model parameters {0("), i E S.).
Lastly, the server updates the global model parameter to 0 [8a]] Eies, 065).
In One-shot Federated Learning (OFL), the iterative exchanges in FL replaced with one-shot communication of local models. Specifically, each client i seeks model ((1) that approximately solves the ERM problem on their local data: minege TD. E( 8 ((h((x),)) y). and sends (10) to the server. Upon receiving the local parameter ((0), corresponding to parametric model :; haro. the server builds an ensemble model of the form x (I) ΣiE(M] (x). This ensemble model is then distilled into a single global model at the server using either public dataset or synthetic data (generated by the server). Existing OFL algorithms choose weights wi...., WM in three different ways: (i) uniformly at random [46], (ii) based on local label distributions 10]. and (ii) dynamically
adjusted based on generated synthetic data [7].
3

---

2.2 FENS
In FENS, the server builds the ensemble model using a generic aggregator fx ZM 2, parameter- ized by À E A C Ro to obtain global model
X - Z defined to be
<() = (m) TM(x)).
In case of standard aggregators such as weighted averaging. q = M and E (w E R ΣE wi 1), and ( ar (z)) = Eie (z). In general, fx can be non-linear trainable model such neural network. The overall learning procedure in FENS comprises two phases:
E
Local training and one-shot communication: Each client does local training compute (61, identical to OFL, and sends it to the server.
2. Iterative aggregator training: Upon receiving the local parameters g(), the server reconstructs
= hgt and obtains À that approximately solves the following ERM problem:
Σ (R.N)ED
min AEA
D
The above ERM problem is solved using an iterative FL scheme (described above). For doing so, the server transmits the set of local models {TI,...,, TTM } to all the clients. The final model is given by T(x) = fx (I). (z))
When solving for using iterative FL, only the aggregator parameters are transferred between the server and the clients. As we show through experiments, in the subsequent section, training an aggregator model is much simpler than training the local models πy, and shallow neural network suffices for fx- Algorithms I and (Appendix O provide the pseudo for FENS.
Connection with stacked generalization. The of trainable aggregator corresponds to instance of stacked generalization [4 in the centralized ensemble literature, wherein the aggregation function is regarded as level 1 generalizer, while the clients' models regarded as level 0 gener- alizers. It has been shown that level generalizer serves the role of correcting the biases of level 0 generalizers, thereby improving the overall learning performance of the ensemble (44). While stacked generalization has been primarily studied in centralized settings, through FENS we show that this scheme can be efficiently extended to an FL setting.
Experiments
We split evaluation into the following sections: (i) FENS OFL in Section 3.2 (ii) FENS vs
FL and analysis of when FENS can match FL in Section 3.3, (iii) FENS on real-world cross-silo
Flamby benchmark [32 in Section 3.4 (iv) FENS language dataset in Section 3.5 (v) dissecting
components of FENS in Section 3.6 and (ví) enhancing FENS efficiency in Section 3.7
3.1 Experimental setup
Datasets. We consider three standard vision datasets with varying level of difficulty, including
SVHN [30], CIFAR-10 (30) and CIFAR-100 30). commonly used in several OFL works 7 10 46
well one language dataset AG-News 47 Vision experiments involve 20 clients, except in the scalability study, where client numbers vary; and AG-News uses 10 clients. The original training splits of these datasets are partitioned across clients using the Dirichlet distribution Dir (a). in line with previous works 77 [4). The parameter a determines the degree of heterogeneity. with lower values leading more heterogeneous distributions (see Appendix A Figure 9.. For our experiments involving the realistic healthcare FLamby benchmark, we experiment with 3 datasets: Fed- SiddCCmellyon6, Fed-Heart Deeeer-D and Fed-1SIC2019. Table (Appendix presents an overview of the selected tasks. The datasets consist of a natural non independent and identically distributed (non-IID) partitioning across clients. FENS, each client performs local training using 90% of their local training data while reserving 10% for the iterative aggregator training. We observed that by splitting the datasets, we achieve better performance than reusing the training data for aggregator training. For fairness, OFL and FL baselines run with each client using 100% of their dataset for

---

local training. The testing set of each dataset is split (50-50%) for validation and testing. We use the validation set tune hyperparameters and always report the accuracy on the testing split.
One-shot FL baselines. We compare FENS against 6 one-shot baselines: (i) one-round FE-
DAvG 277; (it) FEDENS [1] the first one-shot method constituting an ensemble with uniform
weights; (iii) FEDKD [0] based on auxiliary dataset; (iv) one-shot version of FED-ET [6]; (v) the
data-free FEDCVAE-ENS 14: and (vi) Co-Boosting 77. based on synthetic data generation. We use
the best-reported hyperparameters in each work for the respective datasets wherever applicable or tune them. Appendix B.1 provides additional details regarding the one-shot baselines.
Iterative FL baselines. For comparison with FL, we consider 6 algorithms: (i) FEDAVG [27]: (ii) FEDPROX (23]: (iit) FEDNOVA 43 (iv) SCAFFOLD (0); (v) FEDYOGI [3]; and
(vi) FEDADAM 33 We tune learning rates for each algorithm. In addition to these baselines,
we implement gradient compression with FEDAVG STC, following the sparsification and quantization schemes of STC [29]. In particular, we set the quantization precision to 16-bit and sparsity level to 50%, to reduce the communication cost of FEDAVG by 4x and keep the remaining setup to the same as above baselines. For the FLamby benchmark experiments, we use the reported hyperparameters which were obtained after extensive tuning, except with one difference. The authors purposefully restricted the number of rounds to be approximately the same as the number of epochs required to train on pooled data (see (32)) ). Since this might not reflect true FL performance, we rerun all FL strategies to convergence using the reported tuned parameters. Precisely, we run up to 10x more communication rounds than evaluated in the FLamby benchmark. We include more details on FL baselines in Appendix B.2
FENS For the CIFAR-10 10 and CIFAR-100 datasets, each client conducts local training for 500 epochs utilizing SGD as the local optimizer with an initial learning rate of 0.0025. For the SVHN and AG-News datasets, local training extends to 50 and 20 epochs respectively with learning rate of 0.01. The learning rate is decayed using Cosine Annealing across all datasets. For the FLamby benchmark experiments, each client in FENS performs local training with the same hyperparameters
the client local baselines of FLamby. We experiment with two aggregator models, a 2-layer perceptron with ReLU activations and another that learns per-client per-class weights. We train the aggregator model using the FEDADAM algorithm where the learning rate is separately tuned for each dataset (Table g Appendix B. To reduce the communication costs corresponding to the ensemble download, we employ post-training model quantization at the server from FP32 to INT8. Appendix B.3 provides more details on FENS.
Configurations. In line with related work 10 24 35,,
we use ResNet-8 13 as the client local model
for our vision tasks and fine-tune DistilBert 34 for our language task. Our FLamby experiments use the same models as defined in the benchmark for each task (see Table Appendix A}. We report the
average results across at least random seeds. For iterative FL baselines, the communication cost corresponds to the round in which the best accuracy is achieved.
3.2 FENS vs OFL
To assess FENS'S efficacy, we experiment non-IID settings, varying a E (0.01, 0.05, 0.1), and present results across datasets and baselines in Table Our observations reveal challenges for one- shot methods under high heterogeneity, with the optimal baseline differing across settings. FEDAVG with one round exhibits the poorest performance. While FEDCVAF-FENS maintains consistent accuracy across various heterogeneity levels and datasets, struggles particularly with CIFAR- 10 and CIFAR- indicating challenges in learning effective local decoder models. Regarding distillation- based methods, FEDKD and Co-Boosting demonstrate similar performance on SVHN and CIFAR- 10. However, FEDKD outperforms Co- Co-Boosting on CIFAR- 100, facilitated by the auxiliary public dataset for KD while Co-Boosting arduously generates its synthetic transfer dataset. FED-Ei FED-ET improves over FEDENS and is also competitive FEDKD. Notably, FENS consistently outperforms the best baseline in each scenario, except for SVHN at a = 0.01, where FEDCVAE-ENS excels. FENS achieves significant accuracy gains, surpassing the best baseline by 11.4 26.9% on CIFAR-10 and 8.7 15.4% on 100, attributed to its advanced aggregator model.
We chart the client communication costs incurred by all algorithms in Figure 3 The clients in FENS expend 3.6 - 4.3x more than one-shot algorithms owing to the ensemble download and iterative
"oource code available at httts:://gitiiii https/hwww.cmmm//scccee/t///tttt
5

---

Table FENS vs shot FL for various heterogeneity levels across datasets. The highest achieved accuracy is presented as bold and the top-performing baseline is underlined. The rightmost column presents the performance difference between FENS and the top-performing baseline.
Method
FEDAVG
FEDENS
FEDKD
FED-ET
FEDCVAE
Co-Boosting
FENS
04.22=0.58
16.59
4.
20.37
07.84
14.76=2.14
44.46
+15.4
0.05
05.53
20.56-3.51
30.01
33.20.1.34
07.90
20.28-1.94
49.70...5
+-0.66
CF-100
0.1
06.04
27.41=2.1
42.38
38.53=104
08.05=0.09
25.50=0.66
51.11
+8.7
0.01
10.35=0.20
15.66=6.11
18.59+2.22
16.94
29.98
24.97=4.72
44.20
+14.2
CF-10
0.05
13.56
39.56
38.81 8..00
37. 87.51=22.8
29.40=2.58
41.25=5.88
68.22
4.39
+-26.9
=7.38
48.40=0.01
64.14+5.17
47.06=2.a1
32.10=1.79
58.24
75.61
+11.4
=1.35=7.00
20.31=3.49
23.62 61.11.10.1
12.63
69.71 1:021
25.32
57.35=12.4
12.3
12.85=5.84
38.91-7.28
37.41
41.14
70.63
46.36
76.76
+6.1
SVHN
0.1
29.80
51.99
61.38
58.9142.1
72.38 10.25
59.19
83.64 0.75
+11.2
(8) CIFAR-100
(b) CIFAR-10 10
(c) SVHN
Ciient comm. cost (log scalle)
FedOVAE Enn
One-shop (All)
FENS
10 G1B
3.7
3.7
3.7
GIB
100 MiB
0.03
0.09
0.53
0.01
0.05
0.1
0.01
0.05
0.1
0.01
0.05
0.1
Data heterogeneity (a)
Bar labels indicate normalized communication
OFL
Figure 3: Total communication cost of FENS against OFL baselines. The clients FENS expend
roughly 3.7 4.3x more than OFL in communication costs.
aggregator training. While these costs are greater than OFL, they are significantly lower than the costs incurred by iterative FL. baselines as shown in Section 3.3 FEDCVAE-ENN has the lowest cost since the clients only upload the decoder component of their VAE model to the server. However, also suffers from significant performance gap with respect other OFL baselines and FENS on the CIFAR- 10 and CIFAR-IOK 100 datasets.
Varying number of clients.
We also assess the performance of different baselines by varying the number of clients. We consider the CIFAR-10 dataset with 0.1 and present the results in Table FENS achieves the best accuracy surpassing the best-performing baseline FEDKD by 5.9 - 11.4% in accuracy points. This again demonstrates the benefits of utilizing an advanced ensemble model with trainable aggregator function.
Table 2: FENS vs one-shot FL on CIFAR-10 with varying number of clients.
M
FEDAVG
FEDENS
FEDKD
FED-ET
FEDCVAB
Co-Boosting
FENS
5
32.00
62.70
71.74:4.11
47.83
27.7
60.86
77.70
+5.9
10
24.00
46.80
15.85
49.22:4.81
30.00:2.21
54.10
76.44
+9.5
20
17.38
48.40.9.01
64. 14.1415.17
47.06
32.10
58. 18.24
75.61:11.8
+11.4
50
18.5718.68
49.89
59.26
51.5912.a1
34.59
51.38
70.53 11.0
+11.2
3.3
FENS Iterative FL
We now compare the accuracy and communication cost of FENS against iterative FL baselines. After our extensive evaluation of all 6 iterative FL baselines (Table TO Appendix ( on the CIFAR-10 10 dataset across various heterogeneity levels, we find that FEDADAM and FEDYOGI consistently perform the best. Hence, we show FEDADAM our remaining evaluations. Figure presents the results, showing FEDADAM as the representative of FL., FEDAVG STC as the representative of gradient compression, FEDKD as the representative of OFI,, and FENS. Moreover, we show two versions of FEDKD, additional with multi-round support to match the communication cost

---

(a) CIFAR-100
CIFAR-10
(c) SVHN
N FEDKD 1R
FECKD 4R
FENS
FEDAVG STC
FEDADAM (Matching FENS accuracy)
FEDADAM (Max accuracy)
80
accuracy
60
40
20
0
0.01
0.05
0.05
0.1
0.05
Client comm. cost (GIB]
100
04
06
80 84
94
26
34
25
10
10
3.7
3.7
3.8
4.3
4.2
3.8
4.2
3.7
0.01
0.05
0.1
0.01
0.05
0.1
0.01
0.05
0.1
Data heterogeneity (a)
Bar labels indicate normalized communication cost w.r.t. OFL
Figure 4: FENS against iterative FL. The R indicates the number of global rounds, signifying the multi-round version of the OFL baseline. FENS achieves accuracy properties of iterative FL (FEDADAM) with modest increase in communication cost compared to OFL (FEDKD). Numerical accuracy results are included in Table TT (Appendix D).
of FENS (details in Appendix B..2 accuracy of FENS and the other with its maximum accuracy to facilitate effective comparison of communication costs.
We also show two versions of FRDADAM, achieving the
We observe that FENS with its iteratively trained aggregator significantly closes the accuracy gap between OFL (FEDKD) and FL (FEDADAM) across all datasets and heterogeneity levels. Remarkably, the boost achieved sufficient to match FEDA DAM's s accuracy at = {0.01, 0.05) on the CIFAR- 10 dataset. This comes at only modest increase in communication costs which are 4x that of OFL across all cases. We observe that FEDADAM incurs 30 80x more communication than OFL to reach the same accuracy as FENS. Even adding multi-round support to FEDKD only marginally improves its performance. While the best accuracy achieved by FEDADAM still remains higher, also comes significant communication costs of 47 96x that of OFL. Furthermore, that communication compression (FEDAVG STC) fails to preserve the accuracy of FL. under high heterogeneity. Thus FENS achieves the best accuracy vs. communication trade-off, demonstrating accuracy properties of iterative FL while retaining communication efficiency of OFL.
observe
3.3.1
When can FENS match iterative FL?
In this section, we aim to understand when FENS can match iterative FL. The performance of ensembles depends upon (i) the quality of local models; and (ii) data heterogeneity. The quality of local models in turn depends on the amount of local data held by the clients. As local models improve at generalizing locally, the overall performance struggles to generate good global model when the local datasets of clients significantly differ. Thus more volume of data does not analogously benefit FL due to high data heterogeneity. However, as heterogeneity reduces, FL excels and benefits significantly from collaborative updates. This suggests that FENS under sufficiently large local datasets and high heterogeneity can match iterative FL's performance. We confirm this intuition through the following experiments the SVHN dataset.
the ensemble is enhanced. In contrast, FL
Setup. We study the performance of FL and FENS by progressively increasing the volume of data held by the clients. To this end, we consider the classification task on the SVHN dataset due to the availability an extended training set of 604 388 samples, i.e., Rs 10x bigger than the default SVHN dataset. We then experiment with fractions ranging from 10 to 100% of the total training set. Each client locally utilizes 90% for one-shot local model training and reserves 10% for iterative aggregator training, similar to previous sections. We then compare FENS with FEDAVG (FL baseline) three

---

FENS
FedAvg
a 0.05 (non-IID)
0.1
(IID)
Test accuracy [9%]
80
70
0.5
0.7
0.1
0.3
0.5
Fraction of training data
Figure 5: Accuracy of FENS for increasing dataset size. Performance of FENS rapidly increases as the data volume increases. At high data heterogeneity, FENS matches iterative FL's accuracy.
levels of heterogeneity: a = {0.05, 0.1, varying from highly non-llD to IID. We tune the learning rate for FEDAVG (details Appendix (B.)) and keep the remaining setup in previous experiments.
Results. Figure 5howss the results and confirms our prior insight behind the effective performance of FENS. Specifically, observe that the growing volume of training data benefits FENS much more than FEDAVG. When the data distribution homogeneous (a = 1), the performance of FENS improves faster than FEDAVG, but still remains behind. On the other hand, under high heterogeneity (a = ), FENS quickly catches up with the performance of FEDAVG, matching the same accuracy when using the full training We conclude that under regimes of high heterogeneity and sufficiently large local datasets, FENS presents practical alternative to communication expensive iterative FL.
3.4 Performance on real-world datasets
In this section, we evaluate the performance of FENS on the real-world cross-silo FLamby benchmark [32). Specifically, we present iter- ative baselines and 2 one-shot baselines along with the client local baselines (Figure (). For the one-shot FEDAVG and FEDPROX OFL base- lines, we additionally tune the number of local updates. FEDKD is infeasible in these settings since it requires a public dataset for distillation, unavailable in the medical setting. FEDCVAE- ENS and Co-Boosting are also infeasible due to the difficulty learning good decoder models or synthetic dataset generators for medical input data, conclusion supported by their poor per- formance on the comparatively simpler CIFAR- 100 task (Table D. Figure 6 shows the results with the first row comparing FENS against iter- ative FL algorithms and the second row against one-shot FL and the client local baselines.
Fed Cameeyoo16 (AUC)
Fed-Heart- Disease (Accuracy)
Fed-ISIC2019 (Balanced Acc.)
FENS
FedAdam
FedProx
FedYogi
Scaffold
FedAvg
0.8
0.6
Performance metric
FedAvg one-shot
FedProx one-shot
Local client baselines
0.6
0.4
Figure 6: FENS in FLamby. FENS is on par with iterative FL (row-1), except when local models weak (Fed-ISIC2019) while remaining superior in the one-shot setting (row-2). Numerical results
When comparing iterative FL., observe that dataset
FENS is on par for the Gord-Hear--Diseeee and performs better for Fed-Came Camelyon The iterative FL performance is affected by high het- erogeneity [32] where the deterioration more significant for Fed-Camely God-Camelyon 16, 6, which learns on large breast slides (10000x than for Fed-Heart- Disease, which learns on tabular data, In such scenarios of heterogeneity, FENS can harness diverse local classifiers through the aggregator model to attain good performance. On the Fed-ISIC2019 dataset, however, FENS does not reach accuracy iterative FL. Clients Fed-ISIC2019 exhibit high variance in local data amounts and model performance, with the largest client having 12k samples and the smallest only 225 (Table Appendix AA). We thus speculate that the Fed-ISIC2019 datasct falls within the low local training fraction regime depicted in Figure 5 exhibiting larger
included in Tables 12 to 17 (Appendix D

---

accuracy gap compared to iterative FL. However, we note that FENS achieves superior performance over one-shot FEDAVG and one-shot FEDPROX, while performing least as well as the best client local baseline across all datasets. Overall, these observations for FL. algorithms have spurred new interest in developing a better understanding of performance on heterogeneous cross-silo datasets 32] We show that FENS remains strong competitor in such settings.
3.5 Performance on language dataset
We now study the performance of FENS the AG-News dataset, com- paring it against top-performing baselines FEDADAM and FEDKD in the iterative and one-shot categories, respectively. Figure 7 shows the
FedKD (one-shot)
FENS
FedAdam (Iterative)
results: at a = 0.1, FEDKD achieves 71.5% accuracy, leaving gap to
Test accurady Bi]
FEDADAM at 82.3%. FENS effectively closes this gap, reaching 78.8%.
As heterogeneity reduces at a 0.3, all algorithms achieve higher ac- curacies. FENS improves upon FEDKD from 79.3% to 84.5% while
60
FEDADAM achieves 88.3%. Thus we observe consistent results on the language task as our vision benchmarks.
20
0.1
0.3
Data heterogeneily [o:)
3.6 Dissecting FENS
Figure FENS on the AG-News dataset.
We extensively evaluate various aggregation functions (details in Ap- pendix (B.4) the CIFAR-10 dataset across diverse heterogeneity levels.
In particular, we assess static aggregation rules including averaging and weighted averaging, paramet- ric aggregator models including linear model, and a shallow neural network. We also evaluate an advanced version of voting 2 which involves computing competency matrices to reach a collective decision. In addition, we evaluate the Mixture-of-Experts (MoE) aggregation rule (36] where only the gating function is trained via federation. Figure g iillustrates the accuracy, communication costs, and breakdown for all aggregations. Trained aggregator models outperform static aggregations, incurring additional costs for ensemble download and iterative training. The NN aggregator emerges as the top performer, offering the best accuracy vs. communication trade-off. Notably, the iterative training cost of the NN aggregator model for several rounds is lower than the OFL phase itself. Regarding 0.01, where extreme heterogeneity induces expert
accuracy, only MoE outperforms NN specialization, while the trained gating network accurately predicts the right expert. However, MoE's performance declines as heterogeneity decreases while its communication costs remain higher due to the size of the gating network.
Average
Weighted average
Voting
One-shot local training Aggregator training
Ensemble download
Linear
Neural network
MoE
0.01
Client comm. cost (GIB)
Test accuracy [19]
80
10
MoE
60
NN
Linear
40
5
Voting
W. Avg.
20
Avg.
0.01
0.05
0.1
0.01
0.05
0.1
10
Data heterogeneity (ax)
Client comm. coat [GiB3]
Figure 8: Accuracy of different aggregation functions on the CIFAR-10 100 dataset. NN offers the best accuracy vs. communication trade-off, with its iterative training taking up only fraction of the total cost. Numerical accuracy values are included in Table (Appendix
3.7 Enhancing FENS efficiency
The FENS global model comprises the aggregator model stacked atop the ensemble of client local models. Although FENS achieves strong accuracy, the ensemble model can computationally and memory intensive. We used FP32 to INT8 quantization in our previous experiments which reduces the memory costs by (Appendix B.3). In this section, we explore two additional approaches to reduce FENS'S overheads.

---

What if we distill FENS into a single model? To enable efficient inference, we can distill the FENS global model into single model the server using KD once the training is completed. Specifically, we distill the FENS ensemble model comprising 20 ResNet-8 client models and the shallow aggregator neural network into a single ResNet-8 model. Table [ presents the results on the CIFAR-10 10 dataset for a {0.01, 0.05, J.1} distilled using CIFAR- 100 as the auxiliary dataset. We observe slight accuracy drop arising from the process of distillation, which is standard behavior [39]. While we distill using the standard distillation algorithm [15], we note that this accuracy gap can be further reduced through the use of more advanced distillation methods [6 49
Table 4: FENS vs FEDADAM under similar memory footprint on CIFAR- 10. DS stands for downsized.
Table 3: Accuracy of FENS after distilla- tion on the CIFAR- 10 dataset.
Algarithm
a- 0.01
0.05
0.1
Memory (MiB)
Algorithm
a= 0.01
a 0.05
0.1
FIENS 43.3212.:2
66.90
74.14
377.03
FENS 44
086.2--0
15.61
FENS-DS
43.08::141
64-59.3774
22-43-216
17.95
PENS distilled
13.81+258
65.56 65.56-4.25
71.39+121
PEDADAM
39.32177a
68.74
18.73
18.85
PIDADAM-DS
29.69:3.02
63.77
72.25
0.88
What if we match the memory footprint of FENS FEDADAM? While the above approach enables efficient inference, we still need to mitigate training time costs. We now consider downsized (DS) version ResNet-8, where the width of a few layers is reduced so that the total size of the FENS downsized (FENS-DS) model approximately matches the size of the single model in FEDADAM. presenssssents the results on the CIFAR 10 dataset for various heterogeneity levels. Note that no quantization is considered for FENS and FENS-DS, hence the contrast in values with Table Under a comparable memory footprint, FENS-DS remains competitive with the original FEDADAM, with only slight drop in accuracy compared to FENS. On the other hand, using the downsized model as the global model in FEDADAM-DS results in a significant accuracy drop (from 39.32% to 29.69%) under high data heterogeneity (a = 0.01). Thus, the memory overhead of FENS can be alleviated while retaining its communication benefits without too much impact on accuracy.
4 Discussion and Conclusion
Limitations. One limitation is the memory required on client devices to store the ensemble model for aggregator training. We explored quantization and downsizing to mitigate this issue. Future work could investigate aggregator models that do not require access to all client models in the ensemble. This memory issue is only present during training; after training, FENS can be distilled into a single global model on the server, enabling efficient inference as shown in Section 3.7 Another limitation is the increased vulnerability to attacks during iterative aggregator training. unlike OFL, which limits the attack surface to one round. However, this only affects the aggregator model, since the client local models are still uploaded in one shot. Privacy can be further enhanced in FENS through techniques such as differential privacy [9] or trusted exccution environments (28). Specifically, clients can use differentially private SGD for local training, providing a differentially private local model for the ensemble, while the aggregator training could leverage a differentially private FL algorithm [31]
Benefits. In addition to low communication costs and good accuracy, FENS provides three important advantages. First, it supports model heterogeneity, allowing different model architectures across federated clients [21]. Second, FENS enables rapid client unlearning [4], towards the goal of the right to be forgotten GDPR 26). In FENS, unlearning a client can be achieved by simply re-executing the lightweight aggregator training by excluding the requested clients' model from the ensemble. This is more efficient than traditional FL, where disincorporating knowledge from a single global model can be costly. Lastly, if a small server-side dataset is available, such proxy dataset for bootstrapping FL B3] 19]. FENS can train the aggregator model on the server. This makes FENS applicable in model market scenarios of OFL 7 where clients primarily offer pre-trained models.
To conclude, we introduce FENS, a hybrid approach combining OFL and FL. FENS emphasizes local training and one-shot model sharing, similar to OFL, which limits communication costs. It then performs lightweight aggregator training an iterative FL-like fashion. Our experiments on diverse tasks demonstrated that FENS is highly effective in settings with high data heterogeneity, nearly achieving FL. accuracy while maintaining the communication efficiency of OFI.. Additionally, FENS supports model heterogeneity, rapid unlearning, and is applicable to model markets.
10

---

Acknowledgments
Nirupam is partly supported by Swiss National Science Foundation (SNSF) project 200021 200021_20047, 2004777
"Controlling The Spread of Epidemies: A Computing Perspective". The authors thankful to Milos Vujasinovic and Sayan Biswas for their helpful discussions, and to the anonymous reviewers of NeurlPS 2024 for their valuable time and constructive inputs that shaped the final version of this work.
References
[1] Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, llya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. Proceedings of the ACM SIGSAC conference computer and communications security, pages 308-318, 2016.
[2] Ruth Ben- Yashar and Jacob Paroush. Optimal decision rules for fixed-size committees in polychotomous choice situations. Social Choice and Welfare, 18(4):737-746, 2001.
[3] Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloé Kiddon, Jakub Konečný, Stefano Mazzocchi, Brendan McMahan, Timon Van Overveldt, David Petrou, Daniel Ramage, and Jason Roselander. Towards federated learning at scale: System design. In MLSys, 2019.
[4] Lucas Bourtoule, Varun Chandrasekaran, Christopher Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 202/ IEEE Symposium on Security and Privacy (SP), pages 141-159. IEEE, 2021.
[5] Hong-You Chen and Wei-Lun Chao. [ed Making bayesian model ensemble applicable federated learning. In International Conference on Learning Representations, 2021.
[6] Yae Jee Cho, Andre Manoel, Gauri Joshi, Robert Sim, and Dimitrios Dimitriadis. Heteroge- neous ensemble knowledge transfer for training large models in federated learning. In Lud De Raedt, editor, Proceedings of the Thirty-Firy International Joint Conference on Artificial Intel- ligence, IJCAI-22, pages 288 2881-2887. International Joint Conferences Artificial Intelligence Organization, 2022. Main Track.
[7] Rong Dai, Yonggang Zhang, Ang Li, Tongliang Liu, Xun Yang, and Bo Han. Enhaneing one- shot federated learning through data and ensemble co-boosting. In The Twelfth International Conference on Learning Representations, 2024.
[8] Yiqun Diao, Qinbin Li, and Bingsheng He. Towards addressing label skews in one-shot federated learning. In International Conference on Learning Representations. 2023.
[9] Robin Geyer, Tassilo Klein, and Moin Nabi. Differentially private federated learning: A client level perspective. arXiv preprint arXiv:1712.07559 2017.
[10] Xuan Gong, Abhishck Sharma, Srikrishna Karanam, Ziyan Wu, Terrence Chen, David Do- ermann, and Arun Innanje. Preserving privacy in federated learning with ensemble cross- domain knowledge distillation. Proceedings of the AAAI Conference on Artificial Intelligence, 36(11): (111)::1191-11 1899, Jun. 2022.
[11] Neel Guha, Ameet Talwalkar, and Virginia Smith. One-shot federated learning, arXiv preprint 1rXiv: / 2019.
[12] Jenny Hamer, Mehryar Mohri, and Ananda Theertha Suresh. Fedboost: A communication- efficient algorithm for federated learning. In International Conference on Machine Learning, pages 3973-3983. PMLR, 2020.
[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.

---

[14] Clare Elizabeth Heinbaugh, Emilio Luz-Ricca and Huajie Shao. Data-free free one-shot federated learning under very high statistical heterogeneity. In The Eleventh International Conference on Learning Representations, 2023.
[15] Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. In NIPS Deep Learning and Representation Learning Workshop. 2015.
[16] Fotis Iliopoulos, Vasilis Kontonis, Cenk Baykal, Gaurav Menghani, Khoa Trinh, and Erik Vee. Weighted distillation with unlabeled examples. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 7024-7037. Curran Associates, Inc., 2022.
[17] Maximilian Ilse, Jakub Tomczak, and Max Welling. Attention-based deep multiple instance learning. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2127-2136. PMLR, 10-15 Jul 2018.
[18] Divyansh Jhunjhunwala, Shiqiang Wang, and Gauri Joshi. Towards a theoretical and practical understanding of one-shot federated learning with fisher information. In Federated Learning and Analytics in Practice: Algorithms, Systems, Applications, and Opportunities, 2023.
[19] Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Ar- jun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning, Foundations and Trends in Machine Learning. 14(1-2):1 1220,, 2021.
[20] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning, pages $132-5143, PMLR, 2020.
[21] Daliang Li and Junpu Wang. Fedmd: Heterogenous federated learning via model distillation. NeurlPS Workshop on Federated Learning for Data Privacy and Confidentiality, 2019.
[22] Qinbin Li, Bingsheng He, and Dawn Song. Practical one-shot federated learning for cross-silo setting. In Zhi-Hua Zhou, editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 1484-1499 1490. International Joint Conferences on Artificial Intelligence Organization, 2021. Main Track.
[23] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429 450, 2020.
[24] Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. Advances in Neural Information Processing Systems, 33:2351-2363, 2020.
[25] Kuan Liu, Yanen Li, Ning Xu, and Prem Natarajan. Learn to combine modalities in multimodal deep learning. arXiv preprint arXiv:/805. 1730, 2018
[26] Alessandro Mantelero. The eu proposal for a general data protection regulation and the roots of the 'right be forgotten'. Computer Law & Security Review, (9(3):2292235 2013.
[27] Brendan McMahan, Fider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas coamuniiation-efficienn learning of deep networks from decentralized data. In Artificial Intelligence and Statistics, pages 1273-1282. PMLR, 2017.
[28) Aghiles Ait Messaoud, Sonia Ben Mokhtar, Vlad Nitu, and Valerio Schiavoni. Shielding federated learning systems against inference attacks with arm trustzone. In Proceedings of the 23rd ACM/IFIP International Middleware Conference, pages 335-348, 2022.
[29] Alessio Mora, Luca Foschini, and Paolo Bellavista. Structured sparse ternary compression for convolutional layers in federated learning. 2022 IEEE 95th Vehicular Technology Conference: (VTC2022-Spring)) pages 1-5, 2022.
12

---

[30] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
[31] Maxence Noble, Aurélien Bellet, and Aymeric Dieuleveut. Differentially private federated learning on heterogeneous data. In International Conference on Artificial Intelligence and Statistics, pages 101 101105 PMLR, 2022.
[32] Jean Ogier du Terrail, Samy-Safwan Ayed, Edwige Cyffers, Felix Grimberg, Chaoyang He, Regis Loeb, Paul Mangold, Tanguy Marchand, Othmane Marfoq, Erum Mushtaq, Boris Muzel- lec, Constantin Philippenko, Santiago Silva, Maria Teleńczuk, Shadi Albarqouni, Salman Avestimehr, Aurélien Bellet, Aymeric Dieuleveut, Martin Jaggi, Sai Praneeth Karimireddy, Marco Lorenzi, Giovanni Neglia, Mare Tommasi, and Mathieu Andreux. Flamby: Datasets and benchmarks for cross-silo federated learning in realistic healthcare settings. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 5315-5334 Curran Associates, Inc., 2022.
[33] Sashank J. Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and Hugh Brendan McMahan. Adaptive federated optimization. In International Conference Learning Representations, 2021.
[34] Victor Sanh, L Debut, J Chaumond, and T Wolf. Distilbert, a distilled version of bert: Smaller, faster, cheaper and lighter. arxiv 2019. arXiv preprint arXiv: 108, 2019.
[35] Felix Sattler, Arturo Marhan, Roman Rischke, and Wojciech Samek. Cfd: Communication- efficient federated distillation via soft-label quantization and delta coding. IEEE Transactions on Network Science and Engineering, 9(4):2025 2038, 2022.
[36] Noam Shazeer, Azalia Mirhoseini, *Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of- experts layer. In International Conference on Learning Representations, 2017.
[37] MaungJae Shin, Chihoon Hwang, Joongheon Kim, Jihong Park, Mehdi Bennis, and Seong-Lyun Kim. Xor mixup: Privacy-preserving data augmentation for one-shot federated learning. arXiv preprint arXiv:2006.055 48, 2020.
[38] Nir Shlezinger, Erez Farhan, Hai Morgenstern, and Yonina C. Eldar. Collaborative inference via ensembles the edge. In ICASSP 2021 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8478-8482, 2021.
[39] Samuel Don Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A Alemi, and Andrew Gor- don Wilson. Does knowledge distillation really work? In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[40] Mingxing Tan and Quoc Lc. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105-6114 PMLR, 2019.
[41] Manasi Vartak, Harihar Subramanyam, Wei-En Lee, Srinidhi Viswanathan, Saadiyah Husnoo, Samuel Madden, and Matei Zaharia. Modeldb: a system for machine learning model manage- Data Analytics, HILDA '16,
ment. In Proceedings of the Workshop on Human-In-the-Le New York, NY, USA, 2016. Association for Computing Machinery.
[42] Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khaz- aeni. Federated learning with matched averaging. In International Conference on Learning Representations, 2020.
[43] Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem heterogeneous federated optimization. Advances in neural information processing systems, 33:761 2020.
[44] David H. Wolpert. Stacked generalization. Neural Networks, 5(2):241-259 1992.
[45] Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, and Yasaman Khazaeni. Bayesian nonparametrie federated learning of neural networks. In International conference machine learning, pages 7252-7261. PMLR, 2019.
13

---

[46] Jie Zhang, Chen Chen, Bo Li, Lingjuan Lyu, Shuang Wu, Shouhong Ding, Chunhua Shen, and Chao Wu. DENSE: Data-free one-shot federated learning. Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[47] Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
[48] Yanlin Zhou, George Pu, Xiyao Ma, Xiaolin Li, and Dapeng Wu. Distilled one-shot federated learning. arXiv preprini wrXiv:2009.07999) 2020.
[49] Yichen Zhu, Ning Liu, Zhiyuan Xu, Xin Liu, Weibin Meng, Louis Wang. Zhicai Ou, and Jian Tang. Teach less, learn more: On the undistillable classes in knowledge distillation. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[50] Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou. Data-free knowledge distillation for het- erogeneous federated learning. In International Conference on Machine Learning, pages 12878-12889, PMLR, 2021.
14

---

Organization of the Appendix
Datasets
B
Additional Experimental Details
B.
One-shot FL baselines
Iterative FL baselines
B.3
FENS
B.4
Aggregation rules
FENS pseudo code
Numerical results
Compute resources
Broader impact
A
Datasets
As mentioned in Section 3.1 we focus on classification tasks and experiment with datasets in FLamby benchmark including Fed-Canen Ved-Camelyon16, Teed-Heart-Disasse and Fed-ISIC2019, Table 5 overviews the selected tasks in this work.
Table 5: Overview of selected datasets and tasks in FLamby. We defer additional details to 32
DATASET
(N)
PREDICTION (y)
TASK TYPE
CLIENTS
EXAMPLES
MODEL
METRIC
CLIENT
Fed CedeCCmmlyy
Slides
Tumer on Slide
Binary Classifi
239, 150
DeepMIL
h
ALC
cation
Fed-Heatt Discase
Patient Info.
Heart Disease
Binary Classifi
303, 261, 46, 130
Logistic Regression
Accuracy
cation
Fed-ISIC2019
Dermoscopy
Multi-class
12413, 3954, 3363,
EfficienNerr Lincar layer
Classification
225, 819, 439
Accuracy
0.01
0.05
0
Or 100
Class
5
10
20
0
10
20
0
10
20
10
20
0
10
20
Node identifier
Figure 9: Visualizing the effect changing a the CIFAR-10 dataset. Dot size corresponds the number of samples of a given class in given node.
B
Additional Experimental Details
B.1 One-shot FL baselines
The client local training for all OFL baselines (except FEDCVAE-ENS) as well FENS conducted alike, using the parameters reported in [10] for the CIFAR-10 10 and CIFAR-100 datasets. For the SVHN and AG-News datasets, local training is conducted using the SGD optimizer with learning rate of 0.01 for 50 and 20 local epochs respectively and decayed using Cosine Annealing. All vision datasets use batch size of 16 while AG-News uses batch-size FEDCVAF-ENS, we use the same CVAE architecture and local training parameters as reported by the authors [14. However, for the distillation the server, we use ResNet-8 as the classifier model at the server to maintain faimesss with other baselines. For the distillation phase of FEDKD,
for local training. For
15

---

we use the setup described by the authors 100 without inference quantization. For the one-shot version of FED-ET, we set the diversity regularization parameter to the best value of À 0.05. For both FEDKID and FED-ET, we use 10, CIFAR-100, and TinylmageNet as the auxiliary datasets for distillation for SVHN, CIFAR-IO, and CIFAR-100 datasets respectively. We consider a 60 - 40% split for the AG-News dataset where local training is conducted on the 60% split while the remaining 40% is treated as the auxiliary dataset for distillation at the server in FEDKD. Distillation in Co- Co-Boosting using synthetically generated data also uses the best-reported hyperparameters 77.. For one-round FEDAVG, we additionally tune the number of local epochs performed before aggregation by considering {1, 2, 5, 10, 15, 20} epochs.
B.2 Iterative FL baselines
We experiment with 6 different baselines including FEDAVG, FEDPROX, FEDNOVA, FEDADAM, FEDYOGI, and SCAFFOLD. We perform extensive hyperparameter tuning on the CIFAR- 10 dataset for all levels of heterogeneity as detailed below. The server assumes full client participation, i.e., all clients participate in each round. For vision benchmarks, each client performs 2 local epochs per round using a batch size of 16. For our language task, clients train for 50 local steps in each round using a batch-size of 8. We run FL training until convergence and report the best accuracy achieved. We found all algorithms to converge in less than 100 communication rounds on the vision tasks and in 150 rounds the language task.
Hyperparameter tuning. Below we describe our tuning procedure derived from several previous works 223 32 33,, For the FEDAVG algorithm, tune the client learning rate (T²) over the values {0.1, 0.01, 0,001, 0.0001] separately for every a E {0.01, 0.05, 0.1}. For the FEDPROX algorithm, our grid space was {0.1, 0.01} and {1, 0.1, 0.01} for the client learning rate (m) and the proximal parameter (()) respectively. This was again separately tuned for every value of a e (0.01, 0.05, 0.1). For the FEDYOGI and the FEDADAM algorithm,
consider the grid space of
{0.1, 0.01, 0.001, 0.0001} } and {10, 1, O.1.1.1.... 0.01, 0.001) } for the client learning rate (m) and the server learning rate (125) respectively. This explodes the search significantly when tuning for every value of a. From our tuning results for the FEDAVG and the FEDPROX algorithm, we noticed that the optimal parameter values were the same within the following two subgroups of (0.01, 0.05) and {0.1} (T Hence, to keep the tuning tractable, we tune only for one a in each subgroup and reuse the values for other alphas within the same subgroup. For the FEDNOVA algorithm. we use the version with both client and global momentum which was reported to perform the best (3). We consider the search space {0.005, 0.01, 0.02, 0.05, 0.083 for the client learning rate (1)) as done by the authors (45] and tune separately for every value of a. Finally. for the SCAFFOLD algorithm, we consider the search spaces (0.1, 0.01, 0.001, 0.0001} } and {1.0, 0.1, 0.01} for n and 17 respectively and also tune separately for every a. We first conduct the tuning procedure on the CIFAR- 10 dataset and report the obtained hyperparameters Table 6
Table 6: Best hyperparameters obtained for the different algorithms on the CIFAR-10 10 dataset.
FEDAVG
FEDPROX
FEDYOGI
FEDADAM
FEDNOVA
SCAFFOLD
1
111
1
111
1/s
1
7s
f
0.01
0.01
0.01
0.01
0.01
0.01
0.0001
0.0001
1.0
0.05
0.01
0.01
0.01
0.01
0.01
0.02
0.01
0.1
0.1
0.1
0.1
0.01
0.01
0.01
0.01
0.01
0.005
0.01
1.0
We run all iterative FL baselines using the above parameters and present the results in Table 10 Based on our results in the Table for CIFAR-I0, 10, we observe that FEDADAM and FEDYOGE consistently perform the best. Hence, to keep the experiments tractable, we tune and present just FEDADAM as representative of the iterative FL family for our evaluations on the SVHN CIFAR- 100 datasets in Section 3 and AG-News in Section 3.3 For FEDADAM on the AG-News dataset, we note that the training is conducted using only the 60% split (see Appendix B. 1
to achieve a fair comparison with FEDKD. For our experiments involving the extended SVHN dataset in Section 3.3.1 we again tune the client learning rate (1)) for the FEDAVG algorithm over the search space {0. (0.T, 0.01, 0.001, 0.0001) separately for every E {0.01, 0.05, 0.1).
16

---

FEDAVG with gradient compression. To implement FEDAVG with gradient compression, we followed the sparsification and quantization schemes of STC 29 We use the quantization level of 16-bit and sparsity of 50%. This results in a communication cost reduction of 4x against standard FEDAVG in every round. For each dataset and heterogeneity level, we tune the learning rate over the search space {0.1, 0.05, 0.01, 0.001). We keep the remaining setup the same as FEDAVG.
Multi-round FEDKD. Since FENS incurs a communication cost four times that OFL, we also evaluate FEDKD with multi-round support. We explore two approaches: i) pre-training for 3 rounds using FEDAVG, then applying FEDKD, and ii) using FEDKD followed by 3 fine-tuning rounds with FEDAVG. In the first case, each FEDKD client begins training from the global model produced by FEDAVG, while in the second, FEDAVG starts from the FEDKD model. We observe that pre- pre-training with FEDAVG offers little improvement, likely due to the forgetting effect from multiple local training epochs, whereas fine-tuning with FEDAVG boosts FEDKD performance. For our experiments in Section 3.3] we thus present the multi-round version of FEDKD with fine-tuning. We remark that this multi-round support is still insufficient match the performance of FENS, which achieves significantly higher accuracy shown in Table 7 while incurring similar communication costs.
Table 7: FEDKD under multi-round support on the CIFAR-10 dataset.
3 rounds FEDAVG
FEDKD +
FEDKD
FENS
Dataset
Q
FEDKD
3 rounds FEDAVG
0.01
18.59
19.31
21.81 -4.27
44.20
CF-10
0.05
36.84
37.92
43.26=7.35
68.22
0.1
64.14
63.25:16.22
62.61=6.11
75.61=1885
B.3
FENS
Clients in FENS perform local training similar to OFL baselines as described in Section 3.1 and ap- pendix B.1 Let 2, E RC denote the logits obtained from each client model иÿ for all j E [M] on given input where C' is the number of classes. We use one of the two aggregator models as follows. The first one a multilayer perceptron using ReLu activations and final classifier head as follows: W2 E R z = concat (1. zM) E RMC is the concatenated logit vector and ((x) max(x, 0} is the ReLU function. The parameter i determines the number of units in the hidden layer of this perceptron model. The second one is = Σi= X; Oz where A..... Am E RC are weight vectors and @ denotes coordinate- wise product. This model learns per-class per-client weights as the model parameters. For all datasets, the aggregator model is trained using the FEDADAN DAM algorithm where the learning rate is separately tuned for each dataset. Table & presents the tuned learning rate and tuned training parameters per dataset.
= W (WT22 2) where W1 E IR MCxk
Table 8: Aggregator training in FENS. We use FEDADAM as the FL algorithm with the following client (n)) and server (n)) learning rates. The parameter k corresponds to the weight matrices Wi and W2.
Dataset
Aggregator Model
k
Batch Size
Local Steps
Global Rounds
CIFAR-10
f WF((((2)) 2)
40
1.0
0.001
128
1
500
CIFAR-100
DM O
1.0
0.003
128
1
1000
SVHN
f W (WW() 2)
40
0.1
0.01
128
1
500
AG-News
f [W(((
40
1.0
0.001
128
1
500
f S dr Oz
Red-Heeat-Diseaan
0.1
2
5
16
D A:
1.0
0.0005
64
1
2000
Fed-ISIC2019
(WF()
24
0.001
16
1
2500
Model quantization in FENS. Clients in FENS incur a critical cost of downloading the ensemble model from the server to initiate the aggregator training. To reduce the communication burden on the clients, the server employs post-training model quantization of received client local models from FP32 to INT8, reducing the download costs by 4x. Alternatively, the quantization can also be executed on the client side. The quantization results in a drop A 2% test accuracy for every client model compared to the corresponding non-quantized model. However, the subsequent
17

---

aggregator training phase in FENS provides resilience to this drop in the accuracy of client models in the ensemble. In fact, we observe that the final accuracy achieved after aggregator training is slightly higher when using the quantized models as compared to unquantized models due to the regularising effect of quantization on generated logits. The model quantization also provides reduced memory usage on client devices during aggregator training. We use the standard PyTorch quantization library to implement quantization in FENS.
B.4 A Aggregation rules
Averaging Averaging corresponds to the following static aggregation rule: f -i= A θ z where X; M ....,,.. 1/1 and denotes coordinate-wise product.
Weighted Averaging. In weighted averaging, the X1,---. AM RCC are typically assigned based on
local training dataset statistics. In FEDKD 10 A; = Exelan Eve(s) "
x
n²
where 14
EVE(M)
n. corresponds to the number of samples of class j with client i.
Linear. This aggregation corresponds to having single learnable scalar weight for each client
f E ad 2:22 The learnable parameters in this case consist of the vector [w1, W2,.... wMM]
Neural network (NN). Let 2j E RC denote the logits obtained from each client model xj for all j E [MM] on a given input where C is the number of classes. The NN aggregation corresponds to any neural network-based model ʃ ZM + Z that operates on the logits produced by the client models. Denoting = concal .xr) E C as the concatenated vector of logits, I corresponds to the following 2 layer neural network f W oW""" z) where W1 R RBMCk W₂ E R and (x) max (1:, 0} is the ReLU function. Here & determines the number of units in the hidden layer and controls the expressivity of the network. This aggregation is much more powerful than the previously mentioned aggregations, owing to its ability to discern complex patterns across all M x C logits. The learnable parameters comprise the weight matrices {W1, W.).
Polychotomous Voting. Polychotomous voting 2. was originally developed in social choice theory to reach a collective decision when offered C' alternatives (classification labels our case) in a committee of M experts (clients in our case). This method requires as input: (i) classwise "competency" scores of each client: (r( indicating the probability of -cl-nit to vote for label e when the ground truth is r; (ii) Pprior (r) prior probability distribution over correct alternatives r; and (iii) the "benefit" vector of the committee: B(crr) indicating the committee's benefit in choosing label when the correct class is r. Given this information, Ben- Yashar and Paroush (2 derive a criterion for the optimal decision that maximizes expected utility. This criterion is not computed using a closed-form expression, and we generically express it as
f ; TM: Pi...., Par: Pprior; B)
(3)
where f corresponds to a procedure that evaluates and compares benefits for each choice e E [C] given the competency matrices JM { the priors Pprior and the benefit function B. Since the competency matrices for each client model are not directly available in our distributed setting, we learn them by federation in the network. More specifically, each client computes the competency matrix for every client model in the ensemble on its local data and transfers them to the server. The server then aggregates the received competency matrix to produce the final competency matrix per We further use simple benefit function for our experiments
client to be used in decision- that assigns B(c/r) = when e (correct choice) and B(c/r) = 0 when e A r (incorrect choice) and set the prior to be uniform over the set of labels.
Mixture-of-Expertt (MoE). The MoE aggregation 336 considers both the input and the logits
in the following form: f = Eie(M) G(z)..T(()) Thus, MoE aggregation f : (X, ZM) + Z is more expressive compared to other aggregations which only consider logits f z.M + Z. Here, G: + J0, 1 is called a gating network that generates scalar weights for every expert Ti,..., TM based on the input x E X. In FENS with MoE aggregation, only the gating network trained via the federation in the network while }M correspond to the locally trained client models. We use a simple CNN with two convolutional blocks comprising ReLU activation and max pooling layers followed by 2 FC layers with ReI.U activation, which in turn is followed by the final classification head. Despite its expressivity, learning a good gating network incurs significant communication costs and remains difficult under heterogeneous data in federated settings.
M
18

---

Algorithm 1:
FENS from server perspective
Require: .V clients, boolean quantize
Procedure PENS_SFRVER(/S
Initialize model with parameters for local training
Send to all M clients
Receive parameters of locally trained models {0(0,
[M]}
quantize then
((s) < quantization_algee
,P32, INT8) vi [M]
Send {o(0) i [M]] to all M clients'
Initialize aggregator model /A with parameters λn
for 1 = 0, 1 until convergence do
10
Select S, C [M] and send them the aggregator parameters
Receive updated parameters {A( i St}
11
Update global aggregator model A = St Eies, x
12
Algorithm 2: FENS from the clients perspective
: dataset D, loss function É, local steps K and client learning rate Procedure
Split D; randomly into 90% Dii and 10% Di2 Receive parameters 0 from the server
on Da Send converged model parameters (10) to server (one-shot)
Obtain (()) through local training
Receive {0(0, E [M]] from the server while Receive aggregator model parameters de from the server do
Initialize (0) <- A
for 1: = 0, 1....... do
10
Sample mini-batch b E Di2
l 18 E (nwee l (
11
X - 1 Ve,
12
(6) Send X back to the server
13
C FENS Algorithm
Algorithm Nottines the role of the server in FENS. The process begins with the server initializing the parameter corresponding the parametric model /o, which it sends all clients for local training (lines 3 Once clients complete their local training, they return their updated models to the server (line 4 If quantized is enabled, the server quantizes all the local models from FP32 INT8 using a quantization algorithm (line 6). The server then redistributes all models back to the clients (line 7). enabling each contribute the aggregation process that follows. In the final stage, the server iteratively trains an aggregator model in FL fashion, which is designed to combine client models into single, improved global model (lines 9 12). During each round, the server selects subset of clients to update the aggregator model (line 10p refining it further with each iteration until convergence.
Algorithm 2 explains the client-side process in FENS. Each client starts by splitting its local dataset into two parts: one for one-shot local training and smaller part for the iterative aggregator training (line 2 Using the received model, clients first train on D. (line and send their converged local model back to the server (line 5p. subsequent rounds, clients receive from the server the aggregator model (line 7) which they refine it locally using D22 (lines 8 12). Finally, clients send the updated aggregator parameters back to the server (line 13]. contributing the global aggregation process.
1 'The server can potentially send all parameters except client's 's own to reduce costs.
2 'The server any FL algorithm for aggregator training, FEDAVG shown for simplicity.
19

---

D Numerical Results
In this section, we include the numerical values in Tables 9 to
corresponding to the plots presented
in Sections 3.3
3.4 and 3.6 for complete reference.
Table 9: FENS aggregation methods on CIFAR-10, Results of Figure
Algorithm
a = 0.01
a
0.05
a = 0.1
Averaging
15.66=6.11
39.56=6633
48.40=9.01
Weighted Averaging
17.62+33.8
37.72+6.35
62.55+6.08
Polychotomous Voting
26.32+2.54
52.28+3342
63.23#1.12
Linear Aggregator
27.03
56.94
67 1..52
NN Aggregator
44.20
68.22 14.19
75.61++185
MoE (Gating)
49.86+1.61
62. 13.62
69.50+2.61
Table 10: FENS SOTA FL algorithms the CIFAR-10 dataset.
Algorithm
at = 0.01
= 0.05
Q 0.1
FEDADAM
39.324=7.855
68.748
78.736+3..522
FEDAVG
37.600=6.125
60.344+1.705
77.062+33.78
FEDNOVA
32.316 13.844
60.280
1..00:0000
78.732 1:.2222
FEDPROX
37.344 660001
59.772
11.413
75.250
FEDYOGI
39.788
7.726
67.168 7:2799
78.980 #3.044
SCAFFOLD
21.824 13.166 168
27.308
65.892423.00
FENS
44.200 =3.297
68.220##41197
75.613+1885
Table 11: FENS vs. iterative FL. Results from Figure 4
FEDKD IR
FEDAVG STC
FEDKD 4R
FEDADAM
FENS
Dataset
(Max acc.)
0.01
28.98 14.55
24.83
32.22
44.46
50.25 10.58
CF-100
0.05
39.01
29.67+0046
42.12+00:3
49.70=0.86
56.62=0.22
0.1
42.38+0.78
31.87+00.8
42.66-0.89
51.11
59.96=0.s0
0.01
18.59
22.70+7.08
21.81
44.20
39.32-7.85
CF-10
0.05
16.84
50.56+3.60
43.26=7.39
68.2224.19
68. 2..76
0.1
64.14
61.93
62.61 16.17
75.61+1.88
78.73 73 =3.55
0.01
10.1
32.64
25.82 L6.96
57.35
66.85 29.2
SVHN
0.05
37.41+9.00
7..48
49.21
76.76=2.98
83.55=1.52
0.1
61.38+3.00
85.32+2.66
74.33+3.31
83.64
88.04=0.06
Table 12: Figure results. FENS vs. iterative FI. F (row 1).
Table 13: Figure 6 results. FENS vs. one-shot
Fl. Fed- Fred-Camelyon16 (row 2).
Algorithm
AUC
Algorithm
AUC
FEDADAM
0..00
Client 0
0.696.10.03
FEDAvG
0.569
Client
0.683 10.067
FEDPROX
0.584
FEDAVG-OS
0.673.1.0007
FEDYOGI
0.513 20.026
FEDPROX-OS
10.043
SCAFFOLD
0.485
FENS
0.715 40.024
FENS
0.715 +0.024
20

---

Table 14: Figure 6 results. FENS vs. iterative
Table 15: Figure results. FENS vs. one-shot
FL Ced--eert- Disease (row 1).
FL Fed- Moar-eert--oo Heart-Disease (row 2).
Algorithm
Accuracy
Algorithm
Accuracy
FEDADAM
0.781
Client O
0.796=0.007
FEDAVG 0.794:00.004
Client
0.750::0.038
FEDPROX
0.792
Client 2
0,553
FEDYOGL
0..05200000
Client
0.599:t.0047
SCAFFOLD
0.792
FRDAVG-OS
0.698.10.04
FENS
0.781 =0.027
FEDPROX-OS
0.732.0.014
FENS
0.781
Table 16: F FENS vs. iterative FL. Fed-ISIC2019 (row 1).
Table 17: Figure 6 results. FENS vs. one-shot FI. Fed-ISIC20T9 (row 2).
Algorithm
Balanced Accuracy
Algorithm
Balanced Accuracy
FEDADAM
0.710=0.011
Client 0
0.573
FEDAvG
0.731
Client 1
0.315
FEDPROX
0.750
Client 2
0.433:10.005
FEDYOGI
0.731
Client 3
0.325
SCAFFOLD
0.746 10.008
Client 4
0.181 +:0008
FENS
L..022
Client 5
0.21
FEDAVG-OS
0.424
FEDPROX-OS
0.421
FENS
0.579=0.022
E Compute Resources
We use a cluster comprising a mix of 2x Intel Xeon Gold 6240 @ 2.5 GHz of 36 hyper-threaded cores and 2x AMD EPYC 7302 @ 3 GHz of 64 hyper-threaded cores, equipped with 4x NVIDIA Tesla V100 32G and 8x NVIDIA Tesla A100 40G GPU respectively. Training of local models can take up to 2.5 hours in wall-clock time depending on the dataset, while FENS aggregator model training executes in under 30 minutes in wall-clock time. The time required for executing the baselines varies significantly depending on the baseline, with up to hours in wall clock time for Co-Boosting. Across all experiments that are presented in this article, including different seeds and hyperparameter tuning, the total virtual CPU and GPU time is approximately 3500 and 8000 hours respectively.
F
Broader Impact
Federated Learning (FL) has significantly advanced srivacy-preservvng machine learning, particularly in sensitive domains like healthcare and finance, by facilitating collaborative model training without sharing raw data. The development of FENS, which combines FL's accuracy with the communication efficiency of One-shot FL (OFL), carries numerous positive implications. By simultaneously reducing communication costs and maintaining high accuracy, FENS enhances the accessibility and practicality of FL. thereby promoting wider adoption, especially in resource-constrained environments. This has the potential to catalyze advancements in privacy-sensitive sectors like healthcare and finance, where FL is extensively utilized.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our claims in the abstract and the introductions are appropriately scoped and well supported through our extensive empirical assessments spanning multiple datasets, baselines and settings.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations of our work in Section 4
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
22

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have included elaborate descriptions regarding the setup and hyperpa- rameters needed to reproduce the paper in Appendix B complementing our description in Section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
23

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We open-source our code at https: bictps//ggithb... http:/tw.ccmmmaccsppp///eeeeee
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //ips.cc//pill wttpcccpppppppppeeeeeeeeeeeeeeee httpa///ggiieee/eeeSebiiisssssss /Cc////oddiiisssseeeeeee ssi.iiiiccyyyyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Yes, we have included all details in Section 3.1 and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Each of our experiments is repeated with at least three random seeds. We report standard deviations in all tables and display the 95% confidence intervals in our plots.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have included the details on compute resources in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't I make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://eerripsccccppptt////tttee
Answer:
[Yes]
Justification: We have reviewed the Code of Ethics and believe our work adheres to these guidelines.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed the broad impacts of our work in Appendix F
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
25

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We affirm that the medical datasets employed in our experiments within the FLamby benchmark were acquired and utilized in strict accordance with their respective licensing agreements and ethical guidelines. We obtained the necessary permissions and approvals from the appropriate authorities and/or institutions responsible for data collection, and we adhered to all relevant ethical standards and regulations. The owners the original assets used in this paper were properly cited and attributed.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
26

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thood cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We open-source our code at: attps://gittub com/sacs-epfl/fen Our repository is well documented with all the instructions to the code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
27

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "0RPT8tr9cc", "reviewer": "Reviewer_3Dng", "summary": "This paper focuses on the one-shot federated learning with the model ensembling to assist the model aggregation. Specifically, after only one-round of model uploading, the authors provide a new aggregator based on a shallow neural network for the global model. The performance indicates that the proposed method has good potential in the one-shot FL setups.", "questions": "1. I am curious about how \"Revisiting\" is discussed in the paper. I do not see any revisiting part in the writing.\n\n2. I suggest the author to list a pseudo-algorithm in the Section 2 to let the audience better understand the proposed method.", "limitations": "Please refer to the Weakness part.", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. One-shot FL is a very practical scenario in real-world setups, and it has not been richly discussed yet.\n\n2. The usage of the Flamby evaluation datasets is very novel and very close to the real setups.\n\n3. The proposed method is easy to implement and very efficient in communication cost.", "weaknesses": "1. The paper lacks several strong baselines in the experiment. Even though the paper focuses on the one-shot FL setups, the design of FENS is very close to the knowledge-distillation-based FL methods, such as FedGKD [1] and Fed-ET [2]. The paper should include them in the evaluation baseline to see whether the proposed FENS could provide SOTA performance. The selected FedKD is a relatively old baseline from which to compare.\n\n2. I suggest expanding the ablation study to discuss the effect of server training in FENS. I noticed that FENS needs to apply global data for training on the server side, which is a very strong setup. I suggest the author elaborate on the impact of the usage of public data.\n\n3. Following my previous point, the usage of public data is sometimes infeasible in the real-world FL setup. In the experiment setup, the author selected proxy data by splitting the original train data. It is a very strong setup. The previous study [2] used the same setups in the paper, but it at least does some distortion to simulate the proxy data coming from a different source. As a result, I feel inconvincible to the \neffect of FENS as the server has access to a portion of training data.\n\n[1]. He, Chaoyang et al. “Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge.” arXiv: Learning (2020): n. pag.\n\n[2]. Cho, Yae Jee et al. “Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learning.” International Joint Conference on Artificial Intelligence (2022).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback and insightful comments. We noticed that there was a confusion regarding the usage of public dataset in our algorithm. We also noticed that most of the identified weaknesses seem to arise from this confusion. Below, we clarify our algorithm design and address the reviewer's questions.\n\n--------------------\n> On the usage of public dataset at the server.\n\n$\\rightarrow$ Our method does not rely on any public dataset. \nIn FENS, the data remains decentralized throughout. \nIn the second phase of the algorithm, FENS trains the shallow aggregator model via standard FL (lines 131-139).\nWe are happy to provide further details or address any additional questions.\n\n--------------------\n> On the lack of strong baselines such as Fed-ET [1] and FedGKT [2].\n\n$\\rightarrow$  In light of the above clarification, we do not consider Fed-ET [1] since it is an iterative FL method that relies on the usage of public data.\nSimilarly, we do not compare to FedGKT [2] since it is an iterative algorithm that focuses on reducing the computational burden of edge clients.\nHence, the objectives and design of the suggested baselines differ from that of FENS.\nIn particular, FENS focuses on simultaneously achieving OFL-like communication efficiency and FL-like accuracy.\nAccordingly, we have considered 5 one-shot baselines and 6 iterative FL baselines that are more relevant to FENS. \nWe will further clarify our choices in the paper.\n\n--------------------\n> Q. I am curious about how \"Revisiting\" is discussed in the paper. I do not see any revisiting part in the writing.\n\n$\\rightarrow$ Ensembling has been studied in OFL prior to our work, and we revisit this concept by applying it in a different and more powerful way. \nUnlike existing methods, FENS employs a shallow aggregator model atop locally trained client models, thereby generalizing traditional aggregation functions. \nBy introducing a novel two-phase training procedure, which keeps data decentralized, FENS significantly improves accuracy while maintaining low communication costs. \nWe will further clarify the \"revisiting\" aspect in our paper.\n\n--------------------\n> Q.  I suggest the author to list a pseudo-algorithm in the Section-2 to let the audience better understand the proposed\nmethod.\n\n$\\rightarrow$ Thank you for the suggestion. We will do so in the final version of the paper.\n\n--------------------\n\n[1] Cho, Yae Jee et al. “Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learning.”\nInternational Joint Conference on Artificial Intelligence (2022).\n\n[2] He, Chaoyang et al. “Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge.” arXiv: Learning (2020):\nn. pag.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fN8T4DSUay", "reviewer": "Reviewer_zQJt", "summary": "This paper proposes a 2-phase mechanism for learning models across clients in federated learning settings with minimal communication. The method, FENS, first uses a one-shot communication to learn copies of a base model across clients' local data. It then uses standard iterative FL approaches to learn a smaller \"aggregator model\", which aggregates some output across the base models.\n\nThe authors provide experiments on FENS, comparing it to one-shot approaches and iterative approaches to FL. Generally, their results show that FENS is superior to other methods when judged in accordance with the total amount of communication, and can also achieve higher accuracy (essentially via ensembling) than other methods with reduced total communication.", "questions": "1. How do the results in Figures 2 and 4 change if we instead consider total client computation instead?\n1. Why does the MoE approach to aggregation require such a large communication overhead? Can't the size of the dense layers used for routing be tuned?\n1. What happens if you apply a method like FedAdam to a model whose total size is comparable to the total FENS model size?", "limitations": "The checklist is adequate.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I think this is a well-written paper. The authors are very clear about how the method operates, how it fits into the greater landscape of FL methods, and their experiments are similarly quite clear in execution.\n\nOf course, the core of the work (and content-wise, most of the work) is in the empirical evaluation of FENS. There are a lot of laudable qualities to their empirical analysis. First, I like that the authors compare FENS to one-shot & iterative methods separately. The metrics of interest are different between these settings, and I think it's important to perform different analyses in such cases. I also appreciated the fact that the authors explicitly do not try to claim that FENS is optimal for every settings - the ablation study in Section 3.3.1 is a really good and honest one, that tries to understand the utility of FENS if accuracy is the predominating concern, not communication-effiicency.\n\nI also really appreciated the use of the FLamby benchmark. Synthetically partitioned datasets like CIFAR are only of limited utility in evaluating methods, and the FLamby benchmark is a great choice for datasets that have inherent, realistic forms of heterogeneity. In fact, I think that FLamby would make a stronger benchmark for the other ablation studies (though more on that below).\n\nLast, I really liked Section 3.5. I think the comparisons between FENS and things like mixture-of-experts is a natural thing to consider, and I found this section intriguing enough that I actually wish it were explored more.", "weaknesses": "There are some weaknesses to the paper, though I would consider them more as opportunities for improvement. In case it was not obvious from my assessment of the strengths, I think this is a good paper and should be accepted. I break these areas up separately below.\n\n### Limitations of synthetically partitioned datasets & vision datasets\n\nMost of the experiments in this work are on SVHN, CIFAR-10, and CIFAR-100. While these are often used in federated learning research, they are quite limiting. The heterogeneity you are able to study with them is generally limited to label heterogeneity (e.g. due to the Dirichlet mechanism employed). This is only one type of heterogeneity, and generally these tasks are still simpler than realistic, very heterogeneous datasets. FLamby is a great example of realistic heterogeneity, but there are other such datasets, including iNaturalist, GLD-v2, and FLAIR.  I think that these would constitute stronger benchmarks for the work.\n\nSecond, I will note that FENS does not specifically require vision tasks. Especially when considering analogies to the mixture-of-experts literature, it is not clear to me why there is no evaluation on a language model task. Language datasets often exhibit interesting heterogeneity and challenge in optimization.\n\n### Trimming Sections 1 & 2 and expanding other discussions\n\nTo be honest, I think that Sections 1 & 2 are somewhat repetitive, and span nearly 3.5 pages before any of the actual content of the work comes to bear. The contributions section itself is more than half a page, and Figure 1 does not give any real insight into FENS that isn't already in the text.\n\nBy contrast, the actual discussion of the aggregator model architecture is in need of expanding. For example, what architectures would actually be useful for an aggregator model? Why use a 2-layer ReLU network? Could you instead view the client models as a large \"backbone model\" and view the entire combination of backbone + aggregator as a large model? What happens if you train this large model directly? These are all examples of questions that I think are not addressed, and would be much more interesting than repeating the details of FENS multiple times.\n\n### Considering other system metrics & ablations\n\nThe primary metric of interest in this work is total communication. This is a perfectly reasonable metric, but it does obscure other metrics of interest. For example, and as is briefly hinted at in Section 4, FENS incurs a large overhead in terms of server memory as it stores all client models separately. By contrast, the iterative FL methods train a single base model. This means that FENS' memory overhead is more than $M\\times$ as much (where $M$ is the number of clients).\n\nWhat would happen if we equalized server memory usage, so that FENS' base model would be smaller? What happens if we enlarge the models trained by methods like FedAdam? For that matter, what if we measure total client computation, instead of communication costs? Some of these are interesting questions that can actually be answered in part by simply taking a different view of the experiments the authors have already performed. I think that giving these ablations would be hugely interesting, and make the paper much better.\n\nNote that this dovetails with the limitation above - that Sections 1 & 2 are too long. By contrast, Appendix C is interesting, and should probably be added to the main body.\n\n### Data minimization and privacy\n\nOne drawback of FENS that is not really discussed (except in L333) is its lack of compatibility with privacy-preserving technologies. The separate one-shot trained base models are stored separately. This means that techniques like differential privacy are not directly compatible. This is a sever weakness of the method. While I don't think it means the paper isn't interesting, I think it should be discussed. Are there potential future directions that could help avoid this issue? Given the fact that FL is predicated on data minimization & privacy, I think some more up front discussion about this would be useful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive assessment of our work and the suggestions for improvements. \nWe address the reviewer's questions below.\n\n-----------------\n> Q. Why does the MoE approach to aggregation require such a large communication overhead? Can't the size of the dense\nlayers used for routing be tuned?\n\n$\\rightarrow$ Yes, we considered tuning the size of the routing network. \nWe had experimented with models of various sizes, and observed that smaller models cause a deterioration in accuracy. \nOur current setup employs a CNN with two convolutional blocks, followed by two fully connected layers and a final classification head (Appendix B.4). This configuration achieves a reasonable balance between accuracy and communication overhead.\nSince the routing network needs to process images as input, it must be a reasonably sized CNN. \nConsequently, this results in a larger communication overhead compared to using a shallow MLP-based aggregator (referred to as the NN aggregator in Fig. 7), which processes concatenated logits.\n\n-----------------\n> Q. What happens if you apply a method like FedAdam to a model whose total size is comparable to the total FENS model\nsize?\n\n$\\rightarrow$ This is an interesting ablation.\nWe evaluated this suggestion by downsizing client local models such that the total size of FENS approximately matches the size of the single model in FedAdam. \nPreliminary results indicate that under a comparable memory footprint, FENS still remains competitive with the original FedAdam.\nOn the other hand, using the downsized model for FedAdam induces an accuracy drop.\nWe will include the complete ablation in the revised version of our paper.\n\n-----------------\n> Q. How do the results in Figures 2 and 4 change if we instead consider total client computation instead?\n\n$\\rightarrow$ There is indeed a trade-off here and we expect the client computation cost of FENS to be the highest in Fig. 2.\nThis can be seen as the cost of FENS in providing OFL-like communication efficiency and FL-like accuracy which are its primary metrics of interest.\n\n-----------------\n> On the limitations of synthetically partitioned and vision datasets.\n\n$\\rightarrow$ We appreciate the reviewer’s recognition of our use of the FLamby benchmark. We are keen on extending our empirical analysis with a language dataset and will incorporate this in the revised version of the paper.\n\n-----------------\n> On trimming Sections 1 and 2 and including other ablations.\n\n$\\rightarrow$ Thank you for the suggestion. We will shrink the first two sections to enable the inclusion of Appendix C and above ablation in the main body of our revised paper.\n\n-----------------\n> On expanding the discussion on privacy.\n\n$\\rightarrow$\nPrivacy is indeed an important concern and we believe that existing privacy schemes could be directly incorporated into FENS.\nOne approach could be to perform client local training using differentially private SGD [1].\nThus each client can provide a differentially private local model for the ensemble. \nSimilarly, the aggregator training could be conducted via a differentially private FL algorithm [2]. \nAs suggested by the reviewer, we will include a discussion of potential directions in our revised paper. \n\n-----------------\n[1] Abadi, Martin, et al. \"Deep learning with differential privacy.\" Proceedings of the 2016 ACM SIGSAC conference on computer and communications security. 2016.\n\n[2] Noble, Maxence, et al. \"Differentially Private Federated Learning on Heterogeneous Data\". Proceedings of the 25th International Conference on Artificial Intelligence and Statistics (AISTATS). 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rAPDEz21CL", "reviewer": "Reviewer_B5VB", "summary": "This paper introduces FENS, a One-shot FL (OL) approach with the aim of improving the globl model's accuracy without significantly increasing the communication cost of canonical OFL. Different from existing OFL methods, FENS employs and iteratively trains a prediction aggregator model stacked on top of the local model ensemble.", "questions": "1. How to determine the best number of iterations for the aggregator model?\n2. How is the MoE method implemented over the local model ensemble? Why did it incur such a high cost?", "limitations": "As mentioned in the Weaknesses section.", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "- The method is easy to reproduce without requiring addtional training data on the server.\n- Extensive experiments were performed to showcase the claimed performance from multiple aspects.", "weaknesses": "- Lack of theoretical analysis to ground the design of the proposed method.\n- FENS requires dispatching the whole set of local models to every client in the system, which raises privacy concerns.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive feedback and  comments. We address the reviewer's questions below.\n\n--------------\n> Q. How to determine the best number of iterations for the aggregator model?\n\n$\\rightarrow$ This is determined as in standard FL schemes by stopping when the validation loss has converged or validation accuracy is not improving further. \n\n--------------\n> Q. How is the MoE method implemented over the local model ensemble? Why did it incur such a high cost?\n\n$\\rightarrow$ MoE aggregation involves both the input and the logits in the following form: $f(x)=\\sum_{i \\in [M]}G(x)\\_i\\pi_i(x)$. Here, $G: \\mathcal{X} \\rightarrow [0,1]^M$ is the gating network that generates scalar weights for every expert $[\\pi_1,\\ldots,\\pi_M]$  based on the input $x \\in \\mathcal{X}$. \nIn FENS with MoE aggregation, only the gating network is trained via the federation while $[\\pi_1,\\ldots,\\pi_M]$  correspond to the (frozen) locally trained client models.\nFor the gating network, we employ a CNN with two convolutional blocks, followed by two fully connected layers and a final classification head. \nSince the gating model has several layers, it incurs a bigger communication overhead of iterative transfers as compared to just an MLP based shallow aggregator (called NN aggregator) in Fig. 7.\nWe also tried using smaller models, however these resulted in accuracy deterioration. \nWe have included the MoE training details in Appendix B.4.\n\n--------------\n> Lack of theoretical analysis to ground the design of the proposed method.\n\n$\\rightarrow$  We acknowledge that deriving a formal theoretical explanation for our empirical findings is an interesting and valuable research direction.\nHowever, as briefly discussed in Section 2.2, our algorithm is motivated by the study of stacked generalization [1] in the centralized ensemble literature. \nIt has been demonstrated that such stacking of models leads to higher-level models correcting for the biases of lower-level models in the stack, thereby improving overall generalization performance.\nWhile stacked generalization has been primarily studied in centralized setting, through FENS, we demonstrate that this scheme can be efficiently realized under the communication constraints of FL through our novel two-phase training procedure.\n\n--------------\n> Dispatching the whole ensemble to clients raises privacy concerns.\n\n$\\rightarrow$ \nWe agree that privacy is an important concern.\nWe would like to note however that FENS still does not require the clients to transfer their local data but only their fully trained models. \nExploring how to ensure that these model exchanges do not leak information about the data is of independent interest, and could have applications in various FL contexts. \nWhile we leave an in-depth exploration for future work, we believe that existing privacy schemes could be directly incorporated into FENS. \nFor instance, local model training can be done using differentially private SGD [2], thereby ensuring differential privacy of local data despite the sharing of the local models. \nSimilarly, the aggregator training can be conducted via a differentially private FL algorithm [3].\nWe will include this discussion in our revised version.\n\n--------------\n[1] David H. Wolpert. Stacked generalization. Neural Networks, 5(2):241–259, 1992.\n\n[2] Abadi, Martin, et al. \"Deep learning with differential privacy.\" Proceedings of the 2016 ACM SIGSAC conference on computer and communications security. 2016.\n\n[3] Noble, Maxence, et al. \"Differentially Private Federated Learning on Heterogeneous Data\". Proceedings of the 25th International Conference on Artificial Intelligence and Statistics (AISTATS). 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aFnRgFHKhg", "reviewer": "Reviewer_y6N3", "summary": "This work proposes a one-shot federated learning method where, instead of simply aggregating the models at the end of training, a non-linear ensemble of the (frozen) models is trained with an iterative federated learning method. In other words, after training the models locally, the server trains a shallow neural network that takes the logits of all the models as input and outputs the aggregated output. This approach is feasible in the cross-silo setting. The experiments show that it can be significantly better than SOTA one-shot methods and it can sometimes approach the performance of full federated averaging (i.e., with multiple rounds of aggregation).", "questions": "- Does the communication cost comparison in Figure 2 assume quantization? If so, does the communication cost for the iterative FL methods also use quantization?\n- The performances reported in Appendix D are impressive, but Table 15 (Fed-ISIC2019) shows a limitation/non-robustness of one-shot methods on a real-world dataset. Why do you think the gap here is large?", "limitations": "The authors adequately addressed the limitations. There are no negative societal impacts associated with the proposed method.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The approach is simple yet highly efficient in the heterogenous cross-silo settings, as shown in Figure 2.\n- The paper is well-written and the experiments cover most of the interesting cases.\n- The authors share the code for reproducing the results, which further improves the quality of the submission. The code itself is well written and reproducibility is made easy given the clear instructions.\n- The results are demonstrated on various datasets, including real-world health data.\n- The discussion regarding client unlearning is interesting and does show a desirable strength in the proposed method.\n- The authors clearly mention the limitations and propose future directions to explore and mitigate some of the issues.", "weaknesses": "The contribution of the paper is a bit limited in terms of technical novelty. I will not base my rating on novelty alone, of course. However, the technical details of the method are a bit straightforward and not new (e.g., the authors mention Stacked Generalization by Wolpert (1992)). Nonetheless, I will try to be fair and rate this paper's contribution in terms of its effectiveness and (reproducible) experimental results, which can be as important as technical novelty.\n\n- Collecting the logits of all clients for every forward pass of the ensemble is not elegant, especially given the memory and computational costs. The authors do acknowledge this limitation and offer some directions to mitigate this, but they still do not offer a pratical solution that scales well. The authors mentioned quantization but did not share any preliminary findings or insights. I think running experiments where some clients are sampled every round of training the ensemble is straightfoward to implement, so it would be interesting to see that. Also, how would you concatenate the logits?\n- It might be somewhat fair to compare with one or two personalized FL algorithms, since the proposed method already assumes a large memory budget per client, so such clients in the cross-silo setting are also quite likely to be able to maintain some state (or at least, have the server maintain it for them).\n- Another intuitive thing to compare to is split learning methods, where some of the layers are trained with iterative FL methods, and the others are kept local. These methods might be able to reach similar accuracies with a more feasible memory and computational budget.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive feedback and constructive comments. We address the reviewer's questions below.\n\n--------------\n> Q. Does the communication cost comparison in Figure 2 assume quantization? If so, does the communication cost for the iterative FL methods also use quantization?\n\n$\\rightarrow$ Yes, FENS is presented with quantization in Figure 2. \nWhile the iterative FL baseline (FedAdam) in Figure 2 does not use quantization, we have included the relevant baseline with quantization, called FedAvg STC in Figure 4.\nThe quantization in FedAvg STC results in an accuracy drop compared to FedAdam.\n\n--------------\n> Q. The performances reported in Appendix D are impressive, but Table 15 (Fed-ISIC2019) shows a limitation/non-robustness of one-shot methods on a real-world dataset. Why do you think the gap here is large?\n\n$\\rightarrow$ This gap can be explained through the quality of local models in case of the FedISIC2019 dataset together with our experimental results from Section 3.3.1.\nOur experiments in Section 3.3.1 show that local data size plays a key role in determining when FENS can match the performance of iterative FL. \nThe FedISIC2019 dataset, in particular, exhibits high variance in the amount of local data held by the clients, and consequently their local model performance.\nAs reported in Table 3 (Appendix A), the biggest client has 12k samples while the smallest client has only 225 samples. \nThus, we speculate that the FedISIC2019 dataset falls in the low local training fraction regime of the curves in Fig. 5, exhibiting a larger gap compared to iterative FL.\n\n--------------\n> Collecting the logits of all clients for every forward pass of the ensemble is not elegant, especially given the memory and computational costs. The authors do acknowledge this limitation and offer some directions to mitigate this, but they still do not offer a practical solution that scales well.\n\n$\\rightarrow$ We currently use FP32 to INT8 quantization to mitigate this issue (Appendix B.3).\nThis reduces the costs by $4\\times$.\nAn alternative and complementary approach is to consider an ensemble such that the total size is comparable to the single model trained in iterative FL.\nPreliminary results indicate that under a comparable memory footprint, FENS still remains competitive with the original FedAdam. \nWe will include the complete ablation in our revised version. \n\n--------------\n> I think running experiments where some clients are sampled every round of training the ensemble is straightforward to implement, so it would be interesting to see that. How would you concatenate the logits?\n\n$\\rightarrow$ This is indeed a highly relevant suggestion.\nWe explored sampling clients each round by setting logits from non-sampled models to zero in the concatenated vector.\nAdditionally, we also tried learning mask tokens that fill for non-sampled logits.\nPreliminary results show an accuracy drop under high heterogeneity, likely because each model’s contribution is significant in such scenarios. \nTo improve performance, a more sophisticated logit concatenation method may be required.\n\n--------------\n> On suggested comparison to personalized FL baselines and Split Learning.\n\n$\\rightarrow$ We thank the reviewer for their insightful suggestions. \nWe agree that Split Learning might help make the computational budget more feasible.\nIn FENS, we focus on simultaneously achieving OFL-like communication efficiency and FL-like accuracy.\nIn this sense, the objectives of Split Learning and personalized FL slightly differ for a direct comparison.\nPersonalized FL focuses on individual client accuracy, while FENS targets global data accuracy. \nSimilarly, Split Learning reduces computational burden, whereas FENS prioritizes communication efficiency.\nWe believe that both these paradigms can be orthogonally explored within FENS and are keen on exploring these aspects in future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: **Summary:**

FENS is a novel approach combining one-shot federated learning with iterative training of an aggregator model to improve accuracy while maintaining low communication costs. It uses a two-phase process: 1) clients train models locally and send them to the server, 2) clients collaboratively train a lightweight aggregator model using federated learning. Experiments show FENS can achieve accuracy close to iterative FL methods while requiring much less communication, outperforming other one-shot FL approaches.

**Strengths:**

- Novel combination of one-shot and iterative FL approaches
- Strong empirical results on multiple datasets, including realistic heterogeneous data
- Clear presentation and comprehensive experiments
- Code provided for reproducibility
- Addresses an important practical scenario (cross-silo FL with limited communication)

**Weaknesses:**

- Lack of theoretical analysis/justification
- Privacy concerns with sharing full client models
- High memory requirements on clients
- Limited technical novelty in core method
- Unclear how to optimally set some hyperparameters (e.g. aggregator training iterations)

**Key Discussion Points:**

- *Comparison to state-of-the-art:* Some reviewers felt more comparisons to recent methods like Fed-ET were needed. Authors provided additional results showing FENS still outperforms these.
- *Privacy and scalability:* Concerns were raised about sharing full client models and memory requirements. Authors acknowledged these limitations and suggested future directions like differential privacy.
- *Theoretical grounding:* While lacking formal theory, authors clarified the motivation from stacked generalization literature.
- *Applicability beyond vision tasks:* Reviewers suggested evaluating on language tasks. Authors agreed to add this in revision.
- *Alternative metrics:* Suggestion to c onsider client computation costs in addition to communication. Authors acknowledged this tradeoff.

Overall, most reviewers were positive about the empirical results and practical relevance, while noting some limitations in theoretical grounding and scalability. The authors' responses addressed many concerns, leading some reviewers to increase scores. The consensus seems to be that this is a solid contribution to federated learning research, particularly for cross-silo scenarios with limited communication budgets.
Decision: Accept (poster)