Data-Free Diversity-Based Ensemble Selection for One-Shot Federated Learning
Authors: Naibo Wang
Wenjie Feng
yuchen deng
Moming Duan
Fusheng Liu
See-Kiong Ng
Venue: Accepted by TMLR ()

Abstract: The emerging availability of various machine learning models creates a great demand to harness the collective intelligence of many independently well-trained models to improve overall performance. Considering the privacy concern and non-negligible communication costs, one-shot federated learning and ensemble learning in a data-free manner attract significant attention. However, conventional ensemble selection approaches are neither training efficient nor applicable to federated learning due to the risk of privacy leakage from local clients; meanwhile, the "many could be better than all" principle under data-free constraints makes it even more challenging. Therefore, it becomes crucial to design an effective ensemble selection strategy to find a good subset of the base models as the ensemble team for the federated learning scenario. In this paper, we propose a novel data-free diversity-based framework, DeDES, to address the ensemble selection problem with diversity consideration for models under the one-shot federated learning setting. Experimental results show that our method can achieve both better performance and higher efficiency over 5 datasets, 4 different model structures, and both homogeneous and heterogeneous model groups under four different data-partition strategies.

Published in Transactions on Machine Learning Research (11/2023)
Data-Free Diversity-Based Ensemble Selection for One-Shot Federated Learning
Naibo Wang', Wenjie Feng!. 1, Yuchen Deng?. 1, Moming Duan', Fusheng Liu', See-Kiong Ng
(naibowwng, fusheng) @ra. [wenjiee. moming, seekiong) @eous. dengguchen..c@@mmeil.co
Institute of Data Science, National University of Singapore School of Mathematics and Statistics, Changchun University of Technology
t: Corresponding author.
Reviewed on OpenReview: http:///pearrieeee..tttooootttto
Abstract
The emerging availability of various machine learning models creates a great demand to harness the collective intelligence of many independently well-trained models improve overall performance. Considering the privacy concern and non-negligible communication costs, one-shot federated learning and ensemble lcarning in data-free manner attract sig- nificant attention. However, conventional ensemble selection approaches are neither training efficient nor applicable to federated learning due to the risk of privacy leakage from local clients; meanwhile, the many could be better than all principle under data-free constraints makes it even more challenging. Therefore, it becomes crucial to design an effective ensem- ble selection strategy find good subset of the base models as the ensemble team for the federated learning scenario. In this paper, we propose novel data-free diversity- based framework, DeDES, to address the ensemble selection problem with diversity consideration for models under the one-shot federated learning setting. Experimental results show that our method can achieve both better performance and higher efficiency over 7 datascts, 5 different model structures, and both homogeneous and heterogeneous model groups under four different partitio strategies.
1 Introduction
In the cra of large models with great appetites for large-scale data for various machine learning tasks, to handle the data island while addressing the increasing demands on data privacy concern for information sharing, federated learning (FI.) (Li et al. 2020) has become the mainstay for enabling collaborative machine without accessing private data.
learning on decentralized
Traditional federated learning often requires multi-round training process, and 0(mn) gradients models will be acquired for the scenario with m clients and n update rounds, resulting in great potential risk for the leakage of local data and violation of the privacy-preservin principle (Zhu et al. 2019 Geiping et al. 2020). Onc-shot federated learning (Guha et al. 2018 Su et al.) 2023) is proposed to alleviate the above issues, which only requires the clients send their well-t models to the server once, while the communication
costs are significantly roduced. However, is foresocable that the performance of the resulting models one-shot. FL will often be inferior to that in conventional FL (Yurochkin al.) 2019 Lamn al., 2021). As consequence, one-shot FL models cannot be competent in some critical and highly demanding fields, such medical diagnosis and financial regulation.
Other model-centric research lines include model fusion Kasturi et al. 2020). knowledge distillation Zhu et al.) 2021), and ensemble learning (Zhou 2021 Sagi & Rokach 2018 these, ensemble learning is straightforward and cost-cffective harnessing the power of collective models to boost task performance in a data-free manner. For instance, majority voting, the commonly used classical method, produces the final prediction results depending the voting of multiple models. However, given the principle that many

---

Published in Transactions on Machine Learning Research (11/2023)
Datasets Training
Models
Upload
Ensemble Learning
Samples
Centrol Model Server
Same Task Models
Ensemble Teams
Dra GeedRasFL
Ensemble Selection Framework (DeDES)
Dre Fe
Illo
Figure Overview of ensemble learning and ensemble selection process under one-shot federated learning.
could be better than all (Zhou et al. 2002), it may not always be the most effective strategy for the central server in FL to select all available client models, especially for neural network models. Meanwhile, it is time-consuming and inefficient to test cach incoming sample on all models when the ensemble team contains a large number of models. Hence, to ensure the performance of the aggregated model in one-shot FL, il is crucial to maximize population divergence within the constraints population size (Wu et al. 2021 Wu & Liu 2021). Unfortunately, the existing diversity computation methods proposed so far typically require accessing the local datasets of the clients, which is unsuitable for FL scenarios. Therefore, the designing of effective and computation-efficiet ensemble selection strategy as well as computation methods for model diversity for one-shot FL are challenging problems.
In this work, we focus on the ensemble selection problem under the one-shot FL setting and aim to find a near-optimal subset of models to the ensemble for the central model server in a data-free manner. As the basic workflow shows in Figure [ each client trains its local model with their own private dataset and then uploads the well-trained models the central model server. To perform the ensemble learning, the server will select the best ensemble team from all models. Note that during the whole process, the server has no access to the local datascts of clients, which actually follows the data-free manner.
We propose novel framework, Data-free Diversily-based Ensemble Selection (DeDES), to select the best ensemble team from the model set trained in the one-shot FL fashion, which is a clustering based ensemble selection mothod to ensure the diversity without access to any public or clicnt-side datasct. Extensive experiments show that DeDES achieves the best performance for different settings, and robust and effective. Also, can adapt various datasets, data partitions (including the non-iid case), and model structures (both homogeneous and heterogeneous models). To the best of our knowledge, this is the first effort to systematically deal with the ensemble selection issues for one-shot FL.
Our main contributions can be summarized as follows,
We provide the formulation of the ensemble selection problem for one-shot Federated Learning to facilitate clearer comprehension of the topic.
We propose a novel one-shot Federated Learning framework DeDES equipped with the data-free ensem- ble selection strategy, which can evaluate model diversity and conduct ensemble selection in data-free manner for privacy-preserving consideration. Our method is model-agnostic and can be applied to both homogeneous and heterogeneous model groups.
We conduct comprchensive experiments to verify the effectiveness and efficiency of our DeDES, which can also adapt to both homogeneous and heterogeneous settings. In comparison to the stale-of-the-art data- free one-shot federated learning framework, our proposed model ensemble methodology demonstrates a substantial euhancement in performance efficacy.
2
Related Work
2.1
Federated Learning and One-Shot Federated Learning
Various federated learning systems (Bonawitz et al. 2019 Duan al., 2020) have been proposed to assist multiple parties in cooperatively working with others without disclosing their data. E.g., Wu et al. (Wul
2

---

Published in Transactions on Machine Learning Research (11/2023)
et al.
2023) addressed the client drift problem on personalized FL by using a GMM to fit the joint data distrin retribu bons across FT. devices. In particular, one-shot. federated learning (Guha et al.) 2018 Su et al. Feng et al.] (023), including FedKT LLi et al. 2021], Fusion Learning (Kasturi et al.) 2020 and so on, aims to facilitate collaboration through a single round of communication between the server and clients. Su et al. (Su et al.
2023
2023) proposed MA-Echo that employs layer-wise projection matrices to maintain the original loss of local models after aggregation of parameters, which requires all model structures to be the same; FedDISC (Yang al.) 2023) explored one-shot semi-supervised federated learning with a pre-trained diffusion model. Furthermore, with the popularity of pre-trained models, there are emerging interests in collaborative model-centric machine learning (Bommasani et al. 2021) as a platform for users to exchange their trained models with others, and to harness collective intelligence for the targeted machine learning task by combining models. Diao et al. (Dio et al. (022) and Joshi et al. (Jhunjlunwala et al. 2023) separately explored one-shot federated learning through the open-set problem and Fisher information. However, neither addressed the model selection problem that is critical for largo-scale FL systems, which makes our approach distinct and orthogonal to their methods.
2.2 Ensemble Learning and Ensemble Selection
Ensemble learning, which seeks to combine multiple weak base models into a strong model, has been a popular research topic for decades that can be applied to federated learning scenarios. Classical ensemble learning approaches include Voting (Zhou 2021), Bagging (Breiman 1996 Sagi & Rokach 2018 Boosting (Schapire,
1990 2013 and Stacking (Wolpert 1992 Wang et al. 2019).
As an important step in ensemble learning, ensemble selection needs to select an ensemble team from the whole model set for each test sample; there exist three types of ensemble selection approaches, i.e., search- based (Caruana et al., 2004), rank-based (Ma et al. 2015), and cluster-based (Maskouni & Zhou 2018). Notably, the cluster- method stands out for its superior efficacy, which primarily stemns from its reliance on model diversity. Luis A al. (Ortega et al.] (2022) give a thcoretical analysis that the expected loss of an ensemble decreases as the diversily measure among model ensembles increnses. Classic model diversily computation methods include Binary Disagreement (Kuncheva & Whitaker 2003), Cohen's Kappa. (McHugh, 2012), Q Statistics (Zhang & 2014), Generalized Diversity (Partridge & Krzanowski 1997). and Kohavi-
Wilpert Variance (Kuncheva & Whitaker 2003). All of them require accessing the local dataset and cannot
be directly used for federated learning.
2.3 Knowledge Distillation in Federated Learning
Another field of interest for federated learning is knowledge distillation (Lin et al. 2020 Gong et al. 2021 FedGen (Zhu et al. 2021) is data-free method for heterogencous federated learning that necessitates multiple rounds of communication between the server and clients. DENSE [hang el al. 2022) is a data-free one-shot method to train a global model, and it involves grouping all clients as an ensemble, which may lead to decreasod performance and efficiency. Thus, our ensemble selection approach could potentially assist in improving their performance and training elficiency. FedCAVE-KD (Heinbaugh et al., 2023) is an instance of knowledge distillation generation. In this method, locally trained CVAFs and local label distributions are uploaded to a server for data-free knowledge distillation, which ensures privacy and simultaneously improves the generalization of the global model. However, they only send the decoder instead of CNN to the server, so their models cannot be directly used to conduct ensemble, not to say ensemble selection. To conclude, none of the existing methods tackle the ensemble selection problem for one-shot federated learning.
2.4 Client Selection in Federated Learning
Several research works focus on the issue of client selection in federated learning [aaing((Aaar (AbdulRahman et al. 2020), Nishio et al. (Nishio & Yonetani 2019) enables the server to combine as many client updates as feasible within a specified time-frame; Cho et al. (Cho et al., (022) presented a coaputation-efficeen client selection framework to alleviate the bias of model aggregation; Huang et al. (Huang et al., 2020) proposod an sefeiency-boostiin client selection scheme to guarantee the fairness of the training process. However, those methods independent of our ensemble selection topic and cannot be applied to our problem.
3

---

Published in Transactions on Machine Learning Research (11/2023)
3 Problem Definition
We assume that there are different parties (aka FL clients) and a central model server in our one-shot federated learning system, where the parties want to collaborate together on given machine learning task, e.g., classification or regression, and the server builds the model ensemble team for one-shot federated learning. Let M = {MI....,AM own private dataset D; = {(1), 3k) 3k)] with size ne, where each data vector is sampled from an unknown distribution D. Note that M will be uploaded by each party to the central model server. Hence, the ensemble selection problem can be formulated as:
be the model set, in which M, is well-trained on the i-th client over its
Problem Given the model set M and the consland K & m, fond the optimal subset M C CM such that
Mr
mig M
B( {(fMx (x).
(1)
where fMx (.) is the prediction function based on MK, , l is the loss function, e.g., MSE loss function for regression tasks and Gross-Entrye loss function for classification lasks.
Under the ensemble learning sctting, fMx is the aggregation function to combine the prediction M, E MM for the final prediction f [Mg (ar); it can be weighted average for regression, or weighted voting-basse (e.g.- majority or plurality voting) for classification. Under the model fusion setting, fMX is the prediction of the fused model based all models in Mи-
In the following sections, we mainly focus on the classification task and adopt the weighted voting strategy based on the size of local clients' datasets for ensemble learning. For a C-class classification (i.e., the label set is (1....,C)) task, with I(-) as the indicator function, the prediction 9 of the input x is given by
i = arg max
T2J
[() =),
(2)
km Tbk
4 Proposed Framework: DeDES
We present the proposed ensemble selection framework, DeDES, to solve the Problem D without accessing any private dataset from local clients. Algorithm | summarizes the structure of DeDES, which can be adapted to heterogeneous models.
Considering the performance and efficiency of Mik it is necessary to choose small K while keeping the diversity and high-quality among selected elements/mon models. DeDES achieves such goal via the following key components: model filtering, model representation, model clustering, and representative model selection.
4.1 Key Components in DeDES
Model filtering: Coming from multiple parties in FL, the test performance of local models can vary signif- icantly and out-of-control to the central server. Possible inferior models may result from different reasons, including low-quality training data (e.g., being unreliable, contaminated, or noisy), inefficient training (e.g.. trained with improper hyperparameters), etc. Therefore, is necessary filter out such outlier models to eliminate the effect of the noise and help to select. high-quality models efficiently. In Alg, [ we use the OutlierFilter to obtain the outlier models O based on the model scores S provided from each party, Note that S could be the local validation accuracy prediction confidence. OutlierFilter could be any scsre-based unsupervised outlier detection methods ŞZhao et al. 2019 In Alg. 2 we implement a variation of the box-plot method to identify the outliers when their scores are lower than the threshold 8.
Model representation: Given the model structure and parameters, generating effective and suitable representations for the models is crucial to measure their properties, such as similarity and diversity. In Alg. 1 obtain the representations via. the function Representation in Line 4. Intuitively, we
4

---

Published in Transactions on Machine Learning Research (11/2023)
Algorithm 1: DeDES framework
Input:
model set M, training-set sizes N {n+)= truncated threshold pair (Plow Phigh); interval scale 8, model scores {st i= target model-set size K, and representative selection
threshold т.
Output: Optimal model subset (ensemble team) MM
Mk- +0
1. Model filtering: select high-quality condidates by filtering out outliers.
O < OutlierFilee M S, (Piow:Piihh)
D O: oullier models set
N² = {n4 M, & O, VM, E M]; = {sr M & O,YM E M]; MM=MiO;
2. Model representation: get model's feature representation.
R. =Representation(MM)
3. Model clustering: get K-size model clusters for diversily selection.
CM Clustering(R,w
4. Representative model selection: choose the 'best' model in each cluster.
for C E CMr do
Nc (mi VM, ECMM"
mmax= t max(Ne); nined + median(Ne)) b T: user predefined threshold, e.g., 7 = 0.3. if n / "Tmax <T then
10
L mag, M, cCOM]
11
else
L arg max, {8, M, ECNM)
12
13
Mi + Mx ((x)
b Cgz the k-th element of the cluster C
14 return Mix
Algorithm 2: OutlierFilter algorithm for the model filtering
model set M, truncated threshold pair (Ptow, Phigh), interval scale model scores S = {8x ) Output: Outlier model set S SSr
Input:
a non-decreasing order
8 + min(S) + s (Smxpnigh Sm Smxxpoow)
O < 0
for i = to m do
if 8; 8 8 then
L -
return
represent a model by all or part of its model parameters. E.g,, we can choose the parameters of the last layer of the model, which contain individualized and sufficient information about the model behavior (especially for the classifier) and data manifold/span
for local training. Meanwhile, to distill compact information and suppress noise for the representation, espocially for big models such as Resnet-101 (Wu et al.] 2019). dimension reduetion (DR) is also applied the representatious; many unsupervised approaches can be adopted here, including PCA, Kernel-PCA, etc. The target dimension for DR is set to be [M'| by default.
To guarantee the diversity in M2 as we mentioned before, we can utilize the clustcring method to identily the similarity of different models, where models with similar properties are grouped inlo the same cluster and different clusters are as different as possible. We can use the traditional clustering approach here, such as KMeans, Hicrarchical Clustering, and Spectral Clustering, etc., and set the target number of clusters as K. This process is denoted by Clustering in Alg. I which leads to Cue the resultant clusters.
Model clustering:

---

Published in Transactions on Machine Learning Research (11/2023)
Model Repecsontation Selection
Model horing
Parometer P oproues
Ornenniion Reduction
Meto Faconder Matte
Taper
Final Ersemole Team
Representative Modal Selection
Modd Custering
Figure 2: The flow chart of the execution processes of DeDES framework. All model architectures identical and the last layer is used to represent the model. In this flowchart, PCA is used the dimension reduction method. The circles of the same style with the same color and texture represent models with actual high similarity. The model parameter matrix in the figure is the model representation Rye mentioned in Alg.
Representatioe model selection: To choose exactly K models with high performance, we elaborately select the representative element in each cluster while keeping the diversity. Among the models in each cluster, we can intuitively select the model with either the highest model score 8; E S (provided by the individual party) or the largest training dataset (leading to a better-traide trained model). As Line 6-13 in Alg, [ shows, we design a hcuristic selection strategy to make full use of these two ways, which can choose better one than any of the fixed ways the experiment results proves. That is, if the amount of training data for the models inside the cluster is balanced, the model with the highest score is chosen, otherwise, the with the largest training data is chosen.
Figure 2 presents flow chart that exemplifies the implementation of DeDES using the parameters of the last layer as the model representation and PCA as the method of dimension reduction when all model structures are identical. The clustering method dependent on the employed data partition strategies.
Inference: After obtaining the optimal M& with Algorithm I we will conduct ensemble learning with the weighted voting Eq. Note that in the whole process of DeDES, succossfully select the ensemble team Mi based on the model diversily without accessing any of local private data of these parties.
4.2 Adaptation to Heterogeneous models
When the structures of the client models are different, i.e., heterogeneous models, our method transforms them into unified model parameter matrix RM
Assuning that there are types of model structures and each contains any number of models, the model set M' for all those models is,
M' (M,, My,..., Mm group by
Ü M's with M' = (M.1..... .M,,,hh
i=1

---

Published in Transactions on Machine Learning Research (11/2023)
where my denotes the number of models in M' that share the i-th model structure; and the total number of models in M' is m' = m4.
We extract part of the model parameters and flatten them to be one-dimensional vector as the represen- tation of the model. For the model M, MM, let 0, denote the flattened selected parameters (e.g., the parameters of the last layer) of Mj Let V. = ( 0l.m;
} be the vector set for the model subset M';
Thus, the vector set V for all models M' will be V V- V. (0), 02, 0a,..,,,000. Om.).
==1
Since the model structures vary for different M2, the sizes of the vectors V are also different, which makes hard to directly apply the clustering methods. To solve such an issue, we reduce the dimension of all vectors a unified dimension d. Given DR algorithm, we transform the size of all vectors 0 of V into d minøev [0] by applying DR to each sub-vector set Vi, separately. The model parameter matric R. is obtained by combining all these matrices together,
RM [DR(Vi, DR(V,,)) D)(".
(3)
RM E TRm'x xd with each sub-matrix DR(V, d) E Rmixdd We then perform further clustering R.M since all heterogeneous models are transformed to the same dimensions.
4.3 Complexity analysis and Resource Requirements
Lemma 1 Given m client models, K larget models, and d the size of the selected parometers of single model, the overall time complenity of DeDES is O(m log m + md m) O(custering)) where (clustering)) is the time complerity of the applied model clustering method.
Proof The time complexity of each component in DeDES given as follows:
For the model filtering, it takes (m log m) time due to sorting of model scores, which the most time- consuming step; the model representation takes O(md) time, and it linearly depends the size of the selected paromelers in model. The complerily of the model clustering depends on the specific algorithm being employed, that is, it will be O(mKd) if KMeans is used, and be O(m²) if adopting spectral clustering. The representative model selection step takes O(m) time for it linear filtering.
Hence, when employing the KMaans in DeDES, the lotal time complerity O(m log m + md + mkd + m).
Communication Cost. Our approach adheres to the one-shot federated learning scheme, which inherently requires every client to communicate with the server only once - when they transmit their respective models.
5 Experiments
5.1
Experiment Setup
To simulate the real scenarios in FL Li et al. 2022 we designed four types of dataset-partitee strategies to evaluate DeDES, which lead to different local data distribution to train diverse client models M,.
Homogencous (homo): the amount of samples and the data distribution keep the same for all partics.
IID but different quantity (iid-dq): the training data of each party follows the same distribution, but the amount of data different.
Skewed data distribution (nontid-lds): the training data of each party follows different distributions, especially for the label distribution.
Non-iid with & (< C) classes (noniid-lk): the training data of each party only contains of C' classes, which is extreme non-iid setting.

---

Published in Transactions on Machine Learning Research (11/2023)
(a) homo
(b) lid-dq
(c) noniid-lds
(d) noniid-14
Figure 3: Example distribution of four data partition strategies for CIFAR-10 10 with party number m 5.
Each color denotes class, and the height of the bar represents the number of samples of that class.
We used 7 image datasets and 5 types of neural network models in experiments, details be found in the Appendix. We partition all datasets into different groups based on the above strategies and train the model for each client. Figure B shows an example of the data distribution under the different partition strategies for CIFAR-10 with parties.
The detailed runtime setups and configuration DeDES are elaborated in the Appendix, including the learning rate, model representation stralegy, clustering method for different data partitions, etc.
5.2 Baselines
For the model ensemble learning under our problem setting, we follow the designs in the first one-shot FL framework (Guha et al. 2018) and summarize the well-known selection approaches as follows:
Cruss-validation selection (CV): select. Mi using local validation accuracy; Data selection (DS): M = {M, I E top((n1.".. mm)) K)}, i.e., the models trained with the top K size training dataset, which are selected by top;
Random selection (RS): MM consists of K models random selected from M.
Besides, we construct well-known baselines in terms of model fusion, for comparison with the traditional FL methods. They derive one single model M* leading to the highest efficiency for inference as follows
One-Shot Federated averaging (FedAvg): M Σ
2
One-Shol Mean averaging (MeanAug): M E M,.
A Also, we include the following results as the ground-truths for comparison,
All selection (AS): select M the target model set by ignoring K.
Label distribution selection (LDS): utilizing the label distribution instead of model representation
the input of our method]
D; to train a model Moracle, which is unrcalistic for Oracle: using the aggregated dataset D = Um i=1
the real FL scenario.
5.3 Performance Analysis
The effectiveness of ensemble learning. Figure shows the comparison result for types of data partition settings, where TOP and TOP 2 mean single model that gets the best and second test accuracy on the whole test dataset Destt = U Diest, where Deest is the test set for i-th party/clioee we can see, the performance of the ensemble methods such as AS and DeDES are always better than single modols, which validates the offectiveness of ensemble learning under the onc-shot foderated learning setting,
As
Comparison of DeDES with other methods. For m models, the number of possible ensemble teams is 2m which increases exponentially with m. Since testing all teams get the optimal one is impractical unless is very small, compare DeDES with other cxisting methods to demonstrate its superiority with the
Note that the label distribution unavailable the federated learning scenarios.

---

Published in Transactions on Machine Learning Research (11/2023)
100
100
200
95
95-
60-
85
80
20
DeDes
TOF1
JeDes
(a) homo
(b) iid-dq
(c) noniid-Ids
(d) moniid-4
Figure 4: Ensemble learning (weighted voting strategy) performance comparison on EMNIST Digits dataset
for m = 200, K = 80. The x-axis shows different ensemble selection method names, and the y-axis is the test accuracy (%) on the whole test dataset D = = Diest, where Diest is the test set for i-th client.
Table 1: Test accuracy comparison for different datasets on various data partitions and model structures. The best and next-best are bolded and underlined, respectively. If our DeDES method is betler than the ground-truth methods such as AS and LD, the value of the corresponding method are marked in skyblue.
Dataset
Partition
K
DeDES
CV
DS
RS
Fed Avg
MranAvg
AS
L.D
Oracle
homo
400
150
98.03
98.10
98.08
98.07
10.28
10.26
98.10
98.10 10
99.74
EMNIST Digits
liddd
400
150
99.27
98.93
98.88
98.72
10.51
10.48
98.75
99.27
99.71
(100% VGG-5 Spinal FC)
noniid-Ids
400
150
97.67
95.47
91.70
96.67
10.01
9.89
96.99
92.86
99.72
noaild-3
150
98.21
97.87
63.59
94.35
10.11
10.09
97.96
98.13
99.61
homo
100
50
85.19
85,10
84.96
84.96
2.10
2.11
84.94
84.83
89.70
EMNIST Balanced (100% VGG-5 Spinal FC)
idddd
100
50
87.34
87.31
87.35
86.90
2.04
2.04
87.28
87.35
89.25
nonidd-Ids
100
50
83.43
78.65
79.44
78.88
2.19
2.16
82.72
77.28
89.48
noniid-118
100
50
85.43
81.22
81.02
80.32
2.09
2.08
82.99
82.87
89.52
homo
200
80
38.90
32.69
23.08
26.30
17.12
18.64
26.
32.34
92.72
SVHN (100% Rosnet-18)
lid-dq
200
80
65.65
83
66.82
64.50
51.49
17.20
18.65
58.56
65.25
93. 14
nouidd-ids
200
80
32.56
28.94
28.59
30.65
10.15
10.24
30.97
32.89
92. 56
nonid-13
200
80
31.45
23.92
28.62
27.47
10.21
10.71
24.14
36.66
93.22
noniid-Ids Ids
3597
30
54.46
44.41
51.67
46.83
21.05
21.06
55.69
50.45
89.08
PEMNIST
nouiid-lls
3597
100
56.36
54.61
54.04
54.07
21.05
21.06
55.60
54.21
89.08
(100% R8s
nouidd-Iss
3597
300
58.32
55. 94
54.97
54.33
21.05
21.06
55.60
56.14
89.08
homo
200
100
32.08
32.07
30.78
30.30
10.18
9.69
32.09
32.08
88.68
CIFARIO (100% Resmet-50)
ldd-dq
200
100
36.97
38. 84
39.03
36.66
10.04
10.03
38.49
38.81
88.10
noniid-kls
200
100
29.71
26.02
29.10
26.67
9.89
9.88
29.23
28.94
87.31
nomiid-14 14
200
100
34.40
32.24
30.00
30.45
10.02
9.87
33.50
34.15
89.67
homo
20
12
20.84
20.58
20.65
20.48
0.99
0.99
22.84
20.85
59.81
CIFARI00 (100% Resnet-50)
lid-dq
20
12
47.38
47.38
47.38
25.10
1.00
0.94
47.37
47.38
00.35
noniid-Idk
20
12
16.31
15.97
16.15
15.78
0.97
18.71
15.32
60.38
noniid-145
12
21.29
20.56
20.26
19.97
0.91
19.61
61.74
help of selected K. Table ! shows the test performance of selective configurations for different datasets and partition configurations. Here, choose the VGG-5 Spinal FC model for the EMNIST dataset considering its state-of-the-art performance (except. for the extremely large transformer models), the ResNet-50 model for the CIFAR10 and CIFAR100 datasets. Additionally, to demonstrate the effectiveness of our method over heterogeneous models, we simultaneously utilize two types of structures, VGG-5 Spinal FC and Resnet-50, to train the local models for the EMNIST digits and letters datasets.
As we can see, the performance of the Oracle method is always the best, since the centralized setting that can utilize all parties' data for training the model. Meanwhile, the performance of the FedAvy or MeanAng is significantly worse (near random guess), with only test accuracy around 2% for the balanced EMNIST datasets, which validates that directly averaging averaging/fusing well-trained models is not suitable for the one-shot federated learning setting.
As shown in Table for the homo partition, the accuracy of different methods are quite similar, which means that they can casily deal with such simple partition case which leads to an iid sctting (the same information for all parties) and each client model has similar performance. For the tid- partition, the Data Selection
(DS) is the best method for most of the datasets; the single TOP 1/2 models as in Fig. 4 (b) have the
9

---

Published in Transactions on Machine Learning Research (11/2023)
Test Accuracy (%) comparison with state- of-art data-free one-shot foderated learning method DENSE for the EMNIST Digits Dataset, VGG-5 (Spinal FC) structure when m=400 (for DeDES, K = 60). We compare the test accuracy (%) of the methods
Table 2:
Table 3: Complete inspection on ensemble
teams for EMNIST Balanced dataset with m = 10, K 6, noniid-lss partition. Here, we illus- trate the rank of the performance of ensemble teams generated by different methods under all 2 = 1024 teams.
for all data partitions.
Method
DeDES
DENSE
AS
FedAvg
Method
Rank
Accuracy (%)
DeDES
34/1024
84.34
homo 98.03
92.91
98.10
10.28
AS
114/1024
83.39
iid-dq
99.27
10.02
98.75
10.51
CV
241/
82.29
noniid-Ids
97.67
61.46
96.99
10.01
DS
348/1024
80.86
noniid-13
98.21
55.59
97.96
10.11
RS
669/1024
76.54
largest dataset with samples of every class in the label set, which leads to their strong generalization ability under this partition. Therefore, we conclude that the more data we have, the better performance we will get for the iid setting, since the best way is to select K models with top or top K largest data sets.
When the data partition is non-iid (nontdd-Ids and noniid-lk), we can see that DeDES achieves the best performance for most of the datasets, with different m and K, which validates the effectiveness of our method. DeDES is not as good as the AS method for the CIFAR-100 dataset, this is because CIFAR-100 has 100 labels, thus the data amount of each individual label for local parties is too tiny to train generalized model. Under this condition, the AS method will get more information than other methods and therefore have better performance. But for other datasets, especially for the EMNIST where all local models are more generalized, DeDES will get better performance than others, even the method of all selection.
In summary, our method outperforms other baseline methods for ensemble learning, regardless of the model structures and datasets employed. For the results of heterogeneous models, please refer to the supplementary.
In addition, compare DeDES with the state-of-the-ee data-free one-shot federated learning method, DENSE (Zhang et al. 2022). DENSE leverages a generator coupled with the knowledge distillation (KD) technique to train a global model in a data-free way. As shown in Table 2 the performance of DeDES in all data partition scenarios is superior lo model averaging and DENSE, demonstrating the effectiveness of our method. Furthermore, it can be discerned that while DENSE delivers appreciable results under the homogencous (homo) data distribution, its performance deteriorates the point of non-convergence in the vid-dq scenario, implying that the distribution of training data quantity has a significant impact on DENSE. When dealing with non-iid data partition, the global model's test accuracy attained by DENSE through data generation and knowledge distillation only manages to hit 50%-60% of that secured by ausemble-baser methods. This suggests that the data created by the generator is influenced by non-iid data distribution, thereby subsequently impacting the performance of knowledge distillation.
Complete inspection on ensemble teams. Given m = 10, there are 2 = 1024 potential ensemble teams available for selection. Table enumerated the accuracy of all 1024 teams and the ranking of selected teams generated by different selection methods. We can see that the ensemble team selected by DeDES is ranked higher than other bascline methods, which validates the efficacy of our method.
5.4 Impact on Efficiency
As Table I shows, DeDES is just little inferior to the ground-truth All Selection (AS) method in some cases. However, AS is considerably less efficient than our method, and the performance gap between the two methods is small. In other cases, our method even outperforms AS. Therefore, it validates that our approach can reduce the inference time significantly for ensemble learning without great sacrifice for the performance.
It is easy to know that when all models are homogencous, the inference time for ensemble learning (i.c., weighted voting) increases linearly with K, i.c., the inference time T for one test sample is O(K c), where is a constant inference time for one sample by one model. The inference time for the ensemble team
10

---

Published in Transactions on Machine Learning Research (11/2023)
Ensemble Test Accuracy (%)
0.18
80
Cohen's Kappa Value
70
0.16
60
0.14
nonis-td
50
0.12
40
0.10
0.08
20
ronid is
0.06
0
50
200 250 300 350 400 K, linear with inference time
100
150
AS
RS Ensemble Selection Method
CV
DS
DeDes
Figure 5: The relationship of K and Ensemble Test Accuracy of DeDES for the EMNIST Lellers Datasel, VGG-5 (Spinal FC) structure when m=400. Here we show the accuracy of the AS method when K = 400.
Figure 6: The Cohen's Kappa value of ensemble leamns selected by different ensemble selection meth- ods for the CIFAR10 Dataset, Resnet-50 structure when m=50, K = 30.
generated by the AS method will be Figure shows the relationship between K and ensemble test accuracy of DeDES. Note that when K = m 400, we will show the accuracy of the AS method instead of DeDES since we typically will not use a selection method such as DeDES to select all K = m = 400 models.
The experimental results shown in Figure indicate that when K reaches a certain value, the test accuracy will not increase significantly, sometimes even docrcase. Therefore, with suitable K (usually around 50% of m), we can substantially reduce our ensemble inference time while achieving good ensemble performance.
Physical running time comparison and analysis. To further exemplify the high efficiency of our method, we show the actual physical running time of various selection algorithms in the Appendix, as well as provide an analysis of why and when will our selection method be better than the AS method. To conclude, it is very efficient to use DeDES instead the AS method when the number of test samples exceeds a relatively small threshold.
Validation for large-scale federated learning system. To verify the effectiveness and efficiency of our method for large-scale FL system, tested method the FEMNIST dataset which is benchmark dataset for federated settings that has in total 3597 clients and is non-i.i.d Table [ reveals that when K = 100 << our method outperforms the 'All Selection' method which incorporates all 3597 models into the ensemble team that will yield significant computational load and is highly tine-consuming. However, by only choosing 2.78 % (100/3597) of the models by DeDES, we can both get better ensemble performance and save approximately 97.22 (100 2.78 %) of the ensemble's execution time. This case substantiates that a model selection method is indispensable for a large-scale federated learning setting.
5.5 Clustering/ SiiberinngDDversity Diversity validation
To validate our clustering results, we compare the Cohen's Kappa (CK) (McHugh 2012
value of the ensemble
tcams selected by different methods to mcasure their diversitics. shown Figure G compared to other baseline methods, the diversity of the ensemble team generated by DeDES is higher (lower CK), which also means the agreement of the whole team's models is low. Since only select model from every cluster, this finding also indicates that method can really cluster similar models together, which validates that DeDES generates an ensemble team with high diversity. Notably, the AS method also shows high diversity compared to DeDES and achieves high ensemble test accuracy. This finding supports the conclusion that greater diversity among models enhances the ensemble's performance.
11

---

Published in Transactions on Machine Learning Research (11/2023)
Table 4: The impact of model filtering method for the EMNIST Balanced Dataset, VGG-5 (Spinal FC) structure when m=200, K 100 on the noniddlds partition, where DeDes NoMP refers to our DeDES method that does not incorporate model filtering.
Method
RS
CV
DS
AS
DeDES NoMF
DeDES 81.22
Ensemble Test Accuracy (%) 74.21
74.36
78.99
79.61
80.83
Table 5: The impact of model clustering method for the CIFAR-10 Datasct, Resnct-50 structure when m=100, = 60. We compare the test accuracy (%) of the methods for all data partitions.
Method
DeDES_ KMeans
DeDES Hierarchical DeDES Spectral
AS
CV
DS
R$
homo 35.39
36.61
28.66
35.24
36.37
35.11
34.75
lid-dq 42.86
43.82
41.51
43.79
43.85
44.40
42.13
noniid-Ids
35.76
34.36
32.86
35.04
32.91
34.12
32.72
noniid-14
39.55
37.59
36.84
38.19
36.94
34.35
22.86
5.6
Ablation Studies
We examine each component's impact on model filtering and clustering methods in this section. Please see the Appendix for additional studies such as model representations, dimension reduction, etc.
5.6.1 The Significance of Model Filtering
As illustrated in Table ! we compare the accuracy of DeDES with and without the model filtering algorithm, the results indicate that the use of the model filtering algorithm can effectively eliminate noisy and under- fitting models, leading an improvement in the final ensemble performance. In contrast to the AS method, model filtering can prevent unsuitable models from participating in the ensemble learning process, leading to improved efficiency and efficacy of ensemble learning. Therefore, model filtering is essential component in conducting ensemble selection for onc-shot federated learning.
5.6.2 Comparative analysis of clustering methods
Table 5 presents the performance comparison of DeDES by implementing varions clustering algorithms to form ensemble teams. As we can see, for iid data, hierarchical clustering shows superior performance to other methods, although the discrepancy in accuracy among the different methods remains negligible in an iid setting, we mentioned in section 5.3 Conversely, under non-iid conditions, KMeans is anticipated exhibit the highest performance. This expectation especially true under extreme non-iid circumstances where models trained on similar data are likely to cluster spherically, thus enhancing the effectiveness of the KMeans clustering method over other clustering algorithms.
6 Conclusion
In this paper, we propose novel data-free diversity-bas based framework DeDES to address the ensemble selection problem for models under one-shot federated learning. Experimental results show that our method can achieve both better performance and higher efficiency for various model structures and datasets, especially for the non-iid data partitions. To our knowledge, this is the first paper to systematically address the ensemble selection problem under one-shot federated learning setting, which is an essential application for model-centric collaborative machine learning.
Acknowledgment
This rssearch/project project is supported by the National Research Foundation Singapore and DSO National Lab- oratories under the AT Singapore Programme (ATSG Award No: AISCC--PP2200-001))
12

---

Published in Transactions on Machine Learning Research (11/2023)
References
Sawsan AbdulRahman, Hanine Tout, Azzam Mourad, and Chamseddine Talhi. FedMCCS: multicriteria client selection model for optimal iot federated learning, IEEE Internet of Things Journal (IoT-J), 8(6); 4723-4735, 2020.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bossclut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.. 07255, 2021.
Keith Bonawitz, Brendan McMahan, et al. Towards lederated learning at scale: System design. Proceedings of Machine Learning and Systems, 1:374-388, 2019.
Leo Breiman. Bagging predictors. Machine learning, 24:123 140, 1996. doi: 10.1007/BF00055653
Rich Caruana et al. Ensemble selection from libraries of models. In JCML, 2004.
Yae Jee Cho, Jianyu Wang, and Gauri Joshi. Towards understanding biased client selection in federated learning. In International Conference Artificial Intelligence and Statistics, pp. 10351-10375. PMLR, 2022.
Yiqun Diao, Qinbin Li, and Bingsheng He. Towards addressing label skews in onc-shot federated learning. In The Eleventh International Conference on Learning Representations, 2022.
Moming Duan, Liang Liang, et al. Self-balancing federated learning with global imbalanced data in mobile systems. IEEE Transactions on Parallel and Distributed Systems, 32(1):59- 71, 2020.
Chun-Mei Feng, Bangiun Li, Xinxing Xu, Yong Liu, Huazhu Fu, and Wangmeng Zuo. Learning federated visual prompt in mull space for mri reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pultern Recognition (CVPR), 2023.
Jonas Geiping, Hartmut Bauermeister, Hannah Droge, and Michael Moeller. Inverting gradients- easy is it to break privacy in federated learning? NewrIPS, 33:16937 16947, 2020.
Xuan Gong, Abhishek Sharma, Srikrishna Karanam, Ziyan Wu, Terrence Chen, David Doermann, and Arun Innanje. Ensemble attention distillation for privacy-preserving federated learning. In Proceedings of the IEEE/CVE International Conference on Computer Vision (ICCV), pp. 15076 15086, 2021.
Neel Guha, Amect Talwalkar, and Virginia Smith. Onc-shot federated learning. In NearIPS 2018 Workshop on Machine Learning on the Phone and other Consumer Devices, 2018.
Clare Elizabeth Heinbaugh, Emilio Luz-Ricca, and Huajie Shao. Data-free one-shot federated learning under very high statistical heterogeneity, In The Eleventh International Conference on Learning Representations, 2023.
Tiansheng Huang, Weiwei Lin, Wentai Wu, Ligang Hc, Keqin Li, and Albert Y Zomaya. An efficiency- boosting client selection scheme for federated learning with fairness guarantee. IEEE Transactions Parallel and Distributed Systems, 32(7):1552-1564 2020.
Divyansh Jhunjhunwala, Shiqiang Wang, and Gauri Joshi. Towards thcorctical and practical understanding of one-shot. federated learning with fisher information. In Federated Learning and Analylics in Pructice: Algorithms, Systems, Applications, and Opportunities, 2023.
Anirudh Kasturi et al. Fusion learning: A one shot federated learning. In International Conference on Computational Science, pp. 424-436. Springer, 2020.
Ludmila Kuncheva and Christopher Whitaker. Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Machine learning, (1):1181122) -07, 2003.
13

---

Published in Transactions on Machine Learning Research (11/2023)
Thanh Chi Lam, Nghia Hoang, Bryan Kian Hsiang Low, and Patrick Jaillet. Model fusion for personalized learning. In Proceedings of the 38th International Conference on Machine Learaing (TCML.), pp. 5948- 5958. PMLR, 2021.
Qinbin Li, Bingsheng He, and Dawn Song. Practical one-shot federated learning for cross-silo setting, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, (IJCAI), pp. 1484- 1490. International Joint Conferences Artificial Intelligence Organization, 8 2021. doi: 10.24963/ijen /021/205. URL https://doi.org// 2et.:/.44963jacaa/ 2021/205
Qinbin Li, Yiqun Diao, Quan Chen, and Bingsheng He. Federated learning on non-iid data silos: An experimental study. In ICDE, pp. 965 978. IEEE, 2022.
Tian Li, Virginia Smith, et al. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine, 37(3):50-600 600006600000 2020.
Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pp. 2351 2363, 2020.
Zhongchen Ma, Qun Dai, and Ningzhong Liu. Several novel evaluation measures for rank-based ensemble pruning with applications to time series prediction. Expert systems with applications, 42((1:280-22) 2015.
Mojtaba Amiri Maskouni and Xiaofang Zhou. Auto-ces: automatic pruming method through clustering ensemble selection. In Australasian Database Conference, pp. 275-287. Springer, 2018.
Mary L. McHugh. Interrater reliability: the kappa statistic. Biochemia medica, 22(3))276-282) 2012.
Takayuki Nishio and Ryo Yonetani. Client selection for federated learning with heterogencous resources mobile edge. In ICC 2019-2019 IEEE international conference on communications (ICC), pp. 7. IEEE, 2019.
Luis A Ortega, Rafael Cabañas, and Andres Masegosa. Diversity and generalization in neural network ensembles. In International Conference on Artificial Intelligence and Statistics, pp. 11720-11743. PMLR, 2022.
Derek Partridge and Wojtek Krzanowski. Software diversity: practical statistics for its measurement and exploitation. Informalion and sofbware technology, 39(10):707-717 1997.
Omer Sagi and Lior Rokach. Ensemble learning: A survey. Wiley Interdisciplinary Reviews: Dala Mining and Knowledge Discovery, 8(4):e1249 2018
Robert E Schapire. The strength of weak learnability, Machine learning, 5:197 227, 1990. doi: 10.1007/ BF00116037.
Robert E Schapire. Explaining adaboost. In Empirical inference, pp. 37-52. Springer, 2013.
Shangchao Su, Bin Li, and Xiangyang Xue. One-shot federated learning without server-side training. Neural Networks, 164:203-215 2023. doi: 10.1016/j.neumet.2000 20..2002223000
Yuyan Wang et al. Stacking-based ensemble learning of decision trees for interpretable prostate cancer detection. Applied Soft Computing, 77:188 204, 2019.
David И Wolpert. Stacked generalization. Newral Networks, 5(2):241 259, 1992.
Yanzhao Wu and Ling Liu. Boosting deep ensemble performance with hierarchical pruning. In 1CDM, pp. 1433-1438. IEEE, 2021.
Yanzhao Wu, Ling Liu, et al. Boosting ensemble accuracy by revisiting ensemble diversity metrics. In CVPR, pp. (6469-16477 2021.
14

---

Published in Transactions on Machine Learning Research (11/2023)
Yue Wu, Shuaicheng Zhang, Wenchao Yu, Yanchi Liu, Quanquan Gu, Dawei Zhou, Haifeng Chen, and Wei Cheng. Personalized federated learning under mixture of distributions. In Proceedings of the 40th International Conference on Machine Learning (ICML), pp. 37860 37879, 2023.
Zifeng Wu, Chunhua Shen, and Anton Van Den Hengel. Wider deeper: Revisiting the resnet model for visual recognition. Pattern Recognition, 90:119-133, 2019.
Mingzhao Yang, Shangchao Su, Bin Li, and Xiangyang Xue. Exploring one-shot semi-supervised federated learning with pre-trained diffusion model. arXie preprint orXiv:2355.04063 2023.
Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, and Nghia Hoang, Statisti- cal model aggregation via parameter matching. Advances in Neurul Information Processing Systems 32 (NeurIPS), 32, 2019.
Huaxiang Zhang and Linlin Cao. A spectral clustering based ensemble pruning approach. Newrocompuling, 139:289 297, 2014.
Jie Zhang, Chen Chen, Bo Li, Lingjuan Lyu, Shuang Wu, Shouhong Ding, Chunhua Shen, and Chao Wu. Dense: Data-frec one-shot federated learning. Advances in Neural Information Processing Systems, 35: 21414 21428, 2022.
Yue Zhao, Zain Nasrullah, and Zheng Li. Pyod: A python toolbox for scalable outlier detection. JMLR, 20 (96):1-7, 2019.
Zhi-Hua Zhou. Ensemble learning. In Machine learning. Springer, 2021.
Zui-Hua Zhou, Jianxin Wu, and Wei Tang. Ensembling neural networks: many could be better than all. Artificial intelligence, (37(1-2) 263, 2002.
Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. Advances in Neural Information Processing Systems (NeurIPS), 32, 2019.
Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou. Data-free knowledge distillation for heterogeneous federated learning. In International Conference on Machine Learning, pp. 12878-12889. PMLR, 2021.
15


Reviews:
[{"review_id": "J7eXl9PuQK", "reviewer": "Reviewer_VK3t", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n- The paper proposes a novel data-free diversity-based framework, DeDES, to address the ensemble selection problem with diversity consideration for models under the one-shot federated learning setting.\n- The authors provide a comprehensive literature review and a detailed explanation of the proposed method, which makes it easy to understand and replicate.\n- The experimental results show that the proposed framework outperforms existing state-of-the-art methods in terms of accuracy and efficiency.\n- The proposed framework is evaluated on several benchmark datasets, which demonstrates its effectiveness and potential for real-world applications.\n- The paper is well-structured and clearly presents the proposed framework and its experimental results.\nCons:\n- The experimental results are limited to a specific set of benchmark datasets and may not generalize to other datasets or real-world applications, such as FEMNIST.\n- More thorough related work discussions are suggested to include most recent publications, such as:\nPersonalized Federated Learning under Mixture of Distributions.\nInternational Conference on Machine Learning (ICML'23), 2023.\n- The paper does not provide a detailed discussion of the computational complexity and resource requirements of the proposed framework, which may limit its scalability in large-scale federated learning systems.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a novel data-free diversity-based framework, DeDES, to address the ensemble selection problem with diversity consideration for models under the one-shot federated learning setting. The authors demonstrate that their method can achieve better performance and higher efficiency over various datasets and model structures. The proposed framework is evaluated on several benchmark datasets, and the experimental results show that it outperforms existing state-of-the-art methods in terms of accuracy and efficiency. The paper is well-structured and clearly presents the proposed framework and its experimental results. The authors provide a comprehensive literature review and a detailed explanation of the proposed method, which makes it easy to understand and replicate. The experimental results are presented in a clear and concise manner, and the authors provide a thorough analysis of the results.\nOverall, the paper presents a novel and effective approach to the ensemble selection problem in federated learning. The proposed framework has the potential to be applied to a wide range of applications and can significantly improve the performance and efficiency of federated learning systems.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1). Include more datasets such as FEMNIST.\n2). More thorough related work discussions are suggested to include the most recent publications\n3). Include the computational complexity and resource requirements (communication costs) of the proposed framework and experiments on it.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Swnxywl5uu", "reviewer": "Reviewer_TB4q", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- This paper performs a lot of empirical evaluations.\n- The improvement over baseline seems large.\n\n\nWeaknesses:\n\n- Lack of theoretical guarantees.\n- What re the model scores? It does not seem to be defined in the paper.\n- Algorithm 1 makes use of 4 tricks. However, it is unclear how much each trick contributes to the utility improvement. While the authors discuss model filtering affects the accuracy in Table 5, what about dimensionality reduction and clustering? And what happens if you always select the model with the highest score / model with the most number of local data within a cluster, rather than using the mixed approach proposed in algorithm 1.\n- p_low, p_high and scale s should be the input for algorithm 1 as well\n- How does the authors tune the hyperparameters of the proposed method? There are a lot of additional hyperparameters (5) In the proposed method. The authors only seem to discuss how K affects the utility. What about other hyperparameters? More importantly, how do we select these hyperparameters? It is not intuitive to me what is a good grid for searching hyperparameters like \\tau, p_low and p_high.\n- Table 2 and Figure 5 seems to be showing conflicting results. In Table 2, AS sometimes perform worse compared to DeDES. However, Figure 5 seems to suggest that larger K provides stronger ensemble test accuracy. Could the authors explain this? Also in Figure 5, when m=400, if K could be selected to be 400 as well, isn’t that suggesting that Algorithm 2 filters out no outliers?\n- The argument that ‘AS is considerably less efficient’ is not convincing to me. First of all, where are the test data located? From Figure 1, it does not seem that the test data are assumed to live on the edge. In that case, there will be no downward communication from server to client. In that case, while AS takes O(m) inference time, it does not need to spend the huge time complexity spent by DeDES. In particular, it does not need to do clustering, whose time complexity could be really bad when d is large. To me the method only improves upon AS in terms of efficiency when you consider the downward communication cost, which is not explicitly discussed in the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors proposed a new one-shot federated learning method called DEDES. The proposed method utilizes several tricks, including model filtering, clustering, etc. Empirical performance seems to improve over baselines.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Could the authors provide the physical running time for AS and DeDES for comparison? Explanation on why the proposed method is better than AS is also crucial.\n- Could the authors provide ablation studies on different hyperparameter combinations, as well as explanations on how to tune those hyperparameters?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "g1hqL5ZNZ4", "reviewer": "Reviewer_62Qy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n• The authors explore a topic less investigated in the research literature, considering scenarios with one-shot learning for federated learning. In these settings, the challenge is to maximize the performance of the aggregate model by combining the models from the participants appropriately. The topic is interesting, and the application is well justified. \n\n• The method is novel and the way to form the ensemble of models relying on clustering algorithms seems to bring benefits for applications in federated learning with non-IID data. The authors show that using a subset of models using their proposed method is more efficient (both in terms of performance and computational complexity) than using all the participants’ models at test time. \n\n• The experimental evaluation is comprehensive an provide a good overview of the advantages of the method as well as the analysis of its different components. The datasets and models used for the experiments are reasonable and the results are convincing. \n\n\nWeaknesses: \n\n• Lack of a theoretical analysis or justification on why the proposed algorithm is a good approximation for solving Problem 1. \n• Some elements of the proposed method need more justification. \n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a novel method for one-shot federated learning, where the participants trained the models completely before sending them to the aggregator. Then, the authors propose an ensemble-based method for producing the outputs at inference time. For the ensemble, the authors propose to select a subset of the participants´ models relying on a clustering algorithm that considers the diversity of the models, the number of training points available for each participant and their local performance. The experimental evaluation shows that the proposed approach outperforms other competing methods, including different ensemble-methods, obtaining a more noticeable advantage in scenarios where the datasets ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The method looks interesting and the empirical evaluation, convincing. However, there are certain aspects that can be improved, especially in the description of the algorithm:\n\n• How does the proposed algorithm solve Problem 1? The algorithm is a heuristic that aims to provide a good performance. But there is no clear justification on why the chosen algorithm is a good approximation for solving Problem 1. \n\n• The paragraph about the model representation is too generic. It is fine that different methods can be applied, but the authors should provide more details on concrete techniques that can be more appropriate for this. For example, from the empirical evidence found in the experiments.  \n\n• In the description of the algorithm (section 4.1), the authors mentioned a list of different clustering algorithms that can be used. From the results in the experiments, I think that the authors should clarify better or discuss with clustering algorithms are more suitable for the scenario considered. For instance, the observation about the performance of DBSCAN in appendix C.3 is interesting, as well as the results in Section 5.6.2 comparing K-Means and hierarchical clustering. \n\n• In Section 4.1, in the paragraph for representative model selection, the justification for using the score or the number of training data points (depending on the training data balance across clients) is missing and it seems a bit arbitrary. Why is this a good strategy? \n\n• Algorithm 2 not is not well described explained. In case the authors want to keep the main paper short, this part can be moved to the appendix if needed, but, in any case, the details of the algorithm should be explained more clearly. \n\n• The privacy analysis in Section 4.3 requires more depth: the aggregator can perform different types of attacks (e.g., model extraction, membership inference attacks). Same happens if the final model is distributed across the participants of the FL task. \n\n• I did not understand well Section 4.2. Could the authors clarify how is it possible to cluster different structures just by comparison using a DR algorithm? What happens when the different model structures have different number of layers?\n\n• Minor comment: In the related work in Section 2.2 it would be nice to cite the seminal papers on Bagging, boosting, etc. (e.g. Breinman and early works from Schapire in the 90s). \n", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The main contribution of this paper is to propose an empirical approach to the ensemble selection problem under one-shot federated learning. The paper performed a comprehensive evaluation of the proposed methods over multiple standard benchmarking tasks and various aspects (e.g., computational overheads).

Two reviewers are positive about the contribution(s), and one reviewer had more concerns than strengths. However, the authors have done a good job of addressing the question and concerns, and therefore, the AE would like to recommend “Accept.”

**To the authors:** The AE encourages the authors to incorporate the promised revisions that have not been incorporated yet in the final version of the paper.
Decision: Accept (poster)