pFedClub: Controllable Heterogeneous Model Aggregation for Personalized Federated Learning
Authors: Jiaqi Wang
Qi Li
Lingjuan Lyu
Fenglong Ma
Venue: NeurIPS 2024 poster (2024)
Keywords: Heterogenous federated learning
Primary area: infrastructure

Abstract: Federated learning, a pioneering paradigm, enables collaborative model training without exposing users’ data to central servers. Most existing federated learning systems necessitate uniform model structures across all clients, restricting their practicality. Several methods have emerged to aggregate diverse client models; however, they either lack the ability of personalization, raise privacy and security concerns, need prior knowledge, or ignore the capability and functionality of personalized models. In this paper, we present an innovative approach, named pFedClub, which addresses these challenges. pFedClub introduces personalized federated learning through the substitution of controllable neural network blocks/layers. Initially, pFedClub dissects heterogeneous client models into blocks and organizes them into functional groups on the server. Utilizing the designed CMSR (Controllable Model Searching and Reproduction) algorithm, pFedClub generates a range of personalized candidate models for each client. A model matching technique is then applied to select the optimal personalized model, serving as a teacher model to guide each client’s training process. We conducted extensive experiments across three datasets, examining both IID and non-IID settings. The results demonstrate that pFedClub outperforms baseline approaches, achieving state-of-the-art performance. Moreover, our model insight analysis reveals that pFedClub generates personalized models of reasonable size in a controllable manner, significantly reducing computational costs.

Controllable Heterogeneous Model Aggregation for Personalized Federated Learning
Jiaqi Wang' Qi Lingjuan Lyu
Fenglong Ma
'The Pennsylvania State University PIoa State University 'Sony AI (iqwang, fenglong @psuu edu, gli@iastate. lingjuan. www.ony.......o.. com
Abstract
Federated learning. a pioncering paradigm, enables collaborative model training without exposing users' data to central servers. Most existing federated learn- ing systems necessitate uniform model structures across all clients, restricting their practicality. Several methods have emerged to aggregate diverse client mod- els; however, they either lack the ability of personalization, raise privacy and security concerns, need prior knowledge, or ignore the capability and function- ality of personalized models. this paper, we present an innovative approach, named pFedClub. which addresses these challenges. pFedCl ub introduces personalized federated learning through the substitution of controllable neural network blocks/layers. Initially, PFedClub uub dissects heterogeneous client models into blocks and organizes them into functional groups on the server. Utilizing the designed CMSR (Controllable Model Searching and Reproduction) algorithm, pFedClub generates range of personalized candidate models for each client, A model-matching technique is then applied to select the optimal personalized model, serving as teacher model to guide each client's training process. We conducted extensive experiments across three datasets, examining both IID and non-IID settings. The results demonstrate that pFedC1 ub outperforms baseline approaches, achieving state-of-the-art performance. Moreover, our model insight analysis reveals that pFedCl ub generates personalized models reasonable size in controllable manner, significantly reducing computational costs?.
Introduction
Federated learning (FL) [1, 2, 4, 9, 10, 11, 12, 13, 14, 15] is prevalent method to train machine learning models collaboratively without centralizing clients' data on cloud server. However, many current FI. training frameworks demand uniformity in deep neural network structures among client models, a requirement often too stringent for practical, real-world applications. An alternative approach involves equipping clients with heterogeneous models, introducing new challenge: how to aggregate these diverse models within the federated learning framework effeettivly.
Recently, various approaches have emerged to address the challenge of aggregating heterogeneous models, particularly the context of personalized FL. Some methods leverage additional information, such class information [16]. logits [17, 18), and label-wise representations [19], as intermediaries for exchanging information between clients and the server. While seemingly straightforward, this approach raises significant privacy concerns, especially regarding the potential exposure of sensitive client data. To mitigate these privacy concerns, techniques such as distillation [20, 21, 22] and model reassembly [23] have been introduced in heterogeneous FL, wherein only model parameters exchanged, akin to traditional FL approaches [1]. Despite demonstrating effectiveness aggregating
"Corresponding author.
2The source code can be found at http:pp://gg.thab.....aaaaaaaaap
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

heterogeneous models, both distillation and model reassembly techniques in FL suffer from a common drawback lack of control over personalized model generation.
Distillation-based approaches inherently necessitate the establishment of a unified model as the global model, informed by prior insights [21]. This global model is then distributed to clients to guide their training efforts. However, a smaller consensus global model may struggle to extract heterogeneous knowledge from clients, often resulting
larger size. This poses challenges for smaller clients with limited computational resources to run the shared large global model effectively. Similarly, model reassembly-baas -based approaches also encounter a related issue. While they generate personalized models for each client, these personalized models may be significantly larger than the clients' capacity, posing challenges for implementation.
To further investigate these limitations, we conducted an experiment using the state-of-the-art model re- assembly approach, pFedHR [23], on the SVHN dataset, where each client operated distinct model. (Additional details and further discussions can be found in Section 4.4.) We evaluated the parameter size of the original models and the average number of parameters in the personalized models received across communication rounds. Subsequently, we il- lustrated the parameter size differences between these two types of models in Figure 1. 'he entire bar rep- resents the average parameter size of the generated personalized model, with the original model size de- picted in blue and the increased size shown in red. personalized models produced by pFedHR are notably larger than the original local models.
Original Increase
CNNI
ChN
Figure 1: Lack of controllability demonstra- tion using pFedHR [23] on the SVHN dataset with seven different client models by compar- ing the original model size (blue bars) the generated personalized client model size (en- tire bars).
Our preliminary findings indicate that the
To address this issue, in this paper, introduce novel approach for heterogeneous model aggrega- tion, named pFedClub ub, aiming to achieve personalized Federated learning through Controllable neural network block substitution, as depicted in Figure 2. pF pFedClub receives heterogeneous models uploaded from clients on the server. pFedClub first decomposes the heterogeneous client models into different blocks and subsequently clusters these blocks based on their functionalities (refer to Section 3.2.1). pF pFedClub explores a novel neural network block substitution technique to achieve this objective, as detailed in Section 3.2.2. Specifically, pFedClut ub aims to substitute the r-th block B within the m-th client model m with block selected from the same group as B during communication round This approach ensures both the functionality of personalized models and their similarity to the original models. To enhance the diversity of the generated models, we permit arbitrary substitutions from the group for the first block (Step 1). For subsequent blocks, we introduce an order-constrained block search strategy (Step 2), ensuring the quality of the generated models. If the order constraint halts the substitution prematurely, the remaining blocks are directly added the substituted ones (Step 3). This completion strategy not only reduces the number of newly added parameters in the stitching blocks but also ensures similar functionality. As pF eeedc1 ub may generate multiple personalized candidate models for each client, employ the similarity-basee technique to select the personalized model, as discussed in Section 3.2.3. The selected personalized model is then distributed the respective client teacher model, guiding the client model training process through knowledge distillation.
It is essential to emphasize that the proposed pFedClub ub framework is designed to be both general and flexible, allowing for the incorporation of strict controllable constraints during the personalized candidate generation process. For instance, constraints such the model size of the generated candidates can be casily integrated into the framework. Furthermore, introducing controllability within PF pFedClus ub enables the mitigation of computational and communication costs compared to current model reassembly-basee offers greater adaptability and efficiency in handling personalized model generation. Experimental results demonstrate that pFedClub achieves state-of-the-an performance on three benchmark datasets under both IID and non-IID settings, demonstrating the effectiveness of the proposed aggregation strategy.
approaches. By incorporating controllable constraints, pF edeccll ub
2

---

Server Update
Stnllarity based Model Matchtng
Sticcing Layer
C
Personalized Model Selection (Section 3.2.3)
3
(Public Dataset p.))
Step L'Anchor Block Selection
Possadied Davvllad
Controllable Block-wrise
(Section
Substitution
Cllest
Group
Generation
f1,2
2
Persoualized Candidates
3
3
5
4
Step 3:
Block
Functlonality wase Block
Group
C
2
Completionn
Decompeslion and Grouping (Section 3.2.))
Group
c1.4
2
3
Strp 2: codder-oontralned Block Search
Uplood
Block Decompositon
Group
Group 3
1
5
3
Clienl
Group 2
Group
Climnt
Client
Client
Client
2
5
2
Update
Figure 2: Overview of the proposed pFedCl ub. We take four clients with heterogeneous models as an example. The numbers denote the blocks' indexes. The number of functional groups is 4. We take Client as an example demonstrate how pFedClub works to generate personalized candidate models. Note that the arbitrary substitution for Block of Client I is the second block from Client 4.
2 Related Work
Model Heterogeneity in Federated Learning. Heterogenous model cooperation is a challenging task in FL.. Researchers have explored submodel training techniques [24, 25], focusing training shared large global model by sending masked heterogeneous models to appropriate clients. However, these approaches often fail to provide personalized models for individual clients. Furthermore, they are considerably constrained by the limitations in freedom of model selection. In addition, FedDF [20] and FedKEMF [21] conduct ensemble distillation, but the settings of FedDF are different from ours and not for model personalization. FedKEMF utilizes mutual knowledge distillation with the requirement predefined model structures. Besides, HeteroFL [24] and FlexiFed [25] are restricted to the requirements of the client model structures. Other related research work needs extra information be exchanged between the server and clients, e-g., logits FCCL [17], class scores in FedMD [16], and label-wise representation in FedGH [19], which raises the concerns of privacy [26]. The most recent work pFedHR [23] provides layer- soyer-wise model reassembly approach solve the challenge of model heterogeneity in federated learning. However, has several limitations,
discuss in Section
Personalized Federated Learning. Instead of maintaining one global model, personalized FL cares more about each local model's performance, which is more sufficient and practical. In [27]. the authors add a proximal term to the local optimization loss function to bound the difference between the local and global model updates. The aggregated global model is treated as the initial shared model from the meta-learning perspective in [28]. In [29], the authors design new regularized client loss to optimize the local model achieve personalization. However, the discussed personalized federated learning work assumes that the clients have to share identical model structures.
3 Methodology
3.1 Overview
denote the training data stored in the n-th local client Lns where x; denotes the data, y: is the ground truth, and [D,]] is the number of training data. Each client employs a deep neural network-based model w, to train on its training data. Note that the client models {w,.... ..]) do not share identical network structures, where N is the total number of clients. The overview of the proposed pFedClub is depicted in Figure 2, containing server update and client update. During each communication round f, the randomly chosen M (M < N) client models {wf, wir will be uploaded to the server to generate their personalized models (=* ... w These personalized models will be sent to the corresponding clients teachers
Let Dn =

---

to guide the update of client models {wi+) provide the details of the model design.
(+1 Wa M
in the next communication round. Next, we
3.2 Server Update
The model heterogeneity of the uploaded M client models {wi: war) } at the f-th communication round makes it challenging generate personalized client teachers (w*1. ... w this challenge, a new controllable block-wise substitution-ban personalized model aggregation approach is proposed, which not only maintains the functionality of each block in the originally
Mr ). To tackle
uploaded client models but also injects new knowledge provided by other models to further achieve model personalization. To this end, we first divide each client model w into blocks and then group blocks into different functionality clusters.
3.2.1 Functionality-wise Block Decomposition and Grouping
Block Decomposition. Except for recurrent deep neural networks such as the long-short term memory network [30], most of the remaining ones, such as the family of convolutional neural networks (CNN), can be treated as block-stacked neural networks", Let Rm denote the number of blocks in client model w We then decompose w into blocks [B ... B''..." ... B m. R.,. J. Note that a block is either convolutional net, fully connected layer, or building block associated with shortcut connection such as residual neural networks [31].
Block Grouping. After decomposing each model, we then apply the K K-means algorithm to group blocks based on their functionality. Specifically, we apply the centered kernel alignment (CKA) technique [32] to calculate the similarity between two blocks as follows:
(I) where x denotes the input of the block B', and B (x(,)) represents the output of the block B (- m or n). Let {Gi.... ..i....... , denote the K functionality groups generated by the K K-means algorithm, where each group O contains multiple blocks from different models with similar functions.
si Bi.,)) = CKA(xim,; xi.,) + CKA(B" (xh,)),
3.2.2 Controllable Block-wise Substitution
Intuitively, if each pair of corresponding blocks of two models has similar functionality, the whole models should also be similar. Based on this straightforward intuition, we propose to replace each block B,,,, with a function-similas one chosen from the group Ge where B',,r E G%. Let G& denote the number of function-sinee similar blocks in each group G Arbitrarily replacing each block without any constraints will produce vast number of candidate models, which is approximately equal to G where R.,. is the number of blocks in model w m. It is time-consuming update all candidates.
To reduce the size of the candidate model pool. a naive solution is to randomly choose fixed number of models from the pool first and then use the model similarity score to select the personalized teacher model. Although this approach increases the diversity of the candidate model generation, is uncontrollable to introduce too much randomness, and the low-quality candidates may reduce the convergence rate of the federated system training. To solve these issues, we design Controllable Model Searching and Reproduction (CMSR) algorithm (as depicted in Algorithm 1), which is a greedy based search approach to reproduce a set of personalized candidate models for each w by considering both diversity and quality. In particular, CMSR consists of three key steps: anchor block selection, order-constrained block search, and block completion. Next, we describe the details of each step.
Step Anchor Block Selection (Alg. lines 3-7). Assume that the first block BTT., in w belongs to the group Gt CMSR then randomly selects one block in G as the substitution. The substituted block will be treated as the anchor/starting block of all the candidate models. As shown in Figure 2, w1.2 is selected as the substitution of wf.1- Note that naive solution to select the anchor block may use the block with the largest similarity score calculated via Eq. (1) instead of randomly choosing one. However, such solution significantly reduces the diversity of the generated candidates, further limits the extra knowledge borrowed from other models, and finally hurts the personalization of teacher models. Besides, B m,1 has a chance to be selected with a probability C
In this paper, do not decompose recurrent deep neural networks, which is our future work.

---

Algorithm 1: The CMSR Algorithm
input :Client model wi, block clusters (9i, .9k] output:Candide CCandidate set {cm,as"'. "1 Sm,
Initialize candidate blocks C. = (}:
for r- t- 1,.., Rm do I/ Step Anchor Block Section if / then
Randomly select one block B from the group that contains B
Add substituted block to [ [B
Record the block index q;
I/ Step 2:
Block Scarch
r / then
10
Initialize block index set Tmm.- []: I/ Assume that B E 9 for B .. S do if >> q then
11
13
14
Add the block to C.. [r]:
15
Add block index Z
16
if T then
17
break;
9 min(I....):
19 // Candidate Generation
20 Use Cn to generate candidates that satisfy order condition;
21 if the number of blocks of the candidate model is smaller than R... then
22 Run Step 3: Block Completion to complete the remaining blocks:
return :(cmm,, ".
Cm..m
Step 2: Order-constrained Block Search (Alg. lines 8-18). After selecting the anchor block in Step I, CMSR then finds the substitutions for the following blocks. The simplest solution is repeating the previous step Rm 1 times to generate candidate model. As we discussed before, it may generate low-quality candidate models.
To avoid this problem and generate controllable high-quality candidates, we maintain the order of the selected blocks, even from different client models, hard constraint [33]. Mathematically, let B p.y denote the substitution the r-th block of B f.... where p the model index and q the block index. For any substitution of the (r + 1)-th block B should satisfy the constraint q < u. As shown Figure 2, the substitutions of w include w².3 and wh.3 since the index of the first block's substitution w1,2 is 2.
Only using the order constraint is sufficient for the proposed pFedClub generate high-quality and informative candidates. First, maintaining the order of block functions can guarantee that the candidate model has functionality similar to the original model. Second, it also avoids pre- refiniin the order of operation types, which releases the constraints of personalized teacher model generation and further increases the diversity of candidates. Third, such an approach is capable of generating similar-sized personalized models for clients. It is essential for several applications with limited computational resources, such smart devices. However, PFedHR cannot control the size of the generated candidates.
It is worth noting that, aside from the order constraint, pFedClub is highly adaptable and can easily incorporate other types of constraints. In Section 4.4, we will delve into the integration of the model size constraint, demonstrating the framework's versatility and ability accommodate various constraints for personalized model generation.
Step 3: Block Completion (Alg. lines 19-22). Step 2 may stop at the certain block < R. due to the block order constraint. To maintain the original functional structure, we will add the remaining blocks of w i.c., {Br mm.+11.." ,B Rt., BBm..... to the substitutions. In such way, pFedClub can
5

---

generate a set of candidates denoted as fc ".. Cm J. where Sm is the number of generated candidate models. As shown in Figure 2. pFedclub will stop after substituting the third block wf,3 since all the block indexes in Group 4 for substituting w are not greater than 4, which is the minimum feasible block index. Thus, pF edclub will complete the generated candidates by directly using wf, as the forth block.
3.2.3 Personalized Model Selection
The final stage of pFedClub is to automatically select the "best" candidate teacher model from (c Cm.s,,. for the m-th client. However, selecting such model is non-trivial because cin. is a reassembled, incomplete model via block substitution, and the dimension sizes of different blocks may not be well-aligned.
Block Stitching. We complete each candidate model c/m.s using the network stitching technique [34]. We use a nonlinear activation function ReLU(-) on top of a linear layer, i.c. ReLU(W X + b) as the dimension mapping function.
Since the parameter values of { [W, b) in the stitching functions are unknown, it is essential to learn them with training data. Here, propose to use public dataset Dp to fine-tune the stitched candidate model c"t m.,a Note that we fix all the parameters in c/m.s (denoted as 0 m, and only update {W, b) in c't m, using the following loss if the public data are labeled:
Del
1
CE(C'm.. (x;; W, b, 0 o),
DPp
where [Dpl denotes the number of data in the public dataset, CE(..)) means the cross-entropy loss, ci "m (xxi W, b, mm.s) presents the predicted label distribution for the data x, by fixing the parameters m, and y; is the ground truth vector.
An unlabeled public dataset can be used to fine-tune the parameters {W, b} using the normalized temperature-scaled cross-entropy loss [35] for pair of data as follows:
exp(cos(c" m. (xx), ef m. (x;))/)
C.j = log
(3)
2p k=/
[kysi] exp(cos(c". m,, (x)). ctt (x;((/)))/
where cos(..-) is the cosine similarity, and is the hyperparamter. Xy is the augmentation of x,. We still fix the parameters O'... and learn {W,b)
It is worth noting that block stitching operation will not significantly increase the number of parame- ters in the candidate model. Besides, some candidate models may be generated using Step 3: block completion. For those candidates, the number of newly added parameters is much smaller. Moreover, the limited number of parameters is helpful for the new candidate models to maintain more original model information. Finally, it makes model computation efficient and speeds up the model training.
Model Selection. Let eim denote the fine-tuned candidate model via Eq. (2). We then calculate the average cosine similarity scores on logits outputted by the original model w and its candidate model ét as follows:
pP1 Σ
[Dpl cos(B3 (x),, (x)s))
1
O'ms.
where 3/, (x;) and B (x:) denote the logits of x; outputted by the models w and of 6 respec- tively. Note that this is forward propagation and does not need to train the models. Finally, the candidate model with the highest similarity scores in the set {am,, "." Sm. will be selected as the final personalized teacher model ws
3.3 Client Update
When the m-th client selected again at the j (j > 1) communication round, the personalized model w generated in the recent communication round will be distributed. Since the teacher model w*

---

usually has different network structure from the client model w we propose to use knowledge distillation to update the client model following [36] by optimizing the following loss:
1
CE(w>, (x:).;)) + (x), (x:))]
(5)
P
where À is hyperparameter and KI((,.) is the Kullback- divergence.
Experiments
4.1 Experimental Setups
Datasets. In our experiments, we utilize three commonly used datasets to validate the performance of the proposed pFedClub, including MNIST* SVHN', and CIFAR-I09. We randomly divide the datasets into three parts: 72% for training. 20% for testing, and 8% as the public dataset. We test two data distribution settings in federated learning, i.e., IID and non-IID, following existing work [23]. For the IID setting, the training and testing data are randomly distributed to N clients. For the non-III setting, each client randomly holds data belonging to two classes.
The proposed pFedClub aims to aggregate heterogeneous client models to boost federated learning performance. Based on the condition of public datasets, consider the following approaches as our baselines: (1) without using public datasets: HeteroFL [24] and FlexiFed [25]; (2) using labeled public data: FedMD [16]. FedGH [19], and pFedHR [23]; and (3) using unlabeled public data: FCCL [17], FedKEMF [21], and pFedHR [23]. We also compare the proposed pF eeCl ub with the general approaches to learning personalized federated learning models, which share the same structure for all clients. The homogeneous baselines include: FedAvg [1], FedProx [27], Per-FedAvg [28], PFedMe [29], PFedBayes [37], and pFedHR [23]. The details of all baselines can be found in Appendix A.
Baselines.
Client Model Deployment. In our experiments, we employ seven client models with different network structures, including MobileNetV1 (38]. MobileNetV2 [39], MobileNetV3 [40], and four manually designed CNN models (denoted as CNNI to CNN4). Each CNN model contains several convolutional blocks and fully connected blocks. The detailed model structures of the four models can be found in Appendix B. We the number of clients N 50 and the number of active clients M = 5 in each communication round, We propose three plans to distribute client models to validate the performance of the proposed pF DFedClub in different scenarios. First, we use all seven types of models and randomly send each model to a client (Model Zoo I). Specifically, CNNI is randomly assigned 8 clients, and each of the remaining six models randomly assigned to 7 clients. Second, we distribute the three MobileNet family models to clients (Model Zoo II). In particular, we randomly assign MobileNetV 1 to 16 clients, and MobileNetV2 and MobileNetV3 are randomly sent to the remaining 34 clients evenly. Finally, we use the four CNNs as the client models (Model Zoo III.. Each CNNI, CNN2, and CNN3 model is randomly assigned to 12 clients. The remaining 14 clients will use CNN4 as the client model.
Implementation Details. We run all the experiments on NVIDIA A100 with CUDA version 12.0 on Ubuntu 20,04.6 LTS server. All baselines and the proposed pFedClub are implemented in Pytorch 2.0.1. For the proposed pFedClu. ub and baseline PFedHR, we set the number of clusters K = 4 following [23], and the local training epoch and the server finetuning epoch are equal to 10 and 3, respectively. The hyperparameter X Eq. (5) 0.2. The hyperparameter r in Eq. (3) 0.07. We use Adam as the optimizer. The learning rate of the local client learning and the server fine-tuning learning rate equal 0.001. We use average client accuracy with three runs as the evaluation metric.
4.2 Performance Comparison
Heterogenous Model Aggregation. Table I lists the experimental results regarding the three-run accuracy of the proposed pFedClub and baselines. Note that HeteroFL and FlexiFed belong to
'https://yann. lecun. www.ooo//c///oo ceceum..cmm/xxx/miistttt http:://hhhtt.tttaaaoodddeee///e httter//www.ss..orrtttooo
7

---

Table 1: Performance (%) comparison with baselines under the heterogeneous settings.
Setting
Public Data
Dataset
MNIST
SVHN
CIFAR-10
Method
IID
Non-III
I1D
Noa-IID
IID
Non-IID
FedMD
91.12 2.44
90.03
2.98
76.22
3.01
75. 14.14:: 3.75
66.38 4 3.96
63.10 4 4.75
FodGH
92.76 = 1.93
91.27 = 2.21
78.41
2.65
75.06
2.87
71.22
+ 2.79
67.37
+ 3.06
Model Zoo
Labeled
PFcdHR
94.67 1.58
92.88
1.10
81.59
1.40
80.88
1.92
73.21
3.221
09.88
3.45
pFedclub
94.02 = 1.41
93.20
= 0.85
84.55
1.17
82.65
1: 1.56
+ 2.87 76.45
73.62
1 3:01
FedKEMF
1.77
1.08
77.56
2.47
74.23
2.77
65.08
3.12
PCCL
91.09 + 2.05
90.21
+ 2.44
79.44
+ 2:33
76.28
2.60
66.85 + 2.66
64.76
+ 2.88
Unlabeled
PFedHI
1.69
1.73
80.66
2.17
78.93
2.55
72.06
2.38
68.54
SFeaciua
93.72 1,90
92.77
1.51
83,94
2.08
81,76
+: 2:32
75.86
1:98
72.87
+ 2:04
FedMD
91.98
0.76
92.01
1.05
80.86 + 1.26
77.53 + 1.53
68.55 + 1.80
63.74 + 2.25
Model Zoo II
Labeled
pFedHR
FodGH
92.13
93.51 = 1.36
1.32
91.14
92.77
1.59
1.24
82,33 +: 1.86
78.15
1.50
80.96 =: 1.90
76.47
1.98
73.60 + 2.38
71.29 +
68.60
71.11 +: 2.76
2.42
3Fedc1us ul
1.07
93.11
1.47
2.04
73.15
FelKEMF
92.01
1.25
91.33
1.70
79.62 1 1.68
17.54
1: 2.04
09.11 1 2.45
06.07 1 2.88
Unlabeled
FCCL
PFedHIR
93.21 + 1.45
1.77
90.89
92.77
1,90
1.69
81.56
1.50
1.79
79.68
77.32
+ 1.79
1.68
71.88 + 1.75
69.94 4 2.24
SFedClua
1.34
93.41
1.85
1.22
1.47
75.52
72.31
HeteroFI.
92.48
1.14
91.25
1.45
80.57
1.37
77.60
1.68
71.08 1.57
67.87
.666
FlexiFed
91.08 = 1.52
90.10 = 1.66
80.69 +: 1.39
76.30 = 1.62
68 09 + 2.
67.15 +
Model Zoo III
FedMD
1.32
91.37
1,56
1.59
76.14
1.86
63.50
FedGH
92.93
1.52
91.44
1.08
79.03
1 1.44
75.28
1. 1.75
67.68 1 1.75
70.77 1 1.93
Labeled
PFedHR
1.93
1.64
2.36
1.71
72.
3Fedclus
93.24 1 1.36
92,66
0.98
82.69
1.61
81.21
1 1.68
2:02 74.88
71.94
1.82
FeIKEMP
92.18
0.75
91.60
1.03
78.88
1.68
16.16
1.77
08.04
65.80
PCCL
92.65 = 1.84
91.07
= 1.92
80.32
+ 1.71
76.02
1.82
67.16 + 2.42
06.73
2.68
Unlabeled
pFodHR
93.84
1.25
93. 16
1.50
81.76
2.12
78.40
2.50
71.74 + 1.68
08.23 1.79
pFedClub
93.77 1.16
93.52
1.37
82,50
1.25
81.03 1 1.49
74.71
1.40
71.68
1.59
the submodel training technique and require that each client model must be part of the global model. Thus, they are only tested with Model Zoo These results show that our proposed approach outperforms all the baselines under most settings, especially the more complicated datasets SVHN and CIFAR-1O. Compared with the most recent work pFedHR, our proposed work pF @FedClut shows superior performance over that SVHN and CI datasets under both the IID and non-lID settings. For pFedClut ub, the use the labeled public data able to boost the performance compared with the setting using unlabeled public data, which aligns with the observations in [23]. In addition, from Model Zoo III to Model Zoo I, the performance of pFedClut ub on the more complicated datasets, SVHN and CIFAR- 10, improves with the increase of diversity of the model zoos.
Homogeneous Model Aggregation. The clients are assigned the same model structures to verify the effectiveness of pFedCl ub under the homogenous setting. We test the performance with CNN2 and MobileNetV2 under the non-lll) setting and compare it with the state-of-thh-arr homogenous federated learning work. The results are shown in Table 2. Note that pFedHR and pFedClub are under the setting where the public data is labeled. We observe that the results of all the approaches on the MNIST dataset are relatively high, even with simple CNN2 model, classification on the MNIST dataset an easy task. Besides, pFedClub outperforms state-of-the-ar baselines on SVHN and CIFAR- 10 datasets using the CNN2 model or MobileNetV2 model. These results demonstrate that pFedClub ub is also effective for the homogeneous setting.
4.3 Ablation Study
Table 2: Performance (%) comparison with baselines under the homogeneous setting.
We conduct the ablation study to validate the effec- tiveness of each designed module in our proposed ap- proach with Model Zoo III under the non-III setting. In particular, we use the following four basclines: (1) pF pFedClubmea: in the anchor block selection stage (Step in Section 3.2.2), we naively select the most similar block calculated by Eq. (1) for the first block, instead of randomly selecting one. (2)
Model
Appreach
MNIST
SVHN
CIFAR-10
FedAvg
90.52
62.49
58.01
FedPPrxx
90.87
63.77
59.65
Per FedAvg
91.04
63.59
59.81
CNN2
PFedMe
91.79
64.27
60.14
PFedBayes
92.54
63.19
60.08
pFodHR
92.62
64.59
61.79
pFeaclub
92.18
66.97
64,25
FedAvg
92.07
79.42
62.13
PF @FedClum ubmin: different from pFedClub, use the block with the smallest index number as the
we
Vo
FedProx
92.85
80.33
63.60
Per-FedAvg
92.62
82.45
70.88
PFedMe
93.05
81.79
72.13
substitution. The substituted block is either itself or
PFedBayes
93.71
83.05
72.44
other models' first block. (3) pFedCl ubnoc: we conduct the block search without using the order con-
pFedHR
93.15
83.88
73.65
pFedc ub
93.68
85.26
74.97
8

---

straint in Step 2; and (4) pF eedcllubbbb we do not conduct the block completion process (Step 3 in Section 3.2.2).
We report the results in Table and provide the fol- lowing observations: (1) Removal of any one module will cause the performance drop, thus demonstrating the individual contribution of each design in our pro- posed pFedClub. (2) When we use simple strate- gies (i.e., oFedClubmar and SredClubmmin in the anchor block selection, the drop in the perfor- mance is smaller than studies (i.e., pFeeee ube and pFedC 1 nbbe). It indicates that maintaining the block number and keeping the model completion matter more than the anchor block selection. Overall, each designed module has its own contribution, and the systematic combination of all the designed modules guarantees the effectiveness of our proposed pFedClub.
Table 3: Ablation study performance (%) comparison.
Dataset
SVHN
CIFAR-10
Method
IID
Non-IID
IID
Non-IID
pFedClubmux
78.69
73.50
70,52
06.89
pF pFedClubmin
77.50
72.09
69.96
66.84
predclubsee
65.26
61.08
62.19
58.14
pF pFedclubate
63.01
59.47
61.38
56.02
pFedclub
82.50
81.03
74.71
71.68
4.4 Controllability Analysis
Experimental Setups.
The major advantage of the proposed pFedClub ub is enabling the generation of controllable personalized candidate models. To clearly exhibit the insights, we use Model Zoo as the heterogeneous model set, and each type of model is assigned corresponding client. Besides, each client will be mandatorily active during all the communication rounds. We use unlabeled public data for model training under the non-IID setting. To quantitatively evaluate the controllability of the personalized models generated by pFedClub, ub, we propose to use the average of the model size change percentage over T communication rounds as the metric for each client, which is defined as follows:
1 T
WH m
Wm
(6)
Wm
where |w*'ml denotes the parameter size of the personalized teacher model at round t. and [wml denotes the model parameter size of the original client model.
Model Comparison. Since the proposed pF pFedcluk ub is model reassembly- -based framework, for a fair comparison, choose
pFedHR the baseline. Besides, mentioned in Section 3.2.2 Step 2, the proposed 3FedC1i uub ub is flexible to incorporate other constraints. In this experiment, we take the model size into consideration and denote the model pFedClube ub The reason is that for real-world FL applications. such as training a model with smart devices, their computational capability is limited. Larger personalized models may make these devices stop working. To facilitate flexible management of the generated model's size in pFedClub" we introduce an additional parameter, 1 1. which provides flexible controllability decide the generated model size following the constraint: w [ I (1 + +1))+//w l/'l In this experiment, we set 77 0.1.
Results. Figure 3 illustrates the comparative per- formance with respect to accuracy and the model size controllability of pFedHR, pFedClum ubb and pFedClub* on the SVHN dataset under non-IID conditions. We observe that: (1) Both PF ub and #FecClub* show a superior performance over PFedHR shown in Figure3(a). (2) As for the model size control, we can ob- serve that pFedClub lub and pF oFedClubt both have better effectiveness over pFedHR in Fig- ure 3(b). For example, For example, for client 1, the average received model parameter size is around 4.8 times as the original model for pFedHR and 1.5 times as the original model for pFedClub. For clients 5 using MobileNets, the average of the received personalized teacher model parameter size using pFedClub is smaller than that of the original model size (Φ < 0). However, á is still a large positive number for PFedHR, which means the clients still receive the personalized teacher models larger than their original ones. (3) When comparing pFedclu ub with pFedC1 lubt the latter shows a slight decrease accuracy due to the rigorous model size constraint. Nonetheless, pFedClub" further refines the control over
20.35
*pFedHR
" #pFodClub
#yFedclbb
$3.21
d/2
PFedCTub
(550
CShi
C6N
C594
#FedClab
Mohhelet1
MallsNent3
(6) Performance
(b) Model Size Contrellabillyy
Figure 3: Controllability analysis.

---

the size of received personalized teacher models across all clients, ensuring they are not larger than than the original models (ø < 0). This confirms the capability of both pF pFedClub and pFedclub to effectively manage personalized model sizes, essential feature for applications sensitive to computational resources. Additional details on model size comparisons between pFedc1 u/o and pFedHR across various communication rounds are provided in Appendix C.
4.5
Computational Cost Comparison
The proposed model can be treated as a fine- een-graieed model reassembly technique, which uses controllable block-wise substitution to generate personalized candidates. In this
Ruuning Time (veuua))
200
Wa M ml Mo may po
experiment, we aim to compare the computational cost of the server between pFedClub and pFedHR using the
100
computational time at each round on the SVHN dataset un-
50
der the non-IID setting with Model Zoo III for 50 clients. We record the consumed computation time on the server side for each communication round. The results are shown in Figure 4. We can observe that the computation time at the server side of our approach pFedClub is generally shorter than that of the baseline PFedHR. Also, with the algorithm running with respect to the com- consistent and stable compared with the significant
20
49
60
59
100
Communication Round
Figure 4: Server running time v.s. com- munication round.
munication round, approach becomes shift of pFedHR. These results confirm that pF pFedC1 ub is an efficient approach for heterogeneous model aggregation compared with pFedHR.
4.6 System Running Time v.s. Accuracy
Except for controllability and computa- tional costs, system running time another key factor evaluate the utility of the pro- posed pF pFedClute ub. Toward this end, we conduct an experiment compare the con- sumed time to reach a fixed accuracy. The
sptedin apFaihh
50.7%
Thme 69
JECESS
34%
40L1E%
experimental setting is the same as the one that we described in Section 4.5. We take PFedHR as baseline for comparison again. The results are shown in Figure 5. We can observe that proposed approach pFedCl ub takes less time to achieve the target accuracy on the SVHN and CIFAR-10 datasets under the non-III setting compared with PFedHR. These results demonstrate the effectiveness of the proposed pFedClub for the heterogeneous model aggregation in federated learning again.
Acouracy
A Accuracy
Figure 5: The consumed running time (in seconds) of models achieve the target accuracy.
4.7 Extra Experimental Results
To validate the model scalability of pFedClub, we conduct the experiments by considering different numbers and different active ratios of clients, and the results are shown in Table 4 in Appendix D. Besides, in our model design, there is key parameter K used in Section 3.2.1. We validate the sensitivity of the selection of K, and the results are listed in Table in Appendix E.
Conclusion
This paper introduces pFedClub designed revolutionize personalized federated learning. By leveraging a unique network block substitution method, aFedClub effectively creates tailored and functionally analogous personalized models for individual clients. Moreover, pFedClub is highly adaptable and can easily incorporate other types of constraints to achieve application-driver personalized model generation. Our experimental evaluations, conducted on three diverse datasets under both IID and non-lII settings, unequivocally validate the efficacy of pFedClub in the domain of heterogeneous model aggregation for federated learning. The results affirm the accuracy, efficiency, and flexibility of our proposed method, demonstrating its potential for real-world applications.
10

---

Acknowledgements This work is partially supported by the National Science Foundation under Grant No. 2348541 and 2238275.
References
[1] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. commumication-efficcce learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.
[2] Liang Qu. Ningzhi Tang, Ruiqi Zheng, Quoe Viet Hung Nguyen, Zi Huang, Yuhui Shi, and Hongzhi Yin. Semi-decentralized federated ego graph learning for recommendation, arXiv preprint arXiv:2302. 10900, 2023.
[3] Chung-ju Huang, Leye Wang. and Xiao Han. Vertical federated knowledge transfer via repre- sentation distillation for healthcare collaboration networks. In Proceedings of the ACM Web Conference 2023, pages
2023.
[4] Kaibin Wang, Qiang He, Feifei Chen, Hai Jin, and Yun Yang. Fededge: Accelerating edge- assisted federated learning. In Proceedings the ACM Web Conference 2023, pages 2895-2904, 2023.
[5] Bingyan Liu, Yifeng Cai, Hongzhe Bi, Ziqi Zhang, Ding Li, Yao Guo, and Xiangqun Chen. Beyond fine-tuning: Efficient and effective fed-tuning for mobile/web users. In Proceedings of the ACM Web Conference 2023, pages 2863,,873, 2023.
[6] Xiangrong Zhu, Guangyao Li, and Wei Hu. Heterogeneous federated knowledge graph em- bedding learning and unlearning. In Proceedings of the ACM Web Conference 2023, pages 2444-2454, 2023.
[7] Yuke Hu, Wei Liang, Ruofan Wu, Kai Xiao, Weiqiang Wang, Xiaochen Li, Jinfei Liu, and Zhan Qin. Quantifying and defending against privacy threats on federated knowledge graph embedding. In Proceedings of the ACM Web Conference 2023, pages 2306-2317, 2023.
[8] Tao Guo, Song Guo, and Junxiao Wang. pfedprompt: Learning personalized prompt for vision- language models in federated learning. In Proceedings of the ACM Web Conference 2023, pages 1364-1374, 2023.
[9] Han Xie, Li Xiong, and Carl Yang. Federated node classification over graphs with latent link-type heterogeneity. In Proceedings of the ACM Web Conference 2023, pages 556-566, 2023.
[10] Lars Heling and Maribel Acosta. Federated sparql query processing over heterogeneous linked data fragments. In Proceedings of the ACM Web Conference 2022, pages 1047-1057, 2022.
[11] Haoyu Wang, Handong Zhao, Yaqing Wang, Tong Yu, Jiuxiang Gu, and Jing Gao. Fedke: Fed- erated knowledge composition for multilingual natural language understanding. Proceedings of the ACM Web Conference 2022, pages 1839-1850. 2022.
[12] Chengxu Yang, Qipeng Wang, Mengwei Xu. Zhenpeng Chen, Kaigui Bian, Yunxin Liu, and Xuanzhe Liu. Characterizing impacts of heterogeneity in federated learning upon large-scale smartphone data. In Proceedings of the Web Conference 2021, pages 935-946, 2021.
[13] Jinze Wu, Qi Liu, Zhenya Huang, Yuting Ning, Hao Wang, Enhong Chen, Jinfeng Yi, and Bowen Zhou. Hierarchical personalized federated learning for user modeling. In Proceedings of the Web Conference 202/, pages 957-968, 2021.
[14] Chenglin Li, Di Niu, Bei Jiang, Xiao Zuo, and Jiamming Yang. Meta-har: Federated representa- tion learning for human activity recognition. In Proceedings of the web conference 2021, pages 912-922, 2021.
[15] Weiming Zhuang, Jian Xu, Chen Chen, Jingtao Li, and Lingjuan Lyu. Coala: practical and vision-centric federated learning platform. arXiv preprint 2xXv:24477.16660 2024.
11

---

[16] Daliang Li and Jumpu Wang. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint arXiv: 1710v.1910.03581, 2019.
[17] Wenke Huang, Mang Ye, and Bo Du Learn from others and be yourself in heterogeneous federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10143 10153, 2022.
[18] Lichao Sun and Lingjuan Lyu. Federated model distillation with noise-free differential privacy. arXiv preprint arXiv:2009.05537, 2020.
[19] Liping Yi, Gang Wang, Xiaoguang Liu, Zhuan Shi, and Han Yu. Fedgh: Heterogeneous federated learning with generalized global header. arXiv preprint arXiv:230. 2023.
[20] Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. Advances in Neural Information Processing Systems, 33:2351-2366 2020.
[21] Sixing Yu, Wei Qian, and Ali Jannesari. Resource-aware federated learning using knowledge extraction and multi-model fusion. arXiv preprint arXiv:2208. 07978, 2022.
[22] Jiaqi Wang, Shenglai Zeng, Zewei Long, Yaqing Wang. Houping Xiao, and Fenglong Ma. chooweddge-nhhnneee semi- supervise federated learning for aggregating heterogeneous lightweight clients in iot. In Proceedings of the 2023 SIAM International Conference on Data Mining (SDM), pages 496-504. SIAM, 2023.
[23] Jiaqi Wang, Xingyi Yang, Suhan Cui, Liwei Che, Lingjuan Lyu, Dongkuan Xu, and Fenglong Ma. Towards personalized federated learning via heterogeneous model reassembly. Advances in Neural Information Processing Systems, 2023.
[24] Enmao Diao, Jie Ding, and Vahid Tarokh. Heterofl: Computation and communication effi- cient federated learning for heterogeneous clients. In International Conference on Learning Representations, 2020.
[25] Kaibin Wang, Qiang He, Feifei Chen, Chunyang Chen, Faliang Huang, Hai Jin, and Yun Yang. Flexifed: Personalized federated learning for edge clients with heterogeneous model architectures. In Proceedings of the ACM Web Conference 2023, pages 2979-2990, 2023.
[26] Lingjuan Lyu, Han Yu, Xingjun Ma, Chen Chen, Lichao Sun, Jun Zhao, Qiang Yang, and S Yu Philip. Privacy and robustness in federated learning: Attacks and defenses. IEEE transactions on neural networks and learning systems, 2022.
[27] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429-455 450,0 2020.
[28] Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach. Advances in Neural Information Processing Systems, 33:3557-3568, 2020.
(29] Canh T Dinh, Nguyen Tran, and Josh Nguyen. Personalized federated learning with moreau envelopes. Advances in Neural Information Processing Systems, 33:21 33:21394-21405, 2020.
[30] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 18): 1997.
[31] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016
[32] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference Machine Learning, pages 3519-3529. PMLR, 2019.
[33] Xingyi Yang, Daquan Zhou, Songhua Liu, Jingwen Ye, and Xinchao Wang. Deep model reassembly, Advances in neural information processing systems, 35:25739-2553 2022.
12

---

[34] Zizheng Pan, Jianfei Cai, and Bohan Zhuang. Stitchable neural networks. arXiv preprint arXiv:2302.06586, 2023.
[35] Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pages 1597-1607. PMLR, 2020.
[36] Ying Zhang.
Tao Xiang, Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4320-4328, 2018.
[37] Xu Zhang, Yinchuan Li, Wenpeng Li, Kaiyang Guo, and Yunfeng Shao. Personalized federated learning via variational bayesian inference. In International Conference on Machine Learning, pages 26293-26310. PMLR, 2022.
(38) Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint 1704::1704..4461,, 2017.
[39] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chied -hicc Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510-4520, 2018.
[40] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1314-1324, 2019.
13

---

Appendix
A. Baselines
In the heterogeneous experiments, we use the following approaches as baselines:
(1) Without using public datasets:
HeteroFL [24]: Local clients' models are required to belong to the same model class and work together to produce one single global inference mode. Specifically, they revise the batch normalization, conduct pre-activity scaling, and design a masked loss to solve this research problem.
The common parts of the clients' models work together, and the different parts work separately to keep the system updated. They provide basic-common strategy, cluster-common strategy, and max-common strategy to conduct the heterogeneous model aggregation under the setting.
FlexiFed [25]:
(2) Using labeled public data:
FedMD [16]: uses transfer learning and knowledge distillation with the labeled public data the server side. Specifically, each client needs train the local model on both the public dataset and the private dataset. Then, clients upload the class scores on the public dataset to the server, and the server calculates the consensus send it back for local update.
FedGH [19]: The clients have their own feature extractors and share the homogeneous global header. The clients train local models on their local data and upload the representation and label for each label back to the server for the global header update. Then, the clients replace their own headers with the global one for inference.
pFedHR [23]: This approach utilizes the model disassembly techniques to decompose local models into layers. The server composes the layers back and tunes the models while stitching the layers using the public datasets.
(3) Using unlabeled public data:
FCCL [17]: FCCL leverages the unlabeled public data and averages the logits from local clients. The approach utilizes consensus logit to guide the local training.
FedKEMF [21]: This approach aggregates knowledge from local models and distills it into global knowledge via knowledge distillation. uses mutual learning to personalize the models on the server side with the unlabeled public data;
pFedHR [23]: This approach is also able to use the unlabeled public data to tune the stitching layers of the candidate models.
The following baselines are used in the homogencous experiments:
FedAvg [1]: It the vanilla version of federated learning, which averages the model parameters from the local clients;
FedProx [27]: It adds the proximal term to the local model training based on FedAvg;
Per-FedAAg [28]: The MAML framework is proposed based on meta learning:
PFedMe [29]: It uses regularized loss and decouples the personalization problem into bi-level optimization;
PFedBayes [37]: It proposes algorithm to take consideration of the global distribution while conducting local model training;
PFedHR [23]: It generates personalized models by model decomposition and composition for local clients to guide local model training.
B. CNN Strutures
In our experiments, we have 4 CNN models with different complexity. The details are shown as follows. each convolutional NN sequential block, there convolutional layer, a max pooling layer, and ReLU function.
14

---

CNN1: Covl:/Comv (kernel size 5) + ReLU - MaxPool2D (kernel size 2, stride 2) } + Cov2:/Conv2d (kernel size = = - ReLU + MaxPool2D (kemel size 2. stride 2) + FCI:(Liness + ReLU) - Dropout FC2:l.near.
CNN2: Covl:(Con:2d (kernel size 5) ReI.U MaxPool2I) (kernel size stride 2) }
+ Dov2:CConv22 (kernel size = 5) ReLU + MaxxPol2D (kernel size = 2, stride 2) +
Cov3: (Cooo22d (kernel size = =) - ReLUI FCI:[Linear- ReLUI Dropout FC2:Linear.
CNN3: Covl: Conv2d (kernel size = 5) + ReLU + MaxPool2D (kernel size = 2, stride = 2)
Covl:(Cono2d
Cov2:/Conv22 (kernel size 5) - ReLU + MaxPool2D (kernel size 2. stride 2)
Cov3:/ Conv2dd (kernel size = 5) ReLU| - Cov4:/Conv2d (kernel size = 5) - ReLU MaxPool2D ReLU] + FCI:/Linear + ReLU
(kernel size- 2, stride 2) Cov5: (kernel size
+ Dropout) + FC2:(Linear + ReI.Uj~>
- (e::L))--FC3:(lmmeeeee FC3:(Linear
Rel.U)-FFC4:ineeur
CNN4: Covl:(Conv2d (kernel size 5) BatchNorm2d - ReIUU] Cov2:/Comm20 (kernel size Cov3://onn2dd (kernel size = 3)
3) + ReLU -> MaxPool2D (kernel size stride = 2)
BatchNorm2d - ReLUJ + Cov4:/Conv22 (kernel size 5) - ReLU - MaxPool2D (kernel size
2. stride = 2) - Dropout } - Cov5:(Conv2d (kernel size = 3) - BatchhNor2dd - RReU
Cov6:|Conv2u4 (kernel size = ReLU + MaxPool2D (kernel size = 2, stride 2) FCI:(Linear
+ ReLU + Dropout] - FC2:/Linear - ReLU) FC3:aLinear ++ ReLU))=FC4:Lmeaar
C. Generated Model Size Comparison
pUedlER
pFedch
In Section 4.4, we have validated that the size of the generated personalized models by the proposed PF DFedClub is much smaller than that produced by pFedHR. Figure 3 shows the relatively average change in model size. In this experiment, we aim to show detailed comparison of model size changes each communication round for each type of hetero- geneous model.
10
The results are shown in Figure 6. We can observe that our approach pFedClut ub conducts better control over the generated models compared with pFedHR in almost every communication round, especially for the client with the MobileNet models, where the size of our generalized model parameters always smaller than the original one (Ø < 0).
D. Model Scalability Analysis
Besides, model scalability is important in feder- ated learning systems. To validate the scalability of PF redClub, we conduct the following experiments by considering different numbers and different active ratios of clients. The results are shown in Table
20
40
60
80
100
Communication Round
Figure 6: Model size controllability compared with PFedHR with respect to the communica- tion round.
Given the different settings of the different numbers of clients and active ratios, we can observe that the performance changes according to our expectations. Moreover, given a fixed active ratio, the performance has slight decrease with the increase in the client number. One possible reason is each client will have a smaller number of training data when the client number increases given certain number of training data. The change in the local model performance will harm the performance of the whole system in a certain range. Furthermore, given a fixed number of clients, the increase in the active ratio will slightly boost the performance because it enables more clients to contribute to the update process with their local training at each communication round. The experiment results demonstrate the scalability of our proposed approach, considering the change in the client number and active ratio.
15

---

Table Scalbility of pFedClub with different numbers of clients and different active ratios on the SVHN dataset.
Data
IID
non-IID Active Ratio
Client Number
Active Ratio
10%
20%
30%
10%
20%
30%
30
82.84
84.68
85.02
81.23
82.58
84.47
50
82.50
84.02
84.89
81.03
82,41
83.15
100
79.26
82.55
83.97
77.06
79.43
80.22
Table 5: Hyperparameter study of the number of clusters. The performance (%) of pFedClub with different values of K on the SVHN and CIFAR-10 datasets.
Public
Dataset Cluster
SVHN
CIFAR-10
Dataset
IID
Non-IID
IID
Non-IID
3
80.45
78.21
72.02
68.04
Unlabeled
4
82.50
81.03
74.71
71.68
5
82.66
81.17
75.26
71.89
3
80.76
79.88
73.07
68.95
Labeled
4
82.69
81.21
74.88
71.94
82.77 5
81.35
75.89
72.38
E. Hyperparameter Study
In our design, K the number of the groups based on the function-wise clustering with K-means. In this experiment, we aim to study how the hyperparameter K affects the performance. We maintain the experimental setting in Sections 4.5 and 4.6. The results are shown in Table 5. We can observe that the performance of pFedClub will slightly increase with the increase of K. In the main experiments, the performance can increase slightly. One possible reason is that a large K is able to produce more specific function Lunction-based groups and further identify the function the blocks more accurately. We can also observe that the performance generally stable with the change of hyperparameter K in certain range.
we follow [23] and set K 4. When
F. Limitations and Broader Impacts
This work focuses on the controllable personalized model generation for the heterogeneous federated learning setting, Although the proposed pFedClus ub outperforms baselines and is able to generate size-controllable client models, it still has several limitations. First, the proposed controllable model searching and reproduction (CMSR) algorithm is heuristic algorithm that is designed based on intuitions. Thus, the results may suboptimal. Second, in the system running time experiments, we have demonstrated that the proposed pF FedClub can reduce the learning time compared with the
model pFedHR. However, compared with traditional averaging methods, the running time the server is still much longer. We plan to design more efficient algorithm for the server update. Finally, in the experiments, we only test the image classification task, which limits the efficacy test on other tasks. We will test more diverse tasks with the proposed pF FedClub in the future.
This research significantly enhances federated learning by enabling efficient aggregation of heteroge- neous models without compromising data privacy. This approach not only improves user experience across diverse sectors by providing tailored solutions but also supports sustainable computing prac- tices of large-scale machine learning operations.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our abstract and introduction reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation in the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
Our paper has no theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer:[Yes
Justification:
We provide the codes and experiment details to reproduce the results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer:[Yes
Justification:
We provide the code in the supplementary file and we use the open datasets.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines (https: /tt:://nppppe cc/ des CoodeSuomei ss ssion assionoolioo icy for more details.
publ public/ / guides/
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips. cc/ httpsscccccccccccccce ptppppplc//// hwtpc//gg/s/////eeeeceeeecscssse //odee////eeas CCCoooSSubaa commissionni wwwssssoooiiiooooooo onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We specify the details as asked.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We run experiment multiple times. We provide the mean and the STD.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
19

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the details Implementation Details Sec 4.1. We report the running time in Sec 4.5.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http//eurrppppeeee http://s.....ooccaattttcccooa..a
Answer: [Yes]
Justification: Yes.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We provide the broader impacts the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the original paper. code package, and dataset.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, papers ssw thcode com dat set. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
Justification: We provide the details of the designed model and provide the codes in the supplementary file.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "XUeu9qlNKJ", "reviewer": "Reviewer_5tH7", "summary": "This paper addresses heterogeneous model aggregation in federated learning. To this end, the authors introduce pFedClub, which aims to generate personalized models for federated clients while ensuring that the models remain within size constraints. Specifically, pFedClub consists of three main steps: first, it decomposes models into multiple blocks and clusters them using the K-means algorithm; second, it replaces original blocks with others from the same clusters to create a set of candidate models; third, it selects the optimal personalized model for each client using a public dataset and an initial model transferred to the server. Extensive experiments illustrate its significant improvement over existing methods in this field.", "questions": "See **Weaknesses**", "limitations": "See **Weaknesses**", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The work is well motivated and explores an interesting problem in federated learning. \n2. The presentation of this paper is clear, and the authors comprehensively and intuitively describe the proposed pFedClub. \n3. The paper conducts sufficient experiments and compares the proposed method with previous works. The numerical results demonstrate the superiority of pFedClub.", "weaknesses": "1. The proposed work requires a public dataset, which is unsuitable in federated learning due to the privacy concerns. Is this work applicable to a public dataset different from the training data distribution? For example, the clients collaboratively train a model for CIFAR-10, while the server holds a public dataset from ImageNet. \n2. Although the proposed work achieves remarkable under convolutional neural networks, it is unclear how pFedClub performs under transformers. Is the proposed work suitable for a setting where clients hold three different sizes of LLM, i.e., LLaMA-7B, LLaMA-13B, and LLaMA-70B?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "`>>> W1`\n\nThank you for your comments. We would like to emphasize that our research question addresses a significantly challenging problem, where each client maintains a unique model. Aggregating these heterogeneous models on the server is particularly difficult without the use of public data. Additionally, we adhere to established work that serves as baselines in our experiments, which also incorporate public data during model training. While we acknowledge that differences in data distributions between public and private data can affect performance, our model design mitigates this issue, similar to the approach taken by pFedHR.\n\nFollowing the reviewer’s suggestion, we conducted an additional experiment where clients train models using CIFAR-10, and the server utilizes a public dataset from ImageNet. All other hyperparameters were kept consistent with those listed in Table 1 for Model Zoo 1 under the non-IID setting. The results of this experiment are as follows:\n\n| Public Data | Labeled | Unlabeled |\n|-------------|---------|-----------|\n| CIFAR-10    | 73.62   | 72.87     |\n| ImageNet    | 71.50   | 69.33     |\n\nWe observe that using ImageNet as the public dataset does decrease performance compared to using CIFAR-10 directly. However, the performance drop is limited, and it still outperforms the best baseline, pFedHR (Labeled: 69.88 and Unlabeled: 68.54), as shown in Table 1. \n\n`>>> W2`\n\nThank you for your feedback. In our current work, we focused on reassembling convolutional neural networks in our experiments, as they consist of distinguishable layers. However, our model design is flexible and can also be applied to transformer-based models, as demonstrated in [1], which segments transformer models into functional blocks and reassembles them into candidates. We did not test the transformer or more advanced models mentioned by Reviewer 5tH7 due to the following reasons: This work focuses on controllable personalized model generation, specifically designed for resource-constrained clients. Training transformer-based models from scratch typically requires a large amount of training data, which may be impractical for clients using mobile devices due to computational costs. Therefore, we did not include experiments with transformers in this study. We can include additional results involving transformer models in the final version of our paper if required.\n\n[1] Yang, Xingyi, Daquan Zhou, Songhua Liu, Jingwen Ye, and Xinchao Wang. \"Deep model reassembly.\" Advances in neural information processing systems 35 (2022): 25739-25753.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cekm2YmEwd", "reviewer": "Reviewer_AXhq", "summary": "The paper proposes a `pFedClub` method for personalized federated learning that enables controllable heterogeneous model aggregation, addressing limitations of existing approaches such as lack of personalization, privacy concerns, and uncontrolled model size growth.\n\nExtensive experiments conducted on three benchmark datasets using various CNN-based model structures validate the effectiveness of the proposed method under both IID and non-IID settings.", "questions": "1. Table 5 indicates that the model achieves the best performance in both IID and non-IID settings when K equals the number of activated clients. However, this raises a question about the necessity of the K-means method in this scenario. When K equals the number of activated clients, the input data naturally satisfies the minimization objective of the K-means algorithm, rendering the clustering step redundant. It would be valuable to explore how K affects the results when the number of activated clients increases, for example, to 10 or 20. This analysis would provide deeper insights into the scalability and robustness of the proposed method.\n\n2. While the current experiments focus on CNN-based structures, it would be beneficial to validate the proposed method on other neural network architectures. Specifically, evaluating pFedClub on Transformer-based structures would demonstrate its versatility and applicability across different model types. This expansion of the experimental scope would strengthen the paper's contributions and broaden its potential impact in the field of federated learning.\n\n3. The supplementary material should include a comprehensive set of implementation details for all methods used in the comparisons. This should encompass not only the proposed pFedClub method but also such as the hyparameter for baseline approaches.", "limitations": "None", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- They conduct extensive experiment including the discussion about the hyparameter $K$ to validate the controllability of the proposed method and computational efficiency on the server.", "weaknesses": "1. The writing and structure of the paper need improvement, particularly in the \"Order-constrained Block Search\" paragraph. The concept of order is unclear, especially the meaning of $q < u$ in line 177. It's not evident whether this refers to a similarity score or another metric. The author should provide a clearer explanation of this constraint.\n2. In equation (1) on line 141, the meaning of 'CKA' is not defined. The authors should explain what CKA stands for and how it's calculated. Additionally, it's unclear whether this computation occurs on the server. If clients must transmit input $x_{m,i}^t$ and output to the server, this raises privacy concerns that should be addressed.\n3. The paper doesn't specify whether the features $x_{m,i}^t$ and $x_{n,j}^t$ in equation (1) have the same dimensions. This should be clarified to ensure a proper understanding of the similarity calculation.\n4. The sampling process for the Anchor Block selection is ambiguous. The probability distribution over all models for this selection is not clearly defined.\n\nOverall, the authors should formulate the proposed method more rigorously, using well-defined notations and providing clear explanations for each step of the algorithm. This would significantly improve the paper's readability and reproducibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "`>>> W1`\n\nThank you for pointing this out. We follow existing work [1] to maintain the natural order of the blocks, as we aim for the generated candidate models to be similar to handcrafted network structures. Here, the natural order index is defined as the position of each block. For example, CNN1 listed in Appendix Section C consists of four layers: Conv1, Conv2, FC1, and FC2, with corresponding block order indexes of 1, 2, 3, and 4. In the generated model, we do not expect the FC1 layer to appear before the two convolutional layers, Conv1 and Conv2. Therefore, we constrain the order of blocks when generating candidate models.\n\nWe also have an example in Figure 2. In step 2: Order-constrained Block Search, if the first selection for the target network (in red) is the second block (in green), the next selected block for block 2 of the target network must have an index larger than 2. In this example, we select two “3” blocks as candidates for the second block replacement. Similarly, the third block replacement will use indexes “4” and “5”. We will add these details in the final version of our paper.\n\n[1] Yang et al. Deep model reassembly. NeurIPS, 2022.\n\n`>>> W2`\n\nThanks for the reviewer’s suggestion. CKA represents the centered kernel alignment, which is a method generally used to calculate the similarity of the neural network representations [2]. Specifically, given two kernels of feature representations K and L, we first calculate the Hilbert-Schmidt Independence Criterion (HSIC) by HSIC(K,L) = $1/(n-1)^2$ tr(KHLH), where n is the dimension of the representation of the features, and H is the centering matrix. CKA(K,L) = HSIC(K,L)/$\\sqrt{(HSIC(K,K)HSIC(L,L))}$. Due to the page limit, we did not add those specific details in the original paper. We will add them in the final version.\n\nAs we stated in Sec. 3.2, all the operations including CKA occur on the server. The clients do not need to transmit x and output to the server. Instead, clients only transmit their model parameters to the server, following the conventional federated learning approach. Upon receiving these parameters, the server uses public data as input to the initial block of models, with the output of each block serving as input for the subsequent block. We have detailed the experimental settings with public data, demonstrating that our method performs effectively with both labeled and unlabeled public data, outperforming other baselines.\n\n[2] Kornblith et al. \"Similarity of neural network representations revisited.\" ICML, 2019.\n\n`>>> W3`\n\nThanks for the reviewer’s valuable question. To ensure the generalization of our approach, the dimensions of the two features do not have to be the same, and CKA is able to handle this situation based on what we have addressed in W2 [2].\n\n`>>> W4`\n\nThanks for the reviewer’s suggestion. The anchor block selection process considers the groups G. For model $w_m^t$, assuming its the first block $B_{m,1}^{t}$ belongs to the group $G_k^t$, we randomly select one block from $G_{k}^{t}$ as the substitution. Any block from the group $G_k^t$ has the equal probability $1/G_k$ to be selected (Line 169), where $G_k$ is the number of the blocks belonging to $G_k^t$.  With such a strategy, compared with the strategy using the two blocks with the largest similarity score, it increases the diversity of the generalized candidates. At the same time, this random strategy is effective, generalized to all models, and simple to implement.  As the reviewer suggested, we will add more descriptions of the anchor block selection process in the final version.\n\nWe do appreciate the reviewer’s suggestion. We introduced the proposed algorithm CMSR in the appendix A. As the reviewer suggested, we will further enhance the readability and reproducibility in the final version. \n\n`>>> Q1`\n\nThe clustering processing is based on the functionality in Line 139 with no direct relation with the number of clients based on our design. With an increasing cluster number in some range, it indicates that we will get a more specific approach to differentiate the functions of each block, which may have some boost on the performance, as shown in Table 5.\n\n We totally agree with the reviewer about the exploration of the activated client number. We have conducted the experiments and shown the results in Table 4. In Table 4, we maintain the same value of K = 4 as the main experiment. Then we study a different total number of clients and different active ratios under the IID and non-IID settings. Furthermore, we provided the experiment observations and respective model scalability analysis in Appendix F. Combining the results in Table 4 and Table 5 and its analysis in Appendix G and F, it demonstrates the scalability and robustness of our proposed method.\n\n`>>> Q2`\n\nThanks for the valuable question. In our limitation part, we admitted that our existing approach has not been tested on other tasks, such as those with the transformer-based structures. We sincerely appreciate the constructive suggestions from the reviewer. We will add more discussion in our limitation part and expand our work following this direction in the future.\n\n`>>> Q3`\n\nWe appreciate the reviewer's suggestion. Our experiment setup, detailed in Section 4.1, includes descriptions of the datasets, baselines, client model deployment, and implementation specifics. We have ensured that common parameters are consistently maintained across different baselines while retaining the unique parameters of each baseline at their default values. As suggested by the reviewer, we will include a more comprehensive set of implementation details in the final version of our paper.\n\nWe are sincerely grateful for the reviewer's constructive questions and suggestions. We hope our responses have adequately addressed your concerns. Thank you once again for your valuable feedback, which has significantly contributed to enhancing the quality of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XNHBaG3pEv", "reviewer": "Reviewer_PfmG", "summary": "This paper presents a controllable model reassembly approach to enable heterogeneous model cooperation in federated learning. The designed CMSR algorithm provides the control of the space to save the computational cost.  Furthermore, the approach also achieves model personalization for each local client. They test the proposed approach on benchmark datasets and compare with other baselines under different settings.", "questions": "1, Please address the concerns in the weakness part.\n2, After the blocks are stitched, the parameters of the blocks and/or the stitching layer would be trained? If trained, how are they trained? If not, how do you deal with the parameters of the stitching layers?", "limitations": "There are no potential negative societal impacts of the work. There are two limitations as follows:\n(1)\tThis approach still raises extra computational cost at the server side. \n(2)\tThat would be great if this approach can be extended to other tasks and other domains.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1, This paper targets one of the challenges in federated learning, which is the model heterogeneity. To the best knowledge, most existing related works are based on knowledge distillation.  This work presents a controllable approach to conduct block assembly and reassembly from local models to achieve heterogeneous model cooperation and model personalization. The idea itself is interesting and practical.\n2, They take efficiency, generalization, and personalization into consideration. They provide comprehensive analysis and provide detailed discussion under various settings, which support their statement soundly. \n3, Their presentation and logic are both easy to follow and understand. The framework, experiment results, and discussion are clearly presented.", "weaknesses": "Weakness\n1, In their approach, the authors employ K-means clustering. The reviewer is curious about how the value of K is selected and how this selection influences the results.\n\n2, One of the main contributions compared to pFedHR is the enhanced controllability. I am interested in understanding the nature of this controllability, specifically the extent to which the generated models can be controlled.\n\n3, The paper focuses solely on image classification. Adhering to the review guidelines, the reviewer is not requesting additional experiments, but the reviewer is interested in exploring whether the existing methodology could be applicable to other tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "`>>> W1`\n\nThank you for the reviewer’s question. In the main paper, we set K to 4, following the methodology outlined in [1]. To further explore how the value of K affects our results, we conducted a hyperparameter study on K. The results of this study are presented in Table 5, with detailed analysis provided in Appendix G due to page constraints. We observed that performance slightly improves with a larger value of K within a certain range. This improvement may be due to the fact that a larger K can differentiate functions more specifically.\n\n[1] Wang et al. \"Towards personalized federated learning via heterogeneous model reassembly.\" Advances in Neural Information Processing Systems 36 (2024).\n\n`>>> W2`\n\nThanks for the reviewer’s question. Our controllability is demonstrated in sec 3.2.3 controllable block-wise substitution. Furthermore, we provided the controllability analysis in section 4.4. We define a hyperparameter $\\phi$ to quantify the controllability of the personalized models generated by pFedClub. Also, we add the flexible controllability parameter $\\eta$ to decide the generated model size. The results are shown in Figure 3. Compared with pFedHR (shown in blue), the generated models by the pFedClub (shown in red) and pFedClub+(shown in green) are significantly smaller than the ones generated by pFedHR.\n\n`>>> W3`\n\nThank you for the reviewer's valuable comments. As we mentioned in the limitations section, our experimental results are currently limited to image classification tasks. However, given the generalized design and intuitive nature of our approach, we believe it can be adapted to other tasks, provided that the models can be assembled and reassembled. We are committed to exploring these possibilities in future research directions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0413gIGNFk", "reviewer": "Reviewer_WRnm", "summary": "The authors address a key issue in personalized federated learning, which enables clients with heterogeneous model structures to participate in federated learning with consideration of effectiveness and efficiency. This method is based on model assembly and reassembly, in which the blocks and layers can be treated as modules. After that, the server selects the personalized models and assigns them to the clients. The received models will be used as the teacher to guide the local update. The authors run extensive experiments to demonstrate the effectiveness of their algorithm.", "questions": "Please see the above weaknesses.", "limitations": "No negative social impact to the reviewer’s best knowledge. The study of K should be put in the main content as that is an important part of the algorithm.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-organized and clearly motivated. Its logical structure and presentation aid comprehension, while the clear and accessible framework and figures enhance readability. Experiments, discussions, or analyses robustly support each claim.\n\n2. The focus on controllability renders the algorithm more applicable in real-world scenarios, allowing for greater human involvement in the model generation process. The authors effectively demonstrate the utility of their design through experimental results.\n\n3. The authors have performed extensive experiments, including principal studies on image datasets, ablation studies, hyperparameter evaluations, and thorough discussions. These efforts confirm the validity of the techniques and provide deep insights into the paper's contributions.", "weaknesses": "1. Based on the algorithm itself, it includes the reassembly, assembly, matching, and other operations. The reviewer may be concerned about the computational burden compared with the one without any controllability. \n\n2. How to select the anchor block and why needs to be stated clearly.\n\n3. According to the experiment results, the reviewer is wondering about how this approach can be used with the public data with/without labels and the possible reason why it is robust to the public data with or without labels.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "`>>> W1`\n\nThank you for the reviewer's valuable feedback. We addressed the computational cost comparison in Section 4.5, using computation time as a metric against pFedHR. The results, presented in Figure 4, show that pFedClub generally requires less time than pFedHR and offers more consistent performance. Additionally, we detailed the system running time comparisons with the baseline in Appendix E. Our findings demonstrate that to achieve the target accuracy, our proposed approach is more time-efficient than pFedHR, further underscoring its efficiency.\n\n`>>> W2`\n\nThank you for the question. The anchor block selection process is detailed from line 162 to line 169. For each block, we implement a random substitution, with the first substituted block designated as the anchor. This randomness enhances the diversity of candidate model generation. For a clearer explanation of this process, we have included a detailed algorithm in Appendix A, from lines 3 to 7.\n\n`>>> W3`\n\nThank you for the reviewer's question. On the server side, public data is used solely to assist in (1) calculating the representations of the blocks for function identification and clustering, and (2) conducting stitching layer tuning. These operations utilize the data as foundational information without exploiting any specific attributes of the data itself. Consequently, our proposed approach is compatible with both labeled and unlabeled public data.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: In this paper, the authors present a new method for heterogeneous model aggregation in personalized federated learning. Now, the paper has received four reviews, and the reviewers all think that the paper has studied an important problem with strong results through extensive experiments.

While the paper shows some promise, there are some weaknesses that have been identified in the reviews.    
(1) More experimental evaluations are suggested, eg, under public datasets.  
(2) Some technical details require more clarifications.  
(3) The presentation requires more improvements.

During the rebuttal, the authors provided detailed responses to each reviewer's comments, and the reviewers were generally convinced by the authors' clarifications. During the discussion period, each review showed positive score about this submission, therefore, we recommend to accept this paper in NeurIPS 2024.
Decision: Accept (poster)