Towards Personalized Federated Learning via Heterogeneous Model Reassembly
Authors: Jiaqi Wang
Xingyi Yang
Suhan Cui
Liwei Che
Lingjuan Lyu
Dongkuan Xu
Fenglong Ma
Venue: NeurIPS 2023 poster (2023)
Keywords: Federated Learning

Abstract: This paper focuses on addressing the practical yet challenging problem of model heterogeneity in federated learning, where clients possess models with different network structures. To track this problem, we propose a novel framework called pFedHR, which leverages heterogeneous model reassembly to achieve personalized federated learning. In particular, we approach the problem of heterogeneous model personalization as a model-matching optimization task on the server side. Moreover, pFedHR automatically and dynamically generates informative and diverse personalized candidates with minimal human intervention. Furthermore, our proposed heterogeneous model reassembly technique mitigates the adverse impact introduced by using public data with different distributions from the client data to a certain extent. Experimental results demonstrate that pFedHR outperforms baselines on three datasets under both IID and Non-IID settings. Additionally, pFedHR effectively reduces the adverse impact of using different public data and dynamically generates diverse personalized models in an automated manner.

Towards Personalized Federated Learning via Heterogeneous Model Reassembly
Jiaqi Wang'
Xingyi Yang² Suhan Cui
Liwei Che!
Lingjuan Lyu² Dongkuan Xu
Fenglong Ma
'The Pennsylvania State University NNational University of Singapore
Sony Al *North Carolina State University
(jqwang, sxc6192, 1fc5481, fenglong) @psu.edu, xyang@u. edu, lingjuan. Ive sony com,
edu
Abstract
This paper focuses on addressing the practical yet challenging problem model heterogeneity in federated learning, where clients possess models with different network structures. To track this problem, we propose novel framework called pFedHR, which leverages heterogeneous model reassembly achieve personal- ized federated learning. In particular, we approach the problem of heterogencous model personalization Moreover, pF edHR automatically and dynamically generates informative and di- verse personalized candidates with minimal human intervention. Furthermore, our proposed heterogeneous model reassembly technique mitigates the adverse impact introduced by using public data with different distributions from the client data to certain extent. Experimental results demonstrate that pFedHR outperforms baselines on three datasets under both IID and Non-III> settings. Additionally, pFedHR effectively reduces the adverse impact of using different public data and dynamically generates diverse personalized models in an automated manner?.
model-matching optimization task on the
side.
Introduction
Federated learning (FL) aims to enable collaborative machine learning without the need to share clients' data with others, thereby upholding data privacy 2, 3]. However, traditional federated learning approaches [2, 4, 5, 6, 7, 8, 9, 10, 11, 12. 13. 14, 15] typically enforce the use of an identical model structure for all clients during training. This constraint poses challenges in achieving personalized learning within the FL framework. In real-world scenarios, clients such as data centers, institutes, or companies often possess their own distinct models, which may have varying structures. Training on top of their original models should be better solution than deploying new ones for collaborative purposes. Therefore, practical solution lies in fostering heterogeneous model cooperation within FI., while preserving individual model structures. Only few studies have attempted to address the challenging problem heterogeneous model cooperation FL [16, 17, 18, 19, 20), and most of them incorporate the use of public dataset to facilitate both cooperation and personalization [17, 18, 19, 20). However, these approaches still face several key issues:
Undermining personalization through consensus: Existing methods often generate consensual side information, such as class information [17], logits [18, 21], and label-wise representations [22], using public data. This information then exchanged and used to conduct average operations on the server, resulting in a consensus representation. However, this approach poses privacy and
Corresponding author. Source code be found in the link
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

54%
2274
EXDAST EXERN ACPMAE
20%
23000 62%
19%
AM7%
12.055
125
45%
phidil (e) LID with labeted public dataset
FalGil
phodii
19%
sPiditt
(b) Neo-IID with lsheled public dataset
(1) IID with uslabeled public datanet
(d) Non-IID with uslabcled public dattact
Figure 1: Performance changes when using different public data, pFedHR is our proposed model.
security concerns due to the exchange of side information [23]. Furthermore, the averaging process significantly diminishes the unique characteristics of individual local models, thereby hampering model personalization. Consequently, there is need to explore approaches that can achieve local model personalization without relying on consensus- based techniques.
Excessive reliance on prior knowledge for distillation-based approaches: Distillation-han -base techniques, such as knowledge distillation (KD), are commonly employed for heterogeneous model aggregation in FL. [19, 20, 24). However, these techniques necessitate the predefinition of shared model structure based on prior knowledge [20]. This shared model is then downloaded to clients to guide their training process. Consequently, handcrafted models can heavily influence local model personalization. Additionally, a fixed shared model structure may be insufficient for effectively guiding personalized learning when dealing with a large number of clients with non-IID data. Thus, it is crucial to explore methods that can automatically and dynamically generate client-specific personalized models as guidance.
Sensitivity to the choice of public datasets: Most existing approaches use public data to obtain guidance information, such as logits [18, 21] or shared model [20], for local model personalization. The design of these approaches makes public data and model personalization tightly bound together. Thus, they usually choose the public data with the same distribution as the client data. Therefore, using public data with different distributions from client data will cause significant performance drop in existing models. Figure I illustrates the performance variations of different models trained on the SVHN dataset with different public datasets (detailed experimental information can be found in Section 4.4). The figure demonstrates a significant performance drop when using alternative public datasets. Consequently, mitigating the adverse impact of employing diverse public data remains a critical yet practical research challenge in FL.
Motivation & Challenges. In fact, both cansensus-based and distillation-base approaches aim to learn aggregated and shared information used as guidance in personalized local model training, which not an optimal way to achieve personalization. An ideal solution is to generate a personalized model for the corresponding client, which is significantly challenging since the assessable information on the server side can only include the uploaded client models and the public data. To avoid the issue of public data sensitivity, only client models can be used. These constraints motivate us to employ the model reassembly technique [25] to generate models first and then select the most matchable personalized model for specific client from the generations.
To this end, we will face several new challenges. (C1) Applying the model reassembly technique will result in many candidates. Thus, the first challenge is how to get the optimal candidates. (C2) The layers of the generated candidates are usually from different client models, and the output dimension size of the first layer may not align with the input dimension size of the second layer, which leads to the necessity of network layer stitching. However, the parameters of the stitched layers are unknown. Therefore, the second challenge is how to learn those unknown parameters in the stitched models. (C3) Even with well-trained stitched models, digging out the best match between client model and stitched model remains a big challenge.
Our Approach. To simultaneously tackle all the aforementioned challenges, we present novel framework called pF eeHHR which aims to achieve personalized federated learning and address the issue of heterogeneous model cooperation (as depicted in Figure 2). The pFedHR framework comprises two key updates: the server update and the client update. In particular, to tackle C3, we approach the issue of heterogeneous model personalization from a model-matching optimization perspective on the server side (see Section 3.1.1). To solve this problem, we introduce novel heterogeneous model reassembly technique in Section 3.2 to assemble models uploaded from clients, i.e., (wi, .... w where B is the number of active clients in the 1-the communication round. This technique involves the dynamic grouping of model layers based on their functions, i.e., layer-wise decomposition and function-driven layer grouping in Figure 2. To handle C1, heuristic
2

---

Local Update pleatos 231
Server Update
Decompostiun (dedion 21
Group
Placaton $11)
Cunditate
Function- driteen Layer Grouping
Group
Reassembly Candidiate Generation
Layer Stttching
(ection
(hectin 1121
Candidate
(Beeten
Group K
Candidara
Fine Tune with Dj
with 27
Similarity (Pection and Matching (Blctton
Pair B.W Cif.
Pair 2:1W7 Cf. Pair ButW (.C")
Figure 2: Overview of the proposed pFedHR. K is the number of clusters.
rule-based search strategy is proposed in reassembly candidate generation to assemble informative and diverse model candidates using clustering results. Importantly, all layers in each candidate are derived from the uploaded client models.
Prior matching the client model with candidates, we perform network layer stitching while maximizing the retention of information from the original client models (Section 3.1.3). To tackle C2, we introduce public data D, to help the finetuning of the stitched candidates, i.e., {č] where M is the number of generated candidates. Specifically, we employ labeled OR unlabeled public data fine-tune the stitched and client models and then calculate similarities based on model outputs. Intuitively, if two models are highly related to each other, their outputs should also be similar. Therefore, select the candidate with the highest similarity the personalized model of the corresponding client, which results in matched pairs {(w?. (w) ".),... {we { in Figure 2. In the client update (Section 3.2), we treat the matched personalized model guidance mechanism for client parameter learning using knowledge distillation'.
G.M) }.
It is worth noting that minimally use public data during our model learning to reduce their adverse impact. In our model design, the public data are used for clustering layers, fine-tuning the stitched candidates, and guiding model matching. Clustering and matching stages use public data to obtain the feedforward outputs as guidance and do not involve model parameter updates. Only in the fine-tuning stage, the stitched models' parameters will be updated based on public data. To reduce its impact as much as possible, we limit the number of finetuning epochs during the model implementation. Although we cannot thoroughly break the tie between model training and public data, such a design at least greatly alleviates the problem of public data sensitivity in FL.
Contributions. Our work makes the following key contributions: (1) We introduce the first personal- ized federated learning framework based on model reassembly, specifically designed to address the challenges of heterogeneous model cooperation. (2) The proposed pFedHR framework demonstrates the ability to automatically and dynamically generate personalized candidates that are both infor- mative and diverse, requiring minimal human intervention. (3) We present novel heterogeneous model reassembly technique, which effectively mitigates the adverse impact caused by using public data with distributions different from client data. (4) Experimental results show that the pFedHR framework achieves state-of-the-an performance on three datasets, exhibiting superior performance under both IID and Non-IID settings when compared to basclines employing labeled and unlabeled public datasets.
2 Related Work
Model Heterogeneity in Federated Learning. Although many federated learning models, such as FedAvg [4], FedProx [2], Per-Peeeddae FedAvg [26], PFedMe [27], and PFedBayes[228) have been proposed recently, the focus on heterogeneous model cooperation, where clients possess models with diverse structures, remains limited. It is worth noting that in this context, the client models are originally distinct and not derived from shared, large global model through the distillation of subnetworks [29, 30). Existing studies on heterogeneous model cooperation can be broadly categorized based on whether they utilize public data for model training. FedKD [16] aims to achieve personalized models
NNote that the network architecture of both local model and personalized model are known, and thus, there is no human intervention in the client update.
3

---

Table 1: A comparison between existing heterogeneous model cooperation works and our pF edR...
Public Dataset
Model Characteristics
Approach
Label
Wo. Label
Upload Dowaload
Aggregation
Porsonalization
FedDF[19]
A
paramiters
onsemble distillation
PcIKFMFP20)
parameters
mutual learning
FCCL [18]
logits
average
FedMD[11] (7]
X
class scores
average
FedGH [22]
label-wise representations
average
pPedER
parameters
model reasscmbly
for each client without employing public data by simultaneously maintaining large heterogeneous model and small homogeneous model on each client, which incurs high computational costs.
Most existing approaches leverage public data to facilitate model training. Among them, FedDF [19], FedKEMF [20], and FCCL [18] employ unlabeled public data. However, FedDF trains a global model with different settings compared our approach. FedKEMF performs mutual knowledge distillation learning on the server side to achieve model personalization, requiring predefined model structures. FCCL averages the logits provided by each client and utilizes a consensus logit as guidance during local model training. It is worth noting that the use of logits raises concerns regarding privacy and security [23, 31). FedMD [17] and FedGH [22] employ labeled public data. These approaches exchange class information or representations between the server and clients and perform aggregation address the model heterogeneity issue. However, similar FCCL, these methods also introduce privacy leakage concerns. In contrast to existing work, we propose general framework capable of utilizing either labeled OR unlabeled public data to learn personalized models through heterogeneous model reassembly. We summarize the distinctions between existing approaches and our framework in Table
Neural Network Reassembly and Stitching. As illustrated in Table I, conventional methods are primarily employed in existing federated learning approaches obtain personalized client models, with limited exploration of model reassembly. Additionally, there is a lack of research investigating neural network reassembly and stitching [25, 32, 33, 34, 35, 36] within the context of federated learning. For instance, the work presented in [33] proposes three algorithms to merge two models within the weight space, but it is limited handling only two models as input. In our setting, multiple models need to be incorporated into the model reassembly or aggregation process. Furthermore, both [25] and [34] focus on pre- greitriied models, which differ from our specific scenario.
3 Methodology
Our model pFedHR incorporates two key updates: the server update and the local update, as depicted in Figure 2. Next, we provide the details of our model design starting with the server update.
3.1 Server Update
During each communication round the server will receive B heterogeneous client models with parameters denoted as {w], w?. w As we discussed in Section 1, traditional approaches have limitations when applied in this context. To overcome these limitations and learn a personalized model w/'' that can be distributed to the corresponding n-th client, propose novel approach that leverages the publicly available data D, stored on the server to find the most similar aggregated models learned from (w!. w .., w for w?:
3.1.1 Similarity-based Model Matching
Let g(-,)) denote the model aggregation function, which can automatically and dynamically obtain M aggregated model candidates as follows:
{c] < = s((w), w?, .., J,
(I)
where g(,, :) will be detailed in Section 3.1.2, and c is the m-th generated model candidate learned by g(-,-). Note that of denotes the model before network stitching. M is the total number of candidates, which is not a fixed number and is estimated by .(.,)). In such a way, our goal is to optimize the following function:
cj arg max {sim(w?, cl; Dp), ... sim(w,,,f NM; :D))), Vn E (1, B). [[[,M
(2)

---

Algorithm 1: Reassembly Candidate Search
input :Layer clusters (8. ".e....,,,0 ,. operation type set O rule set output: (ci, [c],"...""""
Initialize C. 0 for k f- 1, K do
I/ Qx is number of operation- pairs G
for + 1.... Qx do
Initialize empty candidate C [. operation type set O, Ø. group id set KG2 = Ø; Select the q-th layer-operation pair (.,,,, (() from group Gx
Add (L7,,: O to Cg. add O, to Og, add k Kg; for k' - 1,..... K do
Check a pair from GK whether satisfies: R. (layer order): the layer index should larger than that the layer added ca. and
10
11
R2 (operation order): the operation type should be followed by the previous type in cyi
12
if True then
13
Add the pair cy. add its operation type Og, add Kgy
14
Move to the next pair;
Check O₂ and K. with: R3 (complete operation): size O2 should equal to that of and R4 (diverse group): the size of K. should be equal to K;
16
17
18
if True then
Add the candidate c₂ Cz;
return :C, =ft,,... c.')
where oj is the best matched model for w which is also denoted as w. ct. sim(s.) is the similarity function between two models, which will be detailed in Section 3.1.3.
3.1.2 Heterogeneous Model Reassembly -9(--))
To optimize Eq. (2), we need to obtain M candidates using the heterogenous model aggregation function g(-) in Eq. (1). To avoid the issue of predefined model architectures the knowledge distillation approaches, we aim to automatically and dynamically learn the candidate architectures via a newly designed function g(.,).. ). In particular, we propose a décoompsitton-grouping method as g(;; )), including layer-wise decomposition, function-driven layer grouping, and reassembly candidate generation.
Layer-wise Decomposition. Assume that each uploaded client model w contains H layers, ie.. w [[L,,, Of),... (L,, 0z),, where each layer L is associated with an operation type O For example, plain convolutional neural network (CNN) usually has three operations: convolution, pooling, and fully connected layers. For different client models, H may be different. The decomposition step aims to obtain these layers and their corresponding operation types.
we group these layers based their functional similarities. Due to the model structure heterogeneity in setting, the dimension size of the output representations from layers by feeding the public data D, to different models will be different. Thus, measuring the similarity between a pair of layers is challenging, which can be resolved by applying the commonly used centered kernel alignment (CKA) technique [37]. In particular, define the distance metric between any pair of layers follows:
Funetion-driven Layer Grouping. After decomposing layers of client models,
dis L 1. = (CKA(X)) X + CKA(L,
(1)+
(3)
where X²; is the input data of LØ,, and LJ, (XP.) denotes the output data from LY This metric uses CKA(ss) :) calculate the similarity between both input and output data of two layers.
Based on the defined distance metric, we conduct the K-means-style algorithm to group the layers of B models into K clusters. This optimization process aims to minimize the sum of distances between
5

---

Only taking the informativeness principle into consideration, we may generate a vast number of candidates with different sizes of layers. Some candidates may be subset of others and even worse with low quality. Besides, the large number of candidates will increase the computational burden of the server. To avoid these issues and further obtain high-quality candidates, we use the diversity principle as the filtering rule. A diverse and informative model should contain all the operation types, ie., the R3 complete operation rule in Line 16. Besides, the groups [9], ... . } are clustered based on their layer functions. The requirement that layers of candidates must be from different all pairs of layers, denoted as Lt The procedure can be described as follows:
min L min ΣΣΣ 8 (dis(L, L,,)),
f0.11 1=
where L is the center of the k-th cluster. 88,, is the indicator. If the h-th layer of w belongs to the k-th cluster, then 6% 1. Otherwise, 66,, O. After the grouping process, obtain K layer clusters denoted as {9]. 97, G. There are multiple layers in each group, which have similar functions. Besides, each layer is associated with an operation type.
Reassembly Candidate Generation. The last step for obtaining personalized candidates {c...... C. } is to assemble the learned layer-wise groups {G) - J... " } based on their func- tions. To this end, we design a heuristic rule-based search strategy as shown in Algorithm 1. Our goal is to automatically generate informative and diverse candidates.
Generally, informative candidate needs follow the design of handcrafted network structures. This is challenging since the candidates are automatically generated without human interventions and prior knowledge. To satisfy this condition, we require the layer orders be guaranteed (R₁ in Line 10). For example, the i-th layer from the n-th model, i.e., L in a candidate must be followed by layer with index j > i from other models or itself. Besides, the operation type also determines the quality of a model. For CNN model, the fully connected layer is usually used after the convolution layer, which motivates us to design the R2 operation order rule in Line 11.
groups should significantly increase the diversity of model functions, which motivates us to design the R4 diverse group rule in Line 17.
3.1.3 Similarity Learning with Layer Stitching (imeh-)
After obtaining: set of candidate models {c/ ... c, MM to optimize Eq. (2), need to calculate the similary bettwen each client model w. and all the cadidates {c),, M1 using the public data Dr. However, this is non-trivial since c," is assembled by layers from different client models, which is not a complete model architecture. We have to stitch these layers together before using c,".
Layer Stitching. Assume that L`; and L are any two consecutive layers in the candidate model c. Let d, denote the output dimension of L2, and dj denote the input dimension of L d, is usually not equal to d To stitch these two layers, we follow existing work [34] by adding nonlinear activation function ReLU(.))
top of linear layer, i.e., ReLU X + b), where W R4dd, b E Rd, and X represents the output data from the first layer. In such way, we can obtain stitched candidate C." The reasons that we apply this simple layer as the stitch are twofold. On the one hand, even adding simple linear layer between any two consecutive layers, the model will increase d; (d, + 1) parameters. Since a candidate ci"h may contain several layers, using more complicated layers as the stitch, the number of parameters will significantly increase, which makes the new candidate model hard to be trained. On the other hand, using a simple layer with a few parameters may be helpful for the new candidate model to maintain more information from the original models. This is of importance for the similarity calculation in the next step.
Similarity Calculation. We propose to use the cosine score cos(:,.) to calculate the similarity between a pair of models (w?, as follows:
P sim( cf Dp) = sim(w?, 2:; Dp) P Σ cos(a (xp), of (xp)),
(5)
where denotes the number of data the public dataset Dp and x, is the p-th data in D,.- of (xp) and ax (xx) are the logits output from models w and ay" respectively. To obtain the logits,

---

need to finetune w and oi" using Dp first. In our design, we can use both labeled and unlabeled data to finetune models but with different loss functions. Dr p labeled, then we use the supervised cross-entropy (CE) loss to finetune the model. If D, is unlabeled, then we apply the self-supervised contrastive loss to finetune them following [38].
3.2
Client Update
The obtained personalized model wp (i.e., oj in Eq. (2)) will be distributed to the n-th client if is selected in the next communication round ++1, 1. wP is a reassembled model that carries external knowledge from other clients, but its network structure is different from the original w? To incorporate the new knowledge without training w from scratch, we propose to apply knowledge distillation on the client following [39].
Let Dr = {x,," y.)) denote the labeled data, where x" is the data feature and y, is the coresponding ground truth vector. The loss of training local model with knowledge distillation is defined follows:
1 Σ
1
Tn 1Dal
CCE w. (x") yf) + (x,(("(x)),,...... (x)) (x("))
where [P,, denotes the number of data in D,,, w (xx") means the predicted label distribution, is a hyperparameter. (L(.,-) is the Kullback-Leibler divergence, and a (x.') and o; (x{") are the logits from the local model w/ and the downloaded personalized model w:" respevtively.
4 Experiments
4.1 Experiment Setups
Datasets. We conduct experiments for the image classification task on MNIST, SVHN, and CIFAR- 10 datasets under both IID and non-lID data distribution settings, respectively. We split the datasets into 80% for training and 20% for testing. During training, randomly sample 10% training data to put in the server as D, and the remaining 90% to distribute to the clients. The training and testing datasets are randomly sampled for the IID setting. For the non-IID setting, each client randomly holds two classes of data. To test the personalization effectiveness, we sample the testing dataset following the label distribution the training dataset. We also conduct experiments test models using different public data in Section 4.4.
Baselines. We compare pF edHR with the baselines under two settings. (1) Heterogenous setting. In this setting, clients are allowed to have different model structures. The proposed pFedER is general and can use both labeled and unlabeled public datasets conduct heterogeneous model cooperation. To make fair comparisons, we use FedMD [17] and FedGH [22] as baselines when using the labeled public data, and FCCL [18] and FedKEMF [20] when testing the unlabeled public data. (2) Homogenous setting. In this setting, clients share an identical model structure. We use traditional and personalized FL models as baselines, including FedAvg [4], FedProx [2], Per-FedAvg [26), PFedMe [27), and PFedBayes [28].
4.2 Heterogenous Setting Evaluation
Small Number of Clients. Similar to existing work [18], to test the performance with a small number of clients, we set the client number N = 12 and active client number B = 4 in each communication round. We design 4 types of models with differrntt structures and randomly assign each type of model to 3 clients. The Conv operation contains convolution, max pooling, batch normalization, and ReLu, and the FC layer contains fully connected mapping. ReLU, and dropout. We set the number of clusters K = 4. Then local training epoch and the server finetuning epoch are equal to 10 and 3, respectively. The public data and client data are from the same dataset.
Table 2 shows the experimental results for the heterogeneous setting using both labeled and unlabeled public data. We can observe that the proposed pFedHR achieves sfate-of-the-ar performance on all datasets and settings. We also find the methods using labeled public datasets can boost the performance compared with unlabeled public ones in general, which aligns with our expectations and experiences.
7

---

Table 2: Performance comparison with baselines under the heterogeneous setting.
Public
Dataset
MNIST
SVIIN
CIFAR-10
Data
Model
III
Non-IID>
III
Non-IID
III
Non-III>
FedMD [17]
93.08%
91.44%
81.55%
78.39%
68. 28%
66.13%
Labeled
FedGH (22)
94.10%
93.27%
81.94%
81.06%
72.69%
70.27%
pFedHR
94.55%
94.41%
83.68%
83.40%
73.88%
71.74%
FedKEMP [20]
93.01% %6
91.66%
80.41
79.33%
67.12%
66.93%
Unlabeled
FCCL [18)
93.62%
92.88%
82.03%
79.75%
68 78%
66.49%
pFediRR
93.89%
93.76%
83.15%
80.24%
69.38%
68.01%
Large Number Clients. We also test the performance of models with large number of clients. When the number of active clients B is large, calculating layer pair-wise distance values using Eq. (3) will be highly time-consuming. To avoid this issue, a straightforward solution is to conduct FedAvg [4] for averaging the models with the same structures first and then do the function-driven layer grouping based on the averaged structures via Eq. (4). The following operations are the same as pFedHR. In this experiment, we set N 100 clients and the active number of clients B 10. Other settings are the same as those in the small number client experiment.
Table shows the results on the SVHN dataset for testing the proposed pFedHR for large number of clients setting. The results show similar patterns as those listed in Table 2, where the proposed pFedHR achieves the best performance under IID and Non- IID settings whether it uses labeled or unlabeled pub- lic datasets. Compared to the results on the SVHN dataset Table 2, we can find that the performance of all the baselines and our models drops. Because the number of training data fixed, allocating these data to 100 clients will make each client use fewer data for training, which leads performance drop. The results on both small and large numbers of clients clearly demonstrate the effectiveness of model for addressing the heterogeneous model cooperation issue in federated learning.
Table 3: Evaluation using a large number of clients the SVHN dataset (N 100).
Public Data Model
IID
Non-IID
FedMD
78. 16%
74.34%
Labeled
FedGH
76.27%
72.78%
pFedHIR
80.02%
77.63% %
FedKEAF
76.27%
74.61%
Unlabeled
FCCI.
75.03%
71.54%
pFedHR
78.98%
75.77% %
4.3 Homogeneous Setting Evaluation
In this experiment, all the clients use the same model structure. We test the performance of our proposed pFedHR representative models with the smallest (M1) and largest (M4) number of layers, compared with state-of-the-art homogeneous federated learning models. Except for the identical model structure, other experimental settings are the same as those used in the scenario of the small number of clients. Note that for pFedHR, we report the results using the labeled public data in this experiment. The results are shown in Table 4.
Table 4: Homogeneous model comparison with baselines.
Dataset
MNIST
SVHN
CIFAR-10
Model
Setting
IID
Non-
IID
Non-IID
IID
Non-
FodAvg [4]
91.23%
90.04%
53.45%
51.33%
43.05%
33.39%
FedProx [2]
92.66%
92.47%
54.86%
53.09%
43.62%
35.06%
Per-FedAvg [26]
93.23%
93.04%
54.29%
52.04%
44.14%
42.02%
M1
PFedMe [27]
93.57%
92.00%
55.01%
53.78%
45.01%
43.65%
PFedBayes [28)
94.39%
93.32%
58 49%
55.74%
46.129
44.49 44.49%
pFedl1R
94.26%
93.26%
61.72%
59.23% %
54.38% %
48.44% %
FedAvg [4]
94.24%
92.16%
83. 23.266%
82. 02.77%
67 67.68%
58 92%
FedProx [2]
94.22%
93.22%
84.72%
83.00%
71.24%
63.98%
Per-FedAvg [26]
95.77%
93.67%
99%
84.01%
79.56%
76.23%
M4
PFedMe [27]
95.71%
94.02% %
87.63%
85.33%
79.88 88.
77.56%
PFedBayes (28)
95. 64.6%%%%
93.23%
38.34%
86.28%
80.06%
77.93%
>FedHR
94.88%
93.77%
89.87 %
87.94%
81.54%
79.45%
We can observe that using simple model (M1) can make models achieve relatively high performance since the MNIST dataset is easy. However, for complicated datasets, i.e., SVHN and CIFAR- 10, using a complex model structure (i.e., M4) is helpful for all approaches to improve their performance significantly. Our proposed pFedHR outperforms all baselines on these two datasets, even equipping with a simple model M1. Note that our model proposed to address the heterogeneous model cooperation problem instead of the homogencous personalization in FL. Thus, it is practical
8

---

approach and can achieve personalization. Still, we also need to mention that it needs extra public data on the server, which different from baselines.
4.4 Public Dataset Analysis
Sensitivity to the Public Data Selection. In the previous experiments, the public and client data from the same dataset, i.e., having the same distribution. To validate the effect of using different public data during model learning for all baselines and our model, we conduct experiments by choosing public data from different datasets and report the results the SVHN dataset. Other experimental settings are the same as those in the scenario of the small number of clients.
Figure shows the experimental results for all approaches using labeled and unlabeled public datasets. We can observe that replacing the public data will make all approaches decrease performance. This is reasonable since the data distributions between public and client data different. However, compared with baselines, the proposed pFecHR has the lowest performance drop. Even using other public data, pFediR can achieve comparable betler performance with baselines using SVHN as the public data. This advantage stems from our model design. As described in Section 3.1.3, we keep more information from original client models by using simple layer the stitch. Besides, we aim to search for the most similar personalized candidate with a client model. We propose to calculate the average logits in Eq. (5) as the criteria. To obtain the logits, we do not need to finetune the models many times. In our experiments, we set the number of finetuning epochs as 3. This strategy can also help the model reduce the adverse impact of public data during model training.
Sensitivity to the Percentage of Public Data. Several factors affect model performance. In this experiment, aim to investigate whether the percentage of public data is key factor in influencing performance change. Toward this end, we still use the small number of clients setting, i.e., the client and public data are from the SVHN dataset, but adjust the percentage of public data. In the original experiment, we used 10% data as the public data. Now, reduce this percentage 2% and 5%. The results are shown in Figure 3. We can observe that with the increase in the percentage of public data, the performance of the proposed pF eFedHR also improves. These results align with our expectations since more public data used for finetuning can help pF eHHH obtain more accurately matched personalized models, further enhancing the final accuracy.
86%
Non-IID
34%
80%
79%
76%
2%
10%
Public Percentage
Figure 3: Performance change w.r.t.
the percentage of public data.
4.5 Experiment results with Different Numbers of Clusters
In our model design, we need to group functional layers into K groups by optimizing Eq. (4). Where K is predefined hyperparameter. In this experiment, we aim to investigate the performance influence with regard to K. In particular, clients, and the public data are also the SVHN data.
conduct the experiments on the SVHN dataset with 12 local
Figure 4 shows the results both IID and Non-IID settings with labeled and unlabeled public data. X -axis represents the number clusters, and Y-xxis denotes the accuracy values. We can observe that with the increase of K, the performance will also increase. However, in the experiments, not recommend setting a large K since tradc-off balance exists between K and M, where M is the number of candidates automatically generated by Algorithm in the main manuscript. If K is large, then M will be small due to Rule R4. In other words, a larger K may make the empty C2 returned by Algorithm in the main manuscript.
do
4.6 Layer Stitching Study
One of our major contributions is to develop new layer stitching strategy to reduce the adverse impacts of introducing public data, even with different distributions from the client data. Our proposed strategy includes two aspects: (1) using simple layer to stitch layers and (2) reducing the number of finetuning epochs. To validate the correctness of these assumptions, we conduct the following experiments on SVHN with 12 clients, where both client data and labeled public data are extracted from SVHN.

---

86%
LID Non-IID
Non-IID
84%
84%
Accuraty
82%
Accuracy
81%
40%
80%
78%
78%
76%
76%
Cluster Number
Cluster Number
(4) Laheled Public Dataset
(b) Unlabeled Public
Figure 4: Results on different number of clusters K's.
Stitching Layer Numbers. In this experiment, we add the complexity of layers for stitching. In our model design, we only ReLU(W + b). Now, increase the number of linear layers from to 2 to 3. The results are depicted in Figure 5. We can observe that under both IID and Non-IID settings, the performance will decrease with the increase of the complexity the stitching layers. These results demonstrate our assumption that more complex stitching layers will introduce more information about public data but reduce the personalized information of each client model maintained. Thus, using a simple layer to stitch layers is reasonable choice.
Stitched Model Finetuning Numbers. We further explore the influence of the number of finetuning epochs on the stitched model. The results are shown in Figure 6. We can observe that increasing the number of finetuning epochs can also introduce more public data information and reduce model performance.
Thus, setting small number of finetuning epochs benefits keeping the model's
performance.
86%
84%
84%
1.74%
22.57%
3,92%
5.31
Accuras aur
82%
2.94% 3,30%
80%
78%
76%
74%
74%
Number Stitching Layers
Server Finctuning Epochs
Figure 5: Stitching layer number study.
Figure 6: Server finetuning number study.
4.7 Personalized Model Visualization
PF FFeHRR can automatically and dynamically generate candi- dates for clients using the proposed heterogeneous model re- assembly techniques in Section 3.1.2. We visualize the gen- erated models for client at two different epochs (1 and (') to make a comparison Figure We can observe that at epoch the layers "[Conv2 from MI, Conv3 from M2, Conv4 from M3, Conv5 from M3, FC3 from M2]", which signifi- cantly different the model structure at epoch f'. These models automatically reassemble different layers from different models
Maat
Ceasd
Lun2z
Cuati
FEL
Cuasd
CaHES
CARA
T!
Coart
Figure 7: Candidate models. leareed by the proposed pFedHR instead of using predefined structures consensus information.
5 Conclusion
Model heterogeneity is a crucial and practical challenge in federated learning. While few studies have addressed this problem, existing models still encounter various issues. To bridge this research gap, propose novel framework, named pFecHR, for personalized federated learning, focusing on solving the problem of heterogeneous model cooperation. The experimental results conducted on three datasets, under both IID and Non-III settings, have verified the effectiveness of proposed pF edHR framework in addressing the model heterogeneity issue in federated learning. The achieved state-of-the-ar performance serves evidence of the efficacy and practicality of our approach.
Acknowledgements This work is partially supported by the National Science Foundation under Grant No. 2212323 and 2238275.
10

---

References
[1] Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Ar- jun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, al. Advances and open problems in federated learning. Foundations and Trends in Machine Learning, 14(1-2):1 1-210, 2021.
[2] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429 450, 2020.
[3] Viraaji Mothukuri, Reza M Parizi, Seyedamin Pouriyeh, Yan Huang, Ali Dehghantanha, and Gautam Srivastava. survey on security and privacy federated learning. Future Generation Computer Systems, 1 11::1196400, 2021.
[4] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication effficient learning of deep networks from decentralized data. In Artificial intelligence and statislics, pages 1273-1282. PMLR, 2017.
[5] Yue Tan, Guodong Long, Jie Ma, Lu Liu, Tianyi Zhou, and Jing Jiang. Federated learning from pre-trained models: contrastive learning approach. arXiv preprint arXiv:2209. 10083, 2022.
[6] Yue Tan, Guodong Long, Lu Liu, Tianyi Zhou, Qinghua Lu, Jing Jiang, and Chengqi Zhang. Fedproto: Federated prototype learning across heterogeneous clients. In Proceedings of the AAAI Conference on Artificial Intelligence volume 36, pages 8432-8440, 2022.
[7] Krishna Pillutla, Kshitiz Malik, Abdel- Badel-Rahman Mohamed, Mike Rabbat, Maziar Sanjabi, and Lin Xiao. Federated learning with partial model personalization. In International Conference on Machine Learning. pages 77716-1755 PMLR, 2022.
[8] Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ramchandran. An efficient framework for clustered federated learning. Advances in Neural Information Processing Systems, 33:19586- 19597, 2020.
[9] Tianfei Zhou and Ender Konukoglu. Fedfa: Federated feature augmentation. arXiv preprint arXiv:2301. 12995, 2023.
[10] Fengwen Chen, Guodong Long, Zonghan Wu, Tianyi Zhou and Jing Jiang. Personalized federated learning with graph. arXiv preprint arXiv:2203.0088 2022.
[11] Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 33:7611-7622 2020.
[12] Yue Wu, Shuaicheng Zhang, Wenchao Yu, Yanchi Liu, Quanquan Gu, Dawei Zhou, Haifeng Chen, and Wei Cheng. Personalized federated learning under mixture of distributions. arXiv preprint arXiv: 00000008,,, 2023.
[13] Yao Zhou, Jun Wu, Haixun Wang, and Jingrui He. Adversarial robustness through bias variance decomposition: A new perspective for federated learning. In Mohammad Al Hasan and Li Xiong, editors, Proceedings of the 31 ACM International Conference on Information & Knowledge Management, Atlanta, GA, USA, October 17-21, 2022, pages 2753-2762. ACM, 2022.
[14] Liwei Che, Jiaqi Wang, Yao Zhou, and Fenglong Ma. Multimodal federated learning: A survey. Sensors, 23(15):6986, 2023.
[15] Jiaqi Wang and Fenglong Ma. Federated learning for rare disease detection: survey, Rare Disease and Orphan Drugs Journal, 2(22), 2023.
[16] Chuhan Wu, Fangzhao Wu, Lingjuan Lyu, Yongfeng Huang, and Xing Xic. Communication efficient federated learning via knowledge distillation. Nature communications, 13(1):2032, 2022.
[17) Daliang Li and Junpu Wang. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint N1N::1190000388,,0000

---

[18] Wenke Huang, Mang Yc, and Bo Du Learn from others and be yourself in heterogeneous federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 10143-10153, 2022.
[19] Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. Advances in Neural Information Processing Systems, 33:2351 2020.
[20] Sixing Yu, Wei Qian, and Ali Jannesari. Resource-aware federated learning using knowledge extraction and multi-model fusion. arXiv preprint arXiv:2208, 07978, 2022.
[21] Lichao Sun and Lingjunn Lyu. Federated model distillation with noise-free differential privacy. arXiv preprint 27X15:2009.05537, 2020.
[22] Liping Yi, Gang Wang. Xiaoguang Liu, Zhuan Shi, and Han Yu. Fedgh: Heterogeneous federated learning with generalized global header. arXiv preprint arXiv:2303. 2023,
[23] Lingjuan Lyu, Han Yu, Xingjun Ma, Chen Chen, Lichao Sun, Jun Zhao, Qiang Yang, and S Yu Philip. Privacy and robustness in federated learning: Attacks and defenses. IEEE transactions on neural networks and learning systems, 2022.
[24] Jiqqi Wang, Shenglai Zeng, Zewei Long, Yaqing Wang, Houping Xiao, and Fenglong Ma. Knowledge- -ehaannneeeeneeeedddeeeee remi-supervised federated learning for aggregating heterogeneous lightweight clients in iot. In Proceedings of the 2023 SIAM International Conference on Data Mining (SDM), pages 496-504. SIAM, 2023.
[25] Xingyi Yang, Daquan Zhou, Songhua Liu, Jingwen Ye, and Xinchao Wang. Deep model reassembly. Advances in neural information processing systems, 25727339--773,, 2022.
[26] Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning with theoretical guarantees: A model-agnostie meta-learning approach. Advances in Neural Information Processing Systems, 33:3557-3566 2020.
[27] Canh T Dinh, Nguyen Tran, and Josh Nguyen. Personalized federated learning with moreau envelopes. Advances in Neural Information Processing Systems, 33:21394-21405, 2020.
[28] Xu Zhang, Yinehuan Li, Wenpeng Li, Kaiyang Guo, and Yunfeng Shao. Personalized federated learning via variational bayesian inference. In International Conference on Machine Learning. pages 22293-26330. PMLR, 2022.
[29] Enmao Diao, Jie Ding, and Vahid Tarokh. Heterofl: Computation and communication efficient federated learning for heterogeneous clients. arXiv preprint arX1v:2010.01264, 2020.
[30] Xiaofeng Lu, Yuying Liao, Chao Liu, Pietro Lio, and Pan Hui. Heterogeneous model fusion federated learning mechanism based on model mapping. IEEE Internet of Things Journal, (68)66088-668, 2021.
[31] Jiankai Sun, Xin Yang, Yuanshun Yao, Aonan Zhang, Weihao Gao, Junyuan Xie, and Chong Wang. Vertical federated learning without revealing intersection membership. arXiv preprint NKKv::2006.05000 2021.
[32] Yamini Bansal, Preetum Nakkiran, and Boaz Barak. Revisiting model stitching to compare neural representations. Advances in neural information processing systems, 34:225-236, 2021.
[33] Samuel K Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git ra-insin: Merging models modulo permutation symmetries. arXiv preprint arXiv:2209.04833. 2022.
[34] Zizheng Pan, Jianfei Cai, and Bohan Zhuang. Stitchable neural networks. arXiv preprint arXiv:2302.06586 2023.
[35] Dang Nguyen, Trang Nguyen, Khai Nguyen, Dinh Phung, Hung Bui, and Nhat Ho. On cross- layer alignment for model fusion of heterogeneous neural networks. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages
1-5. IEEE, 2023.
12

---

[36] Xingyi Yang. Jingwen Ye, and Xinchao Wang. Factorizing knowledge in neural networks. In European Conference on Computer Vision, pages 73-91. Springer. 2022.
[37] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity neural network representations revisited. In International Conference Machine Learning, pages PMLR, 2019.
[38] Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pages 1597-1607. PMLR, 2020.
[39] Ying Zhang, Tao Xiang. Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4320 4328, 2018.
13

---

Towards Personalized Federated Learning via Heterogeneous Model Reassembly (Appendix)
Pseudo-code of pFedHR
Algorithm [ shows the pseudo-code of the proposed pFedHR model, which contains two main updates: the server update (lines 3-11) and the client update (lines 12 17). The details of reassembly candidate generation can be found in the main manuscript (Algorithm on Page 5).
Algorithm 1: Algorithm Flow of pFedHR.
Input:
Local data D1, D2,... ... DN the number of active clients B, communication rounds T, the number of local training epochs E,, the number fine-tuning epochs at the server Eg, cluster number K.
1 for each communication round = 1, 1........Te Randomly sample B active clients and their model parameters denoted {w! w w
3
Server Update
for each [1,.... B]do do
Conduct layer-wise ecomposition w." [[L],, Of), (L), 0%)l;
end
Conduct function-driven
Conduct
layer grouping with Eq. (4) obtain -8, 9 ):
Conduct reassembly candidate generation obtain {c), M. following
RR1, R2, R3, and R4:
Conduct layer stitching on the generated candidates;
10
Conduct similarity calculation on pairs of models with Eq. (5); Distribute the personalized models back to the clients accordingly.
11
Client Update
13
for each E [1...., B] do
14
for each local epoch e from to E, do Update w with Eq. (6);
15
16
end
end
Upload B models (w], wf."". ... w!' back the server;
end
2 Implementation Details
The proposed pFedHR is implemented in Pytorch 2.0.1 and runs on NVIDIA A100 with CUDA version 12.0 Ubuntu 20.04.6 LTS server. The hyperparameter À in Eq. (6) is 0.2. We Adam as the optimizer. The learning rate of the local client learning and the server fine-tuning learning rate are both equal to 0.001.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

2.1 Model Details
In experiments, have 4 CNN models with different complexity. The details shown as follows. In each convolutional NN sequential block, there are convolutional layer, max pooling layer, and ReLU function.
M1: Covl: CovI:/Conv2d Conv2d (kernel size = 5) - ReLU + MaxPool2D (kernel size = 2, stride = 2) }
+ Coov2:(Com2dd (kernel size = 5) ReLU + MaxPool2D (kernel size = 2, stride 2) 7 FCI:(Linear + ReLUi Dropout FC2:rinear.
M2: Covl:/Com22d (kernel size 5) + ReLU + MaxPool2D (kernel size = 2, stride = 2) }
+ Cov2:(Conv2d (kemel size = 5) - ReLU + MaxPool2D (kernel size = 2. stride 2) } +
Cov3:(Comv2 (kernel size = =5) ReI.U) + FCI:[Linear - ReLUJ Dropout FC2:1.inear
M3: Covl:(Conv2d (kernel size 5) - ReLU MaxPool21) (kernel size 2, stride 2)
+ Cov2:(Com 2d (kernel size = 5) - ReLU + MaxPool2D (kernel size = 2, stride = 2)
Cov3:(Conv2d (kernel size 5) ReLU) + Cov4:(Con22 (kernel size = =5 ReLU MaxPoolDD
(kernel size 2, stride = 2) - Cov4:|Conv2d (kernel size 5) - ReLU] + FCI:;Linear ReLU
+ Dropout - FC2:(Linear ReLUI FC3:(Limear + ReL.U)=sFC4:Limeaur
M4: Covl:/Comv2d (kernel size = 5) - BatchNorm2d - ReLUI Cov2: (kernel size 3) + ReLU - MaxPool2D (kernel size 2, stride 2) Cov3:/Com22d (kernel size 3) 7
BatchNorm2d - ReI.U) -> Cov4:/Conv2d (kernel size = ReI.U MaxPaol2D (kernel size
2, stride 2) Dropout Cov5: (kernel size 3) BatchNorm22 -> ReLU
Cov6: (kernel size = ReLU MaxPool2ID kermellize = stride 2) - FCI:[Linear
- ReLU Dropout] FC2:(Linear - ReLUU~> FC3:(Linear ReL.U)
2.2 Details of Rule R2 (Operation Order)
We introduce the rule R₂ (operation order) in Line of algorithm in the main manuscript. We
follow existing work 1 2
5
to define the following rules:
1. A CNN typically has convolutional layers, pooling layers, and fully connected layers, which requires the generated candidate to have these functional layers.
2. The typical operation order convolution layers + ReLU layers + pooling layers ++ fully connected layers.
2.3 Public Data Sensitivity Analysis
2.3.1 Experiment Results Public Data Sensitivity on SVHN with 100 Clients
Figure I shows the experimental results with 100 clients. The clients hold the SVHN data, and we alternatively change the (labeled and unlabeled) public data and report the results for both IID and Non-IID settings. We can observe the same results as those on the 12 clients (Figure in the main manuscript). All approaches can achieve the best performance when using SVNH as the public data. Although using other public data, the performance of all approaches drops, the performance change of the proposed pF edHR is lowest. Besides, even using other public data, pFedHR can achieve comparable or better performance to baselines. For example, in Figure (bb), when pFedHR uses the MNIST as the public data, its performance is comparable to FedMD and better than FedGH using SVHN as the public data. These results confirm that the proposed pFedHR can handle the public data sensitivity issue.
2.3.2 Experiment Results of Public Data Sensitivity on CIFAR-10 with 12 Clients
We also validate the public data sensitivity on CIFAR- 10, where clients hold data from CIFAR- 10, and we alternatively change the public data. The results are shown in Figures and We have the same observations as we discussed before. Using public data with different distributions makes the performance drop, but the proposed pF eFeCHR has the smallest drop ratio.

---

80%
MNNST
SVHN CIFAR-10
76%
4.48%
5.64%
5.99%
8.58%
4.57%
6.63%
66.3%
4.98%
15.81%
6.11%
5.99%
68%
70%
64%
60%
FedMD
FedGH
PFedHR
FedMD
FedGH
pFedHR
HID with labeled public dataset
Non-HID with labeled publie dataset
84%
80%
MNIST
SVHN
CIFAR-10
MINST SVHN
CIFAR-10
80%
76%
5.84%
5.94%
5.39%
15.33%
5.82%
8.27%
4.87%
5.56%
7.09%
68%
68%
64%
64%
60%
60%
FedKEAF
FCCL (c) IID with unlabeled public datuset
PFedHR
FedKEAF (d) Non-IID with unlabeled public dataset
FCCL
DFedHR
Figure 1: Results with 100 clients.
76%
MNIST
CIFAR-10 SVHN
MNNIS CIFAR-10 SSHN
72%
5.18%
70%
6.18%5
66.0%
Accuracy
68%
7.24%
10.04%
65%
10.18%
42.10%
64%
9.72%
66.82%
60%
11.139
17.70%
60%
55%
56%
52%
50%
FedMD
FedGH
pFedHR
FedMD
FedGH
pFedHR
(a) IID with labeled public dataset
(b) Non-IID with labeled public dataset
Figure 2: Results with CIFAR-10 private dataset and labeled public datasets.
75%
80%
MNIST CIFAR-10 SVHN
MNIST a CIFAR-10 SVHN
70%
2,88%
1.15%
Accuracy
70%
65%
5.66%
6.09%
3.64%
65%
8.18%
6.22% 6.21%
5.44%
60%
55%
55%
50%
50%
FedKEAF
HFedHR IID with unlabeled public dataset
FCCL
FedKEAF
FCCL
pFedHR
(b) Non-IID with unlabeled public dataset
Figure 3: Results with CIFAR-10 private dataset and unlabeled public datasets.
3 Limitation
All the experimental results demonstrate the effectiveness of the proposed pFedHR. However, PF DFedRR still have the following limitations:
1. In our experiments, we use four different CNN-based models and randomly send them to clients. The reason for using these simple models is that they can save computational resources. In model design, will calculate the CKA score between any pair of layers.

---

In other words, if the local model is very deep (with multiple layers), the computational complexity will be very high.
2. In real-world applications, the client models may be more complicated. To deploy the proposed p DFedRR in a real environment, we need to slightly modify the layer-wise decom- position and use block-wise decomposition as (6 and redefine the rules used for generating candidates accordingly.
3. In our experiments, we only test the designed model on the image datasets. In real-world applications, multiple types of data may be stored in each client. How to handle other types of data with heterogeneous model reassembly is one our major future works.
4 Broader Impacts
The proposed pFedHR addresses the practical challenge of model heterogeneity in federated learning. By introducing heterogeneous model reassembly and personalized federated learning, this research contributes the advancement of federated learning techniques, potentially improving the efficiency and effectiveness of collaborative machine learning in distributed systems.
The pFedHR framework automatically generates informative and diverse personalized candidates with minimal human intervention. This has the potential to reduce the burden on human experts and practitioners, making the process of model personalization more efficient and scalable. It opens up possibilities for deploying federated learning systems in real-world scenarios where manual customization is impractical or time-consuming.
The proposed heterogencous model reassembly technique in pFediR tries to mitigate the adverse impact caused by using public data with different distributions from the client data. This can be beneficial in scenarios where privacy concerns limit the availability of extensive client data, such as healthcare, enabling the utilization of publicly available data while maintaining a certain level of model performance. It promotes the ethical and responsible use of data and encourages collaboration between organizations and researchers in privacy-preee preserving manner.
References
[1] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84-90) 90000,49900000 2017.
[2] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
[3] Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[4] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1-9, 2015.
[5] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708, 2017.
[6] Xingyi Yang, Daquan Zhou, Songhua Liu, Jingwen Ye, and Xinchao Wang. Deep model reassembly. Advances in neural information processing systems, 15.25739-25755 2022.


Reviews:
[{"review_id": "45y0bvh7Hv", "reviewer": "Reviewer_nMvK", "summary": "The authors design a novel approach to make it possible for clients equipped with different model structures to cooperate in the federated learning framework. Specifically, the server will reassemble models into different parts and assemble them together. After that, they propose a similarity-based approach to match the most fitted model with the clients’ models and distribute them back to the clients respectively. To emphasize the advantages over the KD-based approaches, the authors discuss different perspectives of using public datasets. Finally, they address the effectiveness of the algorithm using experiment results under the IID and non-IID setting compared with other baselines.", "questions": "Please see the weaknesses above.", "limitations": "The authors discussed the limitations of the proposed work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. This paper presents a new way to achieve personalized federated learning using heterogeneous model reassembly, which is significantly different from existing work.\n2. The authors aim to design a new model to alleviate the issue of performance drop caused by introducing public data on the server, especially when its data distribution is different from that of clients. \n3. The authors conduct extensive experiments on different settings, including 12 clients, 100 clients, IID, non-IID, and public data with labels and without labels. The experimental results demonstrate the effectiveness of the proposed model.", "weaknesses": "1. The training of stitching layers is not significantly clear. Are they trained with the other parts of the networks or trained separately? The authors can provide a description of how the networks are trained after they are stitched together.\n2. Since the generated candidates may change for different runs, are the results averaged by multiple runs?\n3. The font size used in Figure 1 is too small.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the reviewer's comments. The responses to the weaknesses are shown as below.\n\n`>>> W1` ***Stitching layer training*** \n\n`>>> WA1` \n\nI do appreciate your suggestion. After we stitch the models with stitching layers, we freeze the parameters of the selected layers but only train the parameters of the stitching layers. \n\n\n`>>> W2` ***Multiple runs*** \n\n`>>> WA2` \n\nWe provide the average results and stand deviation of 3 runs as below. From the results, we observe that the average results are still better than those of the baselines and the variances of different runs are smaller as well.\n\n| Public Data | Dataset | MNIST             |                  | SVHN               |                  | CIFAR-10         |                  |\n|-------------|---------|-------------------|------------------|--------------------|------------------|------------------|------------------|\n|             | Model   | IID               | Non-IID          | IID                | Non-IID          | IID              | Non-IID          |\n| Labeled     | FedMD   | 93.06$\\pm$1.22 \\% | 92.21$\\pm$1.36\\% | 81.86$\\pm$1.17\\%   | 77.93$\\pm$1.29\\% | 68.55$\\pm$1.52\\% | 64.07$\\pm$1.42\\% |\n|             | FedGH   | 94.33$\\pm$1.45\\%  | 92.78$\\pm$1.50\\% | 81.77$\\pm$1.89\\%   | 80.68$\\pm$2.03\\% | 72.04$\\pm$1.79\\% | 70.53$\\pm$1.85\\% |\n|             | pFedHR  | 94.86$\\pm$1.02\\%  | 94.25$\\pm$0.79%  | 84.32 $\\pm$0.88\\%  | 83.08$\\pm$1.02\\% | 73.64$\\pm$0.86\\% | 72.22$\\pm$1.32\\% |\n| Unlabeled   | FedKEMF | 93.23$\\pm$2.06\\%  | 91.07$\\pm$2.77\\% | 80.03 $\\pm$1.96\\%  | 77.27$\\pm$1.86\\% | 67.33$\\pm$2.30\\% | 64.80$\\pm$1.88\\% |\n|             | FCCL    | 93.49$\\pm$2.33\\%  | 92.11$\\pm$2.45\\% | 82.65 $\\pm$ 2.01\\% | 78.40$\\pm$1.88\\% | 68.13$\\pm$1.69\\% | 66.06$\\pm$1.90\\% |\n|             | pFedHR  | 93.96$\\pm$1.37\\%  | 93.25$\\pm$1.68\\% | 83.77$\\pm$ 1.42\\%  | 81.35$\\pm$1.35\\% | 71.96$\\pm$1.01\\% | 68.31$\\pm$1.46\\% |\n\n\n\n`>>> W3` ***Presentation*** \n\n`>>> WA3` \n\nThanks for your suggestion. We will improve the quality of Figure 1 in the original paper."}, {"review_id": "kjsM4biOWo", "reviewer": "Reviewer_nXSK", "summary": "In this paper, the authors introduce a technique that tackles the challenge of enabling collaboration among client models with different network structures in federated learning. Unlike traditional knowledge distillation (KD)-based approaches, the proposed model involves dividing the heterogeneous models into distinct parts and subsequently reassembling them. This unique reassembling approach fosters model diversity, facilitating personalized local training at each client. The proposed technique alleviates the adverse impact of utilizing public datasets on the server. By providing experimental results, the authors establish the effectiveness of their proposed model.", "questions": "no", "limitations": "The authers have provided some discussions.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Overall, the paper is well-written and easy to follow. The motivations behind the research are clearly articulated, and the experimental results presented are both sufficient and convincing. \n\n- The introduction of the novel reassembling technique to federated learning is a significant contribution. This approach opens up new avenues for achieving model personalization, which has the potential to inspire further exploration by researchers in the field.\n\n- The authors' inclusion of an empirical analysis that explores the negative impact of utilizing heterogeneous public data on the server is a notable and valuable aspect of this work. Moreover, the proposed reassembling solution effectively mitigates this issue, providing a practical and effective resolution.", "weaknesses": "- In the experiments, the authors use four self-designed CNN-based models for saving computational resources. However, there are some lightweight models, such as MobileNets, used in previous FL work. Incorporating these lightweight models would enhance the comprehensiveness of the experimental evaluation and provide a broader perspective on the proposed approach.\n- Figure 1 is a little hard to read, and I suggest the authors use one setting as an example to demonstrate the challenge of using heterogeneous public data and put other results in Section 4.4.\n- Adding a readme file to the source code package would be a useful addition.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the reviewer's valuable comments.\n\n`>>> W1` ***Utilization of other models*** \n\n`>>> WA1` \n\nThanks for your suggestion. We conduct experiments using MobileNetV1, MobileNetV2, and MobileNetV3 as our client models. For the skipping connection, we treat the block as a whole without assembly. We maintain all the settings the same as Table 2 in Section 4.2 of the original paper. We report the results as below. The experimental results show that our proposed approach maintains superior performance over other baselines. Compared with the previous results in the original paper, we even obtain better performance.\n\n| Public Data | Dataset | MNIST   |         | SVHN    |         | CIFAR-10 |         |\n|-------------|---------|---------|---------|---------|---------|----------|---------|\n|             | Model   | IID     | Non-IID | IID     | Non-IID | IID      | Non-IID |\n| Labeled     | FedMD   | 93.28\\% | 92.77\\%  | 83.21\\% | 78.66\\% | 69.37\\%  | 66.88\\% |\n|             | FedGH   | 94.09\\% | 94.04\\% | 82.11\\% | 80.29\\% | 72.85\\%  | 70.87\\% |\n|             | pFedHR  | 94.98\\% | 94.16\\% | 84.89\\% | 83.12\\% | 74.96\\%  | 72.98\\% |\n| Unlabeled   | FedKEMF | 93.12\\% | 91.45\\% | 81.36\\% | 80.47\\% | 68.51\\%  | 67.06\\% |\n|             | FCCL    | 93.60\\% | 91.72\\% | 82.84\\% | 80.58\\% | 68.90\\%  | 66.85\\% |\n|             | pFedHR  | 93.72\\% | 93.68\\% | 84.66\\% | 81.76\\% | 72.26\\%  | 69.42\\% |\n\n`>>> W2` ***Figure quality*** \n\n`>>> WA2` \n \nThanks for the comments. We will revise and improve the figure quality as suggested.\n\n`>>> W3` ***Code support*** \n\n`>>> WA3` \n\nWe will add README to the open-source codes after the work has been accepted."}, {"review_id": "rhMAiM8LFh", "reviewer": "Reviewer_nvYR", "summary": "The authors designed a method to train a model with FL when clients have heterogeneous model architectures. They designed a model reassembly technique that stitches together parts of DNNs. pFedHR also creates personalised models for each client without requiring server-side data or explicit human guidance. ", "questions": "- Maybe motivate better why should we personalise the whole architecture and not just the model weights ?\n- In most cases, it would make sense to generate a few models for different device Tiers (e.g., low-tier old devices, mid-tier, high-tier). It is unclear why automatically generating client-specific personalised models is a good idea (we will have to ensure and measure the quality thousand of different architectures in production). ", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Heterogeneous FL is an important problem \n- The paper is well-written and easy to understand\n- The authors compared their approach with number of existing algorithms ", "weaknesses": "- The reason to stitch together heterogeneous architectures is not very well motivated. There are many other approaches that aim to learn personalised model for a set of clients with heterogeneous capabilities. For example FjORD [https://arxiv.org/abs/2102.13451]  train a number of subnets using adaptive dropout, HeteroFL [https://arxiv.org/abs/2010.01264] is another model where a superset is trained and submodes are used to address heterogeneity in FL, FedRolex [https://arxiv.org/abs/2212.01548], or [https://arxiv.org/abs/2210.16105] all use weight sharing to address similar tasks.\n\nThe authors should discuss what is the main advantage/motivation of having heterogeneous architectures stitched together over other approaches where the computational complexity of a model can be scaled up/down, and possibly compare with some of the above methods. \n\n- Most experiments were done with few clients (N=12 up to 100). This is very small and might be unrealistic in real-=world applications where we might have million of clients participating in FL. It would be great to show how things scale up. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly thank you for the insightful comments and suggestions. We hope our responses can address your concerns.\n\n`>>> W1` ***Motivation of applying stitching technique*** \n\n`>>> WA1` \n\n(1) Thanks for the suggestions. Our approach is motivated by the challenge that clients may have different model structures and work together to deliver personalized models to enhance local client performance. Given the real-world setting where the clients’ models are heterogeneous, it can be treated as the “bottom-to-top” corporation in federated learning. We also appreciate the related studies mentioned by the reviewer in the comment. We will discuss them as follows.\n\n(2) The models, FjORD, HeteroFL, FedRolex, and AsyncDrop, all have a strong constraint, i.e., the client models are required to be submodels of the global model. This assumption might be too strong in real-world scenarios. In our work, we release this strong assumption, and any models can be used as client models. We assemble and reassemble the models with our designed strategies to obtain our candidate models to enable heterogeneous model cooperation. \n\n(3) In our approach, the model reassembly needs to follow the designed rules in Eq.~(1) of the original paper. This can help restrict the size of the candidate model pool. We also only fine-tune the parameters of the stitching layers rather than the whole generated models. In our experiments, the stitching layer is a nonlinear activation function on top of a linear layer, which is not very deep. Finally, it can be updated in a parallel way on the server side.\n\n`>>> W2` ***Larger scale clients*** \n\n`>>> WA2`\n \n(1) Thanks for your comments. Following FedAvg and pFedMe, we conduct part of our experiments with 100 clients. In our original paper, we test the setting with 12 and 100 clients, respectively. As the reviewer suggested, we conducted an experiment on 200 clients and set the active client ratio to be equal to $10\\%$. All the other settings keep the same as the subsection \"Large Number of Clients\" in Section 4.2 of the original paper. The results are shown below.\n\n| Public Data | Client Number | IID     | Non-IID |\n|-------------|---------------|---------|---------|\n| Labeled     | 100           | 80.02\\% | 77.63\\% |\n|             | 200           | 77.01\\% | 73.53\\% |\n| Unlabeled   | 100           | 78.98\\% | 75.77\\% |\n|             | 200           | 74.34\\% | 71.04\\% |\n\n(2) Compared with the results shown in the original paper, the performance has a slight decrease. Given the same amount of training data but a larger number of clients,  each client will have less training data. This would cause the performance to decrease at a certain range. \n\n`>>> Q1` ***Model personalization*** \n\n`>>> QA1` \n\nWe appreciate your question. With the heterogeneous model structures, we propose an approach via model disassembly and assembly to enable them to cooperate together. After we obtain the candidate models, we select one model for each client as the personalized teacher to guide the local modal update based on the similarity defined in Eq.~(5) of the original paper. In our proposed work, enabling heterogeneous model cooperation leads to different model architectures, and similarity-based matching helps select the most personalized teacher model for each local client at the next communication round.\n\n`>>> Q2` ***Model generation*** \n\n`>>> QA2` \n\nThanks for your valuable suggestion. We would like to share our answers below:\n\n(1) Following our designed rules, we have restricted the size of the candidate model pool and the depth of the generated models. \n\n(2) Using our current framework, we are able to simplify our approach to obtain a few models for clients with several tiers, e.g., low, medium, and high, as the reviewer mentioned. This would work for a more practical and specific domain, e.g., IoT.  We do appreciate your suggestion.\n\n"}, {"review_id": "cO71CR95SH", "reviewer": "Reviewer_xdZD", "summary": "The paper proposes to use the recently published model reassembly technique (NeurIPS 2022) to obtain personalized models through federated learning. At each round, the server collects the current models from the clients and uses reassembly to generate new candidate models, potentially training some stitching layers using a public datasets. Then the closest candidate model (again similarity is evaluated through the public dataset) is sent back to the client who can (later) train its own model distilling knowledge from the candidate model. By using the public dataset only for a few selected operations, the proposed scheme should be more robust to deviations from the training dataset and the public one.\n", "questions": "* in the homogeneous setting, candidate model generation reduces to simply swapping layers across clients' models? or is it still possible to obtain different models' architectures (e.g. with a higher number of layers than any of the original models?)\n* are results reported averages over different experiments or single-run experiments? I am also asking because some of the reported differences are really small (e.g., in table 2 pFedHR outperforms sota by less than 1 percentage point in many configurations)", "limitations": "I think the paper should have discussed the following limitations (see corresponding weaknesses above)\n* performance under low clients' sampling rate\n* the server keeps track of clients' update\n* Computational overhead\n* The clients need to work with models more complex than its own.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The idea to use reassembly for federated learning is a novel one to the best of my knowledge \n* Experimental results are promising and the robustness to the choice of the public dataset is definitely an important plus of the proposed approach.\n", "weaknesses": "* In the proposed scheme, the server does not keep historical aggregate information about the training, as for example it does under FedAvg by storing the last version of the shared model.  Historical information is rather kept at the client, which at each round performs a local training with knowledge distillation from the candidate model selected by the server during the previous communication round to which the client participated.  As a consequence the method does not seem suited for large-scale cross-device settings where clients may be selected only a few times. The authors have considered clients' sampling rates between 1/3 and 1/10. I expect performance to decrease significantly for lower rates.\n* The proposed solution requires the server to maintain the identity of the each client (to be able to send back the relevant candidate model). This prevents the applicability of privacy-preserving techniques like secure aggregation. Note that there are other personalized approaches which do not have this constraints (e.g., Ditto, FedEM,...)\n* Computational overhead. If I understood correctly, the server needs to train the stitching part for every possible client/candidate-model pair, i.e., to train in total BM models, which poses a significant load on the server. \n* Complexity of the proposed solution. It would have been good to perform an ablation study to evaluate if all pFedHR steps are really needed. For example, what if the clients' models are directly compared and the closest one is sent as candidate model to the client without performing any reassembly and stitching?\n* The candidate model can be more complex than the client's model. There is then an implicit assumption that, while the client has selected a given model size for example on the basis of its computational and memory capabilities, it is still able to use a more complex model for knowledge distillation at training time\n* The comparison with the previous literature is not always clear. Two examples:\n\t1. a limitation of previous literature would be that \"the averaging process significantly diminishes the characteristics of individual local models\" I found this sentence too vague.\n\t2. \"however, FedDF trains a global model with different settings compared with our approach.\"  Again, this is too vague, what is the difference with FedDF in a few words?\n* compute\n\t* the authors checked the compute checkbox but I was not able to find any information about computation in the paper or in the supplementary material\n* reproducibility\n\t* while the code is provided there is no readme file about how to use it and how to reproduce the results in the paper.\n* minors:\n\t* footnote numbers should go after punctuation marks\n\t* report the number of clients for table 4\n\t* typos: bettwen and cadidates \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the reviewer's valuable comments. Hope our responses below adequately address your concerns.\n\n`>>> W1` ***Historical information maintaince*** \n\n`>>> WA1` \n\n(1)  Candidate models are reassembled using the layers from the models of active clients. Our designed optimization approach enforces that the selected layers are from different functional groups and different operation sets. Though there is no single global model aggregated, the historical information is still able to be maintained. \n\n(2) We add more experiments with keeping all the settings the same as the experiment in the subsection \"Large Number of Clients\" of the original paper, except that we changed the sample ratio from 0.1 to ` 0.05`. With a lower active client ratio, the size candidate model pool will shrink, and the quality of the personalized model that clients obtain will be affected, which results in a performance drop.\n| Public Data | Active Client Ratio | IID     | Non-IID |\n|-------------|---------------------|---------|---------|\n| Labeled     | 0.05                | 79.21\\% | 75.86\\% |\n|             | 0.1                 | 80.02\\% | 77.63\\% |\n| Unlabeled   | 0.05                | 78.36\\% | 74.70\\% |\n|             | 0.1                 | 78.98\\% | 75.77\\% |\n\n`>>> W2` ***Privacy*** \n\n`>>> WA2` \n\n(1) Thanks for the valuable suggestion. In our setting, the model structures of the clients are heterogeneous, which prevents us from using the classic model aggregation approach. Therefore, the aggregation-based privacy-preserving techniques may not be directly applied to our approach. \n\n(2) In our work, we fully disassemble models into layers, and the candidate models are automatically generated. After this step, the original model structure information would be changed. After assembly with the stitching layers, we tune the stitching layer parameters to obtain the models, which leads to the change of the model parameters. In such a way, it may alleviate privacy concerns. Designing a more secure and private methodology for our current approach would be one of the convincing future works.\n\n`>>> W3` ***Computational overhead*** \n\n`>>> WA3` \n\n(1) Your understanding is correct. We only need to train the stitching parts rather than the whole model. The depth of the stitching layers is not very huge. \n\n(2) We first cluster layers into different groups based on their functionality and follow all the designed rules to get the candidate models. With such constraints, we restrict the size of the candidate model pool.\n\n(3) In most FL settings, we would assume that the server may have considerable computational resources. Tuning the stitching layers of the candidate models can be conducted in a parallel way. \n\n`>>> W4` ***Ablation study*** \n\n`>>> WA4` \n\n(1) We conducted an ablation study. We directly use Eq 5 as the similarity metric to compare and select the most similar models to distribute back to the clients. With all the identical settings in the subsection \"Large Number of Clients\" of the original paper, the results are as below:\n| Public Data | Approach | IID     | Non-IID |\n|-------------|----------|---------|---------|\n| Labeled     | Ablation | 63.68\\% | 56.09\\% |\n|             | pFedHR   | 80.02\\% | 77.63\\% |\n| Unlabeled   | Ablation | 57.40\\% | 51.17\\% |\n|             | pFedHR   | 78.98\\% | 75.77\\% |\n\n(2) We observe that direct applying the similarity as the metric to re-distribute the models would cause the performance to decrease significantly, especially under the non-IID setting.\n\n`>>> W5` ***Client  capacity*** \n\n`>>> WA5` \n\n(1) With the constraints we designed, the size of the candidate model pools and the depth of the candidate model have been restricted. In fact, we find that the size of the largest generated candidate model is usually smaller than that of the largest client model.\n\n(2) Our work is also flexible for restricting the maximum depth of the generated candidate model. In our future work, we would like to add more specific strategies to generate size-aware candidate models for the specific application domain, such as IoT.\n\n`>>> W6-9` ***Presentation, reproduction, and minors*** \n\n`>>> WA6-9` \n\nWe truly thank the reviewer for the suggestions. We will revise as the reviewer suggested.\n\n`>>> Q1` ***Homogeneous setting*** \n\n`>>> QA1` \n\nThanks for your question. In the homogeneous setting, it would still follow all the automatic steps without humans involved. It is still possible to obtain different models’ architecture. When all the models are selected as M4, we show one obtained model structure in the attached PDF file.\n\n`>>> Q2` ***Multiple runs*** \n\n`>>> QA2` \n\nWe have provided the 3-time run average results:\n\n\n| Public Data | Dataset | MNIST             |                  | SVHN               |                  | CIFAR-10         |                  |\n|-------------|---------|-------------------|------------------|--------------------|------------------|------------------|------------------|\n|             | Model   | IID               | Non-IID          | IID                | Non-IID          | IID              | Non-IID          |\n| Labeled     | FedMD   | 93.06$\\pm$1.22 \\% | 92.21$\\pm$1.36\\% | 81.86$\\pm$1.17\\%   | 77.93$\\pm$1.29\\% | 68.55$\\pm$1.52\\% | 64.07$\\pm$1.42\\% |\n|             | FedGH   | 94.33$\\pm$1.45\\%  | 92.78$\\pm$1.50\\% | 81.77$\\pm$1.89\\%   | 80.68$\\pm$2.03\\% | 72.04$\\pm$1.79\\% | 70.53$\\pm$1.85\\% |\n|             | pFedHR  | 94.86$\\pm$1.02\\%  | 94.25$\\pm$0.79%  | 84.32 $\\pm$0.88\\%  | 83.08$\\pm$1.02\\% | 73.64$\\pm$0.86\\% | 72.22$\\pm$1.32\\% |\n| Unlabeled   | FedKEMF | 93.23$\\pm$2.06\\%  | 91.07$\\pm$2.77\\% | 80.03 $\\pm$1.96\\%  | 77.27$\\pm$1.86\\% | 67.33$\\pm$2.30\\% | 64.80$\\pm$1.88\\% |\n|             | FCCL    | 93.49$\\pm$2.33\\%  | 92.11$\\pm$2.45\\% | 82.65 $\\pm$ 2.01\\% | 78.40$\\pm$1.88\\% | 68.13$\\pm$1.69\\% | 66.06$\\pm$1.90\\% |\n|             | pFedHR  | 93.96$\\pm$1.37\\%  | 93.25$\\pm$1.68\\% | 83.77$\\pm$ 1.42\\%  | 81.35$\\pm$1.35\\% | 71.96$\\pm$1.01\\% | 68.31$\\pm$1.46\\% |\n\n\n\n"}]

Author rebuttal:
We truly appreciate the insightful questions. We provide responses to Q1 from Reviewer xdZD. We hope it could help adequately address your concerns.

In the pdf file, there is a figure for the generated model under the homogeneous setting. Thanks again for your question.

Meta-review: This paper aims to address the model heterogeneity problem in federated learning. In particular, the authors proposed a novel framework named pFedHR, which leverages heterogeneous model reassembly to achieve personalized federated learning. Reviewers agreed that the proposed framework is novel, and the experiments results are convincing. The authors' rebuttal has also successfully addressed most of the concerns from reviewers. The authors are highly encouraged to incorporate the suggestions from reviewers to their final version.
Decision: Accept (poster)