pFedClub: Controllable Heterogeneous Model Aggregation for Personalized Federated Learning

Authors: jiaqi  wang
qi  li
lingjuan  lyu
fenglong  ma
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 05:39:03

Abstract: Federated learning, a pioneering paradigm, enables collaborative model training without exposing users’ data to central servers. Most existing federated learning systems necessitate uniform model structures across all clients, restricting their practicality. Several methods have emerged to aggregate diverse client models; however, they either lack the ability of personalization, raise privacy and security concerns, need prior knowledge, or ignore the capability and functionality of personalized models. In this paper, we present an innovative approach, named pFedClub, which addresses these challenges. pFedClub introduces personalized federated learning through the substitution of controllable neural network blocks/layers. Initially, pFedClub dissects heterogeneous client models into blocks and organizes them into functional groups on the server. Utilizing the designed CMSR (Controllable Model Searching and Reproduction) algorithm, pFedClub generates a range of personalized candidate models for each client. A model matching technique is then applied to select the optimal personalized model, serving as a teacher model to guide each client’s training process. We conducted extensive experiments across three datasets, examining both IID and non-IID settings. The results demonstrate that pFedClub outperforms baseline approaches, achieving state-of-the-art performance. Moreover, our model insight analysis reveals that pFedClub generates personalized models of reasonable size in a controllable manner, significantly reducing computational costs.

Main paper: 
 introduction: 
 Federated learning (FL) [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] is a prevalent method to train machine learning models collaboratively without centralizing clients' data on a cloud server. However, many current FL training frameworks demand uniformity in deep neural network structures among client models, a requirement often too stringent for practical, real-world applications. An alternative approach involves equipping clients with heterogeneous models, introducing a new challenge: how to aggregate these diverse models within the federated learning framework effectively.Recently, various approaches have emerged to address the challenge of aggregating heterogeneous models, particularly in the context of personalized FL. Some methods leverage additional information, such as class information [16] , logits [17, 18] , and label-wise representations [19] , as intermediaries for exchanging information between clients and the server. While seemingly straightforward, this approach raises significant privacy concerns, especially regarding the potential exposure of sensitive client data. To mitigate these privacy concerns, techniques such as distillation [20, 21, 22] and model reassembly [23] have been introduced in heterogeneous FL, wherein only model parameters are exchanged, akin to traditional FL approaches [1] . Despite demonstrating effectiveness in aggregating heterogeneous models, both distillation and model reassembly techniques in FL suffer from a common drawback -lack of control over personalized model generation.Distillation-based approaches inherently necessitate the establishment of a unified model as the global model, informed by prior insights [21] . This global model is then distributed to clients to guide their training efforts. However, a smaller consensus global model may struggle to extract heterogeneous knowledge from clients, often resulting in a larger size. This poses challenges for smaller clients with limited computational resources to run the shared large global model effectively. Similarly, model reassembly-based approaches also encounter a related issue. While they generate personalized models for each client, these personalized models may be significantly larger than the clients' capacity, posing challenges for implementation. To further investigate these limitations, we conducted an experiment using the state-of-the-art model reassembly approach, pFedHR [23] , on the SVHN dataset, where each client operated a distinct model. (Additional details and further discussions can be found in Section 4.4.) We evaluated the parameter size of the original models and the average number of parameters in the personalized models received across communication rounds. Subsequently, we illustrated the parameter size differences between these two types of models in Figure 1 . The entire bar represents the average parameter size of the generated personalized model, with the original model size depicted in blue and the increased size shown in red. Our preliminary findings indicate that the personalized models produced by pFedHR are notably larger than the original local models.To address this issue, in this paper, we introduce a novel approach for heterogeneous model aggregation, named pFedClub, aiming to achieve personalized Federated learning through Controllable neural network block substitution, as depicted in Figure 2 . pFedClub receives heterogeneous models uploaded from clients on the server. pFedClub first decomposes the heterogeneous client models into different blocks and subsequently clusters these blocks based on their functionalities (refer to Section 3.2.1). pFedClub explores a novel neural network block substitution technique to achieve this objective, as detailed in Section 3.2.2. Specifically, pFedClub aims to substitute the r-th block B t m,r within the m-th client model w t m with a block selected from the same group as B t m,r during communication round t. This approach ensures both the functionality of personalized models and their similarity to the original models. To enhance the diversity of the generated models, we permit arbitrary substitutions from the group for the first block (Step 1). For subsequent blocks, we introduce an order-constrained block search strategy (Step 2), ensuring the quality of the generated models. If the order constraint halts the substitution prematurely, the remaining blocks are directly added to the substituted ones (Step 3). This completion strategy not only reduces the number of newly added parameters in the stitching blocks but also ensures similar functionality. As pFedClub may generate multiple personalized candidate models for each client, we employ the similarity-based model-matching technique to select the personalized model, as discussed in Section 3.2.3. The selected personalized model is then distributed to the respective client as a teacher model, guiding the client model training process through knowledge distillation.It is essential to emphasize that the proposed pFedClub framework is designed to be both general and flexible, allowing for the incorporation of strict controllable constraints during the personalized candidate generation process. For instance, constraints such as the model size of the generated candidates can be easily integrated into the framework. Furthermore, introducing controllability within pFedClub enables the mitigation of computational and communication costs compared to current model reassembly-based approaches. By incorporating controllable constraints, pFedClub offers greater adaptability and efficiency in handling personalized model generation. Experimental results demonstrate that pFedClub achieves state-of-the-art performance on three benchmark datasets under both IID and non-IID settings, demonstrating the effectiveness of the proposed aggregation strategy. We take four clients with heterogeneous models as an example. The numbers denote the blocks' indexes. The number of functional groups is 4. We take Client 1 as an example to demonstrate how pFedClub works to generate personalized candidate models. Note that the arbitrary substitution for Block 1 of Client 1 is the second block from Client 4.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: Our abstract and introduction reflect the paper's contributions and scope.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Model Heterogeneity in Federated Learning. Heterogenous model cooperation is a challenging task in FL. Researchers have explored submodel training techniques [24, 25] , focusing on training a shared large global model by sending masked heterogeneous models to appropriate clients. However, these approaches often fail to provide personalized models for individual clients. Furthermore, they are considerably constrained by the limitations in freedom of model selection. In addition, FedDF [20] and FedKEMF [21] conduct ensemble distillation, but the settings of FedDF are different from ours and not for model personalization. FedKEMF utilizes mutual knowledge distillation with the requirement of predefined model structures. Besides, HeteroFL [24] and FlexiFed [25] are restricted to the requirements of the client model structures. Other related research work needs extra information to be exchanged between the server and clients, e.g., logits in FCCL [17] , class scores in FedMD [16] , and label-wise representation in FedGH [19] , which raises the concerns of privacy [26] .The most recent work pFedHR [23] provides a layer-wise model reassembly approach to solve the challenge of model heterogeneity in federated learning. However, it has several limitations, as we discuss in Section 1.Personalized Federated Learning. Instead of maintaining one global model, personalized FL cares more about each local model's performance, which is more sufficient and practical. In [27] , the authors add a proximal term to the local optimization loss function to bound the difference between the local and global model updates. The aggregated global model is treated as the initial shared model from the meta-learning perspective in [28] . In [29] , the authors design a new regularized client loss to optimize the local model to achieve personalization. However, the discussed personalized federated learning work assumes that the clients have to share identical model structures. Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discuss the limitation in the appendix.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
overview: 
 Let D n = {(x i , y i )}The model heterogeneity of the uploaded M client models {w t 1 , • • • , w t M } at the t-th communication round makes it challenging to generate personalized client teachers {w * t 1 , • • • , w * t M }. To tackle this challenge, a new controllable block-wise substitution-based personalized model aggregation approach is proposed, which not only maintains the functionality of each block in the originally uploaded client models but also injects new knowledge provided by other models to further achieve model personalization. To this end, we first divide each client model w t m into blocks and then group blocks into different functionality clusters.Block Decomposition. Except for recurrent deep neural networks such as the long-short term memory network [30] , most of the remaining ones, such as the family of convolutional neural networks (CNN), can be treated as block-stacked neural networks 3 . Let R m denote the number of blocks in client model w t m . We then decompose w t m into blocks{B t m,1 , • • • , B t m,r , • • • , B t m,Rm }.Note that a block is either a convolutional net, a fully connected layer, or a building block associated with a shortcut connection such as residual neural networks [31] .Block Grouping. After decomposing each model, we then apply the K-means algorithm to group blocks based on their functionality. Specifically, we apply the centered kernel alignment (CKA) technique [32] to calculate the similarity between two blocks as follows:EQUATIONwhere x t •,i denotes the input of the block B t •,i , and To reduce the size of the candidate model pool, a naive solution is to randomly choose a fixed number of models from the pool first and then use the model similarity score to select the personalized teacher model. Although this approach increases the diversity of the candidate model generation, it is uncontrollable to introduce too much randomness, and the low-quality candidates may reduce the convergence rate of the federated system training. To solve these issues, we design a Controllable Model Searching and Reproduction (CMSR) algorithm (as depicted in Algorithm 1), which is a greedy-based search approach to reproduce a set of personalized candidate models for each w t m by considering both diversity and quality. In particular, CMSR consists of three key steps: anchor block selection, order-constrained block search, and block completion. Next, we describe the details of each step.B t •,i (x t •,i ) represents the output of the block B t •,i (• = m or n). Let {G t 1 , • • • , G t K } denoteStep 1: Anchor Block Selection (Alg. 1 lines 3-7). Assume that the first block B t m,1 in w t m belongs to the group G t k . CMSR then randomly selects one block in G t k as the substitution. The substituted block will be treated as the anchor/starting block of all the candidate models. As shown in Figure 2 , w t 4,2 is selected as the substitution of w t 1,1 . Note that a naive solution to select the anchor block may use the block with the largest similarity score calculated via Eq. (1) instead of randomly choosing one. However, such a solution significantly reduces the diversity of the generated candidates, further limits the extra knowledge borrowed from other models, and finally hurts the personalization of teacher models. Besides, B t m,1 has a chance to be selected with a probability return :{c t m,1 , • • • , c t m,Sm }Step 2: Order-constrained Block Search (Alg. 1 lines 8-18). After selecting the anchor block inStep 1, CMSR then finds the substitutions for the following blocks. The simplest solution is repeating the previous step R m -1 times to generate a candidate model. As we discussed before, it may generate low-quality candidate models.To avoid this problem and generate controllable high-quality candidates, we maintain the order of the selected blocks, even from different client models, as a hard constraint [33] . Mathematically, let B t p,q denote the substitution of the r-th block of B t m,r , where p is the model index and q is the block index. For any substitution of the (r + 1)-th block B t•,u should satisfy the constraint q < u. As shown in Figure 2 , the substitutions of w t 1,2 include w t 2,3 and w t 3,3 since the index of the first block's substitution w t 4,2 is 2. Only using the order constraint is sufficient for the proposed pFedClub to generate high-quality and informative candidates. First, maintaining the order of block functions can guarantee that the candidate model has functionality similar to the original model. Second, it also avoids pre-defining the order of operation types, which releases the constraints of personalized teacher model generation and further increases the diversity of candidates. Third, such an approach is capable of generating similar-sized personalized models for clients. It is essential for several applications with limited computational resources, such as smart devices. However, pFedHR cannot control the size of the generated candidates.It is worth noting that, aside from the order constraint, pFedClub is highly adaptable and can easily incorporate other types of constraints. In Section 4.4, we will delve into the integration of the model size constraint, demonstrating the framework's versatility and ability to accommodate various constraints for personalized model generation.Step 3: Block Completion (Alg. 1 lines [19] [20] [21] [22] . Step 2 may stop at the certain block r < R m due to the block order constraint. To maintain the original functional structure, we will add the remaining blocks of w t m , i.e., {B t m,r+1 , • • • , B t m,Rm } to the substitutions. In such a way, pFedClub can generate a set of candidates denoted as {c t m,1 , • • • , c t m,Sm }, where S m is the number of generated candidate models. As shown in Figure 2 , pFedClub will stop after substituting the third block w t 1,3 since all the block indexes in Group 4 for substituting w t 1,4 are not greater than 4, which is the minimum feasible block index. Thus, pFedClub will complete the generated candidates by directly using w t 1,4 as the forth block.The final stage of pFedClub is to automatically select the "best" candidate teacher model from{c t m,1 , • • • , c t m,Sm } for the m-th client. However, selecting such a model is non-trivial because c t m,sis a reassembled, incomplete model via block substitution, and the dimension sizes of different blocks may not be well-aligned.Block Stitching. We complete each candidate model c t m,s using the network stitching technique [34] . We use a nonlinear activation function ReLU(•) on top of a linear layer, i.e., ReLU(W ⊤ X + b) as the dimension mapping function.Since the parameter values of {W, b} in the stitching functions are unknown, it is essential to learn them with training data. Here, we propose to use a public dataset D p to fine-tune the stitched candidate model c ′ t m,s . Note that we fix all the parameters in c t m,s (denoted as θ * m,s ) and only update {W, b} in c ′ t m,s using the following loss if the public data are labeled:EQUATIONwhere |D p | denotes the number of data in the public dataset, CE(•, •) means the cross-entropy loss,c ′ t m,s (x i ; W, b, θ * m,s) presents the predicted label distribution for the data x i by fixing the parameters θ * m,s , and y i is the ground truth vector. An unlabeled public dataset can be used to fine-tune the parameters {W, b} using the normalized temperature-scaled cross-entropy loss [35] for a pair of data as follows:EQUATIONwhere cos(•, •) is the cosine similarity, and τ is the hyperparamter. x j is the augmentation of x i . We still fix the parameters θ * m,s and learn {W, b}. It is worth noting that block stitching operation will not significantly increase the number of parameters in the candidate model. Besides, some candidate models may be generated using Step 3: block completion. For those candidates, the number of newly added parameters is much smaller. Moreover, the limited number of parameters is helpful for the new candidate models to maintain more original model information. Finally, it makes model computation efficient and speeds up the model training.Model Selection. Let ĉt m,s denote the fine-tuned candidate model via Eq. ( 2). We then calculate the average cosine similarity scores on logits outputted by the original model w t m and its candidate model ĉt m,s as follows:EQUATIONwhere β t m (x i ) and βt m,s (x i ) denote the logits of x i outputted by the models w t m and ĉt m,s , respectively. Note that this is a forward propagation and does not need to train the models. Finally, the candidate model with the highest similarity scores in the set {α m,1 , • • • , α m,Sm } will be selected as the final personalized teacher model w * t m .When the m-th client is selected again at the j (j > t) communication round, the personalized model w * t m generated in the recent communication round will be distributed. Since the teacher model w * t m usually has a different network structure from the client model w j m , we propose to use knowledge distillation to update the client model following [36] by optimizing the following loss:EQUATIONwhere λ is a hyperparameter and KL(•, •) is the Kullback-Leibler divergence.4 ExperimentsQuestion: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer Justification: Yes.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
experimental setups: 
 Datasets. In our experiments, we utilize three commonly used datasets to validate the performance of the proposed pFedClub, including MNIST 4 , SVHN 5 , and CIFAR-106 . We randomly divide the datasets into three parts: 72% for training, 20% for testing, and 8% as the public dataset. We test two data distribution settings in federated learning, i.e., IID and non-IID, following existing work [23] . For the IID setting, the training and testing data are randomly distributed to N clients.For the non-IID setting, each client randomly holds data belonging to two classes.Baselines. The proposed pFedClub aims to aggregate heterogeneous client models to boost federated learning performance. Based on the condition of public datasets, we consider the following approaches as our baselines: (1) without using public datasets: HeteroFL [24] and FlexiFed [25] ; (2) using labeled public data: FedMD [16] , FedGH [19] , and pFedHR [23] ; and (3) using unlabeled public data: FCCL [17] , FedKEMF [21] , and pFedHR [23] . We also compare the proposed pFedClub with the general approaches to learning personalized federated learning models, which share the same structure for all clients. The homogeneous baselines include: FedAvg [1] , FedProx [27] , Per-FedAvg [28] , PFedMe [29] , PFedBayes [37] , and pFedHR [23] . The details of all baselines can be found in Appendix A.Client Model Deployment. In our experiments, we employ seven client models with different network structures, including MobileNetV1 [38] , MobileNetV2 [39] , MobileNetV3 [40] , and four manually designed CNN models (denoted as CNN1 to CNN4). Each CNN model contains several convolutional blocks and fully connected blocks. The detailed model structures of the four models can be found in Appendix B. We set the number of clients N = 50 and the number of active clients M = 5 in each communication round. We propose three plans to distribute client models to validate the performance of the proposed pFedClub in different scenarios. First, we use all seven types of models and randomly send each model to a client (Model Zoo I). Specifically, CNN1 is randomly assigned to 8 clients, and each of the remaining six models is randomly assigned to 7 clients. Second, we distribute the three MobileNet family models to clients (Model Zoo II). In particular, we randomly assign MobileNetV1 to 16 5) is 0.2. The hyperparameter τ in Eq. ( 3) is 0.07. We use Adam as the optimizer. The learning rate of the local client learning and the server fine-tuning learning rate equal 0.001. We use average client accuracy with three runs as the evaluation metric.Heterogenous Model Aggregation. the submodel training technique and require that each client model must be a part of the global model. Thus, they are only tested with Model Zoo 3. These results show that our proposed approach outperforms all the baselines under most settings, especially the more complicated datasets SVHN and CIFAR-10. Compared with the most recent work pFedHR, our proposed work pFedClub shows superior performance over that on SVHN and CIFAR-10 datasets under both the IID and non-IID settings. For pFedClub, the use of the labeled public data is able to boost the performance compared with the setting using unlabeled public data, which aligns with the observations in [23] . In addition, from Model Zoo III to Model Zoo I, the performance of pFedClub on the more complicated datasets, SVHN and CIFAR-10, improves with the increase of diversity of the model zoos.Homogeneous Model Aggregation. The clients are assigned the same model structures to verify the effectiveness of pFedClub under the homogenous setting. We test the performance with CNN2 and MobileNetV2 under the non-IID setting and compare it with the state-of-the-art homogenous federated learning work. The results are shown in Table 2 . Note that pFedHR and pFedClub are under the setting where the public data is labeled. We observe that the results of all the approaches on the MNIST dataset are relatively high, even with a simple CNN2 model, as classification on the MNIST dataset is an easy task. Besides, pFedClub outperforms state-of-the-art baselines on SVHN and CIFAR-10 datasets using the CNN2 model or MobileNetV2 model. These results demonstrate that pFedClub is also effective for the homogeneous setting. We report the results in Table 3 and provide the following observations: (1) Removal of any one module will cause the performance drop, thus demonstrating the individual contribution of each design in our proposed pFedClub.(2) When we use simple strategies (i.e., pFedClub max and pFedClub min ) in the anchor block selection, the drop in the performance is smaller than studies (i.e., pFedClub noc and pFedClub nbc ). It indicates that maintaining the block number and keeping the model completion matter more than the anchor block selection. Overall, each designed module has its own contribution, and the systematic combination of all the designed modules guarantees the effectiveness of our proposed pFedClub.Experimental Setups. The major advantage of the proposed pFedClub is enabling the generation of controllable personalized candidate models. To clearly exhibit the insights, we use Model Zoo I as the heterogeneous model set, and each type of model is assigned to a corresponding client. Besides, each client will be mandatorily active during all the communication rounds. We use unlabeled public data for model training under the non-IID setting. To quantitatively evaluate the controllability of the personalized models generated by pFedClub, we propose to use the average of the model size change percentage over T communication rounds as the metric for each client, which is defined as follows: The proposed model can be treated as a fine-grained model reassembly technique, which uses controllable block-wise substitution to generate personalized candidates. In this experiment, we aim to compare the computational cost of the server between pFedClub and pFedHR using the computational time at each round on the SVHN dataset under the non-IID setting with Model Zoo III for 50 clients. We record the consumed computation time on the server side for each communication round. The results are shown in Figure 4 . We can observe that the computation time at the server side of our approach pFedClub is generally shorter than that of the baseline pFedHR. Also, with the algorithm running with respect to the communication round, our approach becomes more consistent and stable compared with the significant shift of pFedHR. These results confirm that pFedClub is an efficient approach for heterogeneous model aggregation compared with pFedHR. Except for controllability and computational costs, system running time is another key factor to evaluate the utility of the proposed pFedClub. Toward this end, we conduct an experiment to compare the consumed time to reach a fixed accuracy. The experimental setting is the same as the one that we described in Section 4.5. We take pFedHR as a baseline for comparison again.EQUATIONThe results are shown in Figure 5 . We can observe that the proposed approach pFedClub takes less time to achieve the target accuracy on the SVHN and CIFAR-10 datasets under the non-IID setting compared with pFedHR. These results demonstrate the effectiveness of the proposed pFedClub for the heterogeneous model aggregation in federated learning again.To validate the model scalability of pFedClub, we conduct the experiments by considering different numbers and different active ratios of clients, and the results are shown in Table 4 in Appendix D.Besides, in our model design, there is a key parameter K used in Section 3.2.1. We validate the sensitivity of the selection of K, and the results are listed in Table 5 in Appendix E. 
conclusion: 
 This paper introduces pFedClub designed to revolutionize personalized federated learning. By leveraging a unique network block substitution method, pFedClub effectively creates tailored and functionally analogous personalized models for individual clients. Moreover, pFedClub is highly adaptable and can easily incorporate other types of constraints to achieve application-driven personalized model generation. Our experimental evaluations, conducted on three diverse datasets under both IID and non-IID settings, unequivocally validate the efficacy of pFedClub in the domain of heterogeneous model aggregation for federated learning. The results affirm the accuracy, efficiency, and flexibility of our proposed method, demonstrating its potential for real-world applications. In Section 4.4, we have validated that the size of the generated personalized models by the proposed pFedClub is much smaller than that produced by pFedHR. Figure 3 shows the relatively average change in model size. In this experiment, we aim to show a detailed comparison of model size changes at each communication round for each type of heterogeneous model.The results are shown in Figure 6 . We can observe that our approach pFedClub conducts better control over the generated models compared with pFedHR in almost every communication round, especially for the client with the MobileNet models, where the size of our generalized model parameters is always smaller than the original one (ϕ < 0). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We provide the broader impacts in the appendix.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cite the original paper, code package, and dataset.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: Besides, model scalability is important in federated learning systems. To validate the scalability of pFedClub, we conduct the following experiments by considering different numbers and different active ratios of clients. The results are shown in Table 4 .Given the different settings of the different numbers of clients and active ratios, we can observe that the performance changes according to our expectations. Moreover, given a fixed active ratio, the performance has a slight decrease with the increase in the client number. One possible reason is each client will have a smaller number of training data when the client number increases given a certain number of training data. The change in the local model performance will harm the performance of the whole system in a certain range. Furthermore, given a fixed number of clients, the increase in the active ratio will slightly boost the performance because it enables more clients to contribute to the update process with their local training at each communication round. The experiment results demonstrate the scalability of our proposed approach, considering the change in the client number and active ratio. In our design, K is the number of the groups based on the function-wise clustering with K-means. In this experiment, we aim to study how the hyperparameter K affects the performance. We maintain the experimental setting in Sections 4.5 and 4.6. The results are shown in Table 5 . We can observe that the performance of pFedClub will slightly increase with the increase of K. In the main experiments, we follow [23] and set K = 4. When K = 5, the performance can increase slightly. One possible reason is that a large K is able to produce more specific function-based groups and further identify the function of the blocks more accurately. We can also observe that the performance is generally stable with the change of hyperparameter K in a certain range.This work focuses on the controllable personalized model generation for the heterogeneous federated learning setting. Although the proposed pFedClub outperforms baselines and is able to generate size-controllable client models, it still has several limitations. First, the proposed controllable model searching and reproduction (CMSR) algorithm is a heuristic algorithm that is designed based on intuitions. Thus, the results may be suboptimal. Second, in the system running time experiments, we have demonstrated that the proposed pFedClub can reduce the learning time compared with the state-of-the-art model pFedHR. However, compared with traditional averaging methods, the running time on the server is still much longer. We plan to design a more efficient algorithm for the server update. Finally, in the experiments, we only test the image classification task, which limits the efficacy test on other tasks. We will test more diverse tasks with the proposed pFedClub in the future.This research significantly enhances federated learning by enabling efficient aggregation of heterogeneous models without compromising data privacy. This approach not only improves user experience across diverse sectors by providing tailored solutions but also supports sustainable computing practices of large-scale machine learning operations.NeurIPS Paper ChecklistThe source code can be found at https://github.com/JackqqWang/24club.38th Conference on Neural Information Processing Systems (NeurIPS 2024).In this paper, we do not decompose recurrent deep neural networks, which is our future work.https://yann.lecun.com/exdb/mnist/http://ufldl.stanford.edu/housenumbers/https://www.cs.toronto.edu/~kriz/cifar.html 


Decision: In this paper, the authors present a new method for heterogeneous model aggregation in personalized federated learning. Now, the paper has received four reviews, and the reviewers all think that the paper has studied an important problem with strong results through extensive experiments.

While the paper shows some promise, there are some weaknesses that have been identified in the reviews.    
(1) More experimental evaluations are suggested, eg, under public datasets.  
(2) Some technical details require more clarifications.  
(3) The presentation requires more improvements.

During the rebuttal, the authors provided detailed responses to each reviewer's comments, and the reviewers were generally convinced by the authors' clarifications. During the discussion period, each review showed positive score about this submission, therefore, we recommend to accept this paper in NeurIPS 2024.

Reviews: review: paper_summary: paper_summary: This paper addresses heterogeneous model aggregation in federated learning. To this end, the authors introduce pFedClub, which aims to generate personalized models for federated clients while ensuring that the models remain within size constraints. Specifically, pFedClub consists of three main steps: first, it decomposes models into multiple blocks and clusters them using the K-means algorithm; second, it replaces original blocks with others from the same clusters to create a set of candidate models; third, it selects the optimal personalized model for each client using a public dataset and an initial model transferred to the server. Extensive experiments illustrate its significant improvement over existing methods in this field.
strength_weakness: strength_weakness: 1. The proposed work requires a public dataset, which is unsuitable in federated learning due to the privacy concerns. Is this work applicable to a public dataset different from the training data distribution? For example, the clients collaboratively train a model for CIFAR-10, while the server holds a public dataset from ImageNet. 
2. Although the proposed work achieves remarkable under convolutional neural networks, it is unclear how pFedClub performs under transformers. Is the proposed work suitable for a setting where clients hold three different sizes of LLM, i.e., LLaMA-7B, LLaMA-13B, and LLaMA-70B?
questions: questions: See **Weaknesses**
limitations: limitations: See **Weaknesses**
paper_summary: paper_summary: The paper proposes a `pFedClub` method for personalized federated learning that enables controllable heterogeneous model aggregation, addressing limitations of existing approaches such as lack of personalization, privacy concerns, and uncontrolled model size growth.

Extensive experiments conducted on three benchmark datasets using various CNN-based model structures validate the effectiveness of the proposed method under both IID and non-IID settings.
strength_weakness: strength_weakness: 1. The writing and structure of the paper need improvement, particularly in the "Order-constrained Block Search" paragraph. The concept of order is unclear, especially the meaning of $q < u$ in line 177. It's not evident whether this refers to a similarity score or another metric. The author should provide a clearer explanation of this constraint.
2. In equation (1) on line 141, the meaning of 'CKA' is not defined. The authors should explain what CKA stands for and how it's calculated. Additionally, it's unclear whether this computation occurs on the server. If clients must transmit input $x_{m,i}^t$ and output to the server, this raises privacy concerns that should be addressed.
3. The paper doesn't specify whether the features $x_{m,i}^t$ and $x_{n,j}^t$ in equation (1) have the same dimensions. This should be clarified to ensure a proper understanding of the similarity calculation.
4. The sampling process for the Anchor Block selection is ambiguous. The probability distribution over all models for this selection is not clearly defined.

Overall, the authors should formulate the proposed method more rigorously, using well-defined notations and providing clear explanations for each step of the algorithm. This would significantly improve the paper's readability and reproducibility.
questions: questions: 1. Table 5 indicates that the model achieves the best performance in both IID and non-IID settings when K equals the number of activated clients. However, this raises a question about the necessity of the K-means method in this scenario. When K equals the number of activated clients, the input data naturally satisfies the minimization objective of the K-means algorithm, rendering the clustering step redundant. It would be valuable to explore how K affects the results when the number of activated clients increases, for example, to 10 or 20. This analysis would provide deeper insights into the scalability and robustness of the proposed method.

2. While the current experiments focus on CNN-based structures, it would be beneficial to validate the proposed method on other neural network architectures. Specifically, evaluating pFedClub on Transformer-based structures would demonstrate its versatility and applicability across different model types. This expansion of the experimental scope would strengthen the paper's contributions and broaden its potential impact in the field of federated learning.

3. The supplementary material should include a comprehensive set of implementation details for all methods used in the comparisons. This should encompass not only the proposed pFedClub method but also such as the hyparameter for baseline approaches.
limitations: limitations: None
paper_summary: paper_summary: This paper presents a controllable model reassembly approach to enable heterogeneous model cooperation in federated learning. The designed CMSR algorithm provides the control of the space to save the computational cost.  Furthermore, the approach also achieves model personalization for each local client. They test the proposed approach on benchmark datasets and compare with other baselines under different settings.
strength_weakness: strength_weakness: Weakness
1, In their approach, the authors employ K-means clustering. The reviewer is curious about how the value of K is selected and how this selection influences the results.

2, One of the main contributions compared to pFedHR is the enhanced controllability. I am interested in understanding the nature of this controllability, specifically the extent to which the generated models can be controlled.

3, The paper focuses solely on image classification. Adhering to the review guidelines, the reviewer is not requesting additional experiments, but the reviewer is interested in exploring whether the existing methodology could be applicable to other tasks.
questions: questions: 1, Please address the concerns in the weakness part.
2, After the blocks are stitched, the parameters of the blocks and/or the stitching layer would be trained? If trained, how are they trained? If not, how do you deal with the parameters of the stitching layers?
limitations: limitations: There are no potential negative societal impacts of the work. There are two limitations as follows:
(1)	This approach still raises extra computational cost at the server side. 
(2)	That would be great if this approach can be extended to other tasks and other domains.
paper_summary: paper_summary: The authors address a key issue in personalized federated learning, which enables clients with heterogeneous model structures to participate in federated learning with consideration of effectiveness and efficiency. This method is based on model assembly and reassembly, in which the blocks and layers can be treated as modules. After that, the server selects the personalized models and assigns them to the clients. The received models will be used as the teacher to guide the local update. The authors run extensive experiments to demonstrate the effectiveness of their algorithm.
strength_weakness: strength_weakness: 1. Based on the algorithm itself, it includes the reassembly, assembly, matching, and other operations. The reviewer may be concerned about the computational burden compared with the one without any controllability. 

2. How to select the anchor block and why needs to be stated clearly.

3. According to the experiment results, the reviewer is wondering about how this approach can be used with the public data with/without labels and the possible reason why it is robust to the public data with or without labels.
questions: questions: Please see the above weaknesses.
limitations: limitations: No negative social impact to the reviewer’s best knowledge. The study of K should be put in the main content as that is an important part of the algorithm.

Comments: comment: Dear reviewer,

Thanks for your time to consider our reply. We appreciate your valuable comments. We will add more details and explanations to improve the readability of our paper in the camera-ready version. Also, we will extend our proposed approach to a broader range of neural network architectures as the reviewer suggested.

Thank you.
Thank you for your response. I am inclined to raise my score to 5. However, I recommend that the authors include more details and explanations to improve the readability of the paper.

The approach to aggregating heterogeneous models is particularly interesting. I encourage the authors to validate the proposed method on a broader range of neural network architectures.
Thanks for your reply. I will keep my score positive as it is. Thank you.
We sincerely appreciate your time to check the rebuttal and your recognition of our work.
Thanks for the reply. After reading the answers,  they have addressed my concerns. I would like to increase my score appropriately.
Dear Reviewer AXhq,

We greatly appreciate your constructive comments, which have significantly clarified our paper. As the discussion phase nears its end, could you kindly confirm whether our responses have satisfactorily addressed your concerns?

Thank you,

Authors of Paper 3650
Thank you for your feedback. We are delighted to have addressed your concerns and appreciate the improved score.
Thanks for your response. I find the rebuttal addresses my concerns. I believe this is a good paper and benefits ML community. Therefore, I decide to increase my score from 6 to 8.