Decoupling General and Personalized Knowledge in Federated Learning via Additive and Low-rank Decomposition

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 17:02:09

Abstract: To address data heterogeneity, the key strategy of personalized Federated Learning (PFL) is to decouple general knowledge (shared among clients) and client-specific knowledge, as the latter can have a negative impact on collaboration if not removed. Existing PFL methods primarily adopt a parameter partitioning approach, where the parameters of a model are designated as one of two types: parameters shared with other clients to extract general knowledge and parameters retained locally to learn client-specific knowledge.  However, as these two types of parameters are put together like a jigsaw puzzle into a single model during the training process, each parameter may simultaneously absorb both general and client-specific knowledge, thus struggling to separate the two types of knowledge effectively. In this paper, we introduce FedDecomp, a simple but effective PFL paradigm that employs parameter additive decomposition to address this issue.  Instead of assigning each parameter of a model as either a shared or personalized one, FedDecomp decomposes each parameter into the sum of two parameters: a shared one and a personalized one, thus achieving a more thorough decoupling of shared and personalized knowledge compared to the parameter partitioning method.  In addition, as we find that retaining local knowledge of specific clients requires much lower model capacity compared with general knowledge across all clients, we let the matrix containing personalized parameters be low rank during the training process. Moreover, a new alternating training strategy is proposed to further improve the performance.  Experimental results across multiple datasets and varying degrees of data heterogeneity demonstrate that FedDecomp outperforms state-of-the-art methods up to 4.9\%.

Main paper: 
 introduction: 
 Federated learning (FL) [28] allows clients to collaboratively train a global model without directly sharing their raw data. It has garnered widespread attention in the design of multimedia artificial intelligence systems [4, 20, 36, 41] . A central challenge in FL is data heterogeneity, where the data distributions across diverse clients are not independently and identically distributed (non-IID). Such disparities in data distributions hamper the training of the global model, leading to a decrease in the performance of FL [10, 22, 39] .To confront this challenge, the concept of Personalized Federated Learning (PFL) has been introduced. Within PFL studies, it is widely accepted that the knowledge learned by a client should be decoupled into two categories: the general knowledge across all clients and client-specific knowledge for this client [6, 34, 38] . The former is used for sharing among clients to promote collaboration, while the latter is retained locally to keep personalization and reduce the impact of data heterogeneity on collaboration. This understanding prompts mainstream PFL research to propose a partition based method where each parameter in the client's personalized model is designated as one of two types before training begins: parameter shared with other clients to extract general knowledge and parameter retained in personalization to learn client-specific knowledge. A multitude of studies have emerged. For instance, FedPer [2] focuses on personalizing the classifier, whereas FedBN [24] targets the personalization of Batch Normalization layers. FedCAC [34] proposes to select personalized parameters based on measurable metrics.Although the aforementioned methods have achieved some success and attracted widespread attention, they still haven't effectively separated the two types of knowledge. This is because the personalized parameters and shared parameters of a model learn from local data together as a whole, allowing each parameter to potentially absorb both general and client-specific knowledge simultaneously. This means that personalized parameters may contain some general knowledge that should be shared, and shared parameters may contain some client-specific knowledge that should be personalized. matrix containing nine parameters. The blue diagonal rectangle and the orange diagonal rectangle within each parameter represent the general knowledge and personalized knowledge contained in that parameter, respectively, with their area size indicating the amount of knowledge. It can be seen that each parameter contains both general and personalized knowledge. The partition based method divides some parameters into shared parameters (as shown by the blue squares), and the rest into personalized parameters (as shown by the orange squares). Obviously, this "black or white" partitioning method cannot achieve knowledge decoupling within parameters, leading to some client-specific knowledge in shared parameters being shared, thereby reducing the degree of model personalization; while some shared knowledge in personalized parameters is personalized, thus reducing the level of client collaboration.In this paper, we propose a new PFL paradigm based on parameter additive decomposition, called FedDecomp, to address the aforementioned issues. Unlike methods based on partitioning, which classify a parameter as either personalized or shared, FedDecomp decomposes each parameter into the sum of two parameters before training begins: one shared to facilitate knowledge exchange among clients and one retained locally to maintain personalization. Furthermore, we find that in PFL, general knowledge should be retained in shared parameters with high model capacity to cover all clients, while a client's specific knowledge can be learned by personalized parameters with lower model capacity as a supplement to general knowledge. Hence, FedDecomp constrains the matrix containing personalized parameters to be low rank. This allows the personalized part to focus its learning on the most critical aspects of the local knowledge and reduce the overfitting to the local knowledge. Consequently, it helps to retain a significant portion of the general knowledge acquired from other clients, thereby enhancing generalization. Fig. 2 illustrates our proposed method. It can be seen that in our method, both the general knowledge and client-specific knowledge in each parameter can be decoupled and accordingly captured by shared and personalized parameters, achieving more efficient client collaboration and personalization.In addition, different from the current methods which simply train personalized parameters and shared parameters simultaneously, we examine the training order of shared and personalized parameter matrices during local updates. Specifically, we propose to initially train the personalized low-rank part to mitigate the influence of non-IID data, followed by training the shared full-rank part. Our findings suggest that adopting an alternating approach, unlike concurrent training methods, yields greater benefits.Our primary contribution in this paper can be summarized as follows:• We introduce a new method of decomposing shared and personalized parameters in PFL, namely FedDecomp. Specifically, we decompose each layer of the personalized model into the sum of a shared full-rank part to preserve general knowledge and a personalized low-rank part to preserve client-specific knowledge.  
related work: 
 PFL has emerged as a prevalent research direction to handle the non-IID problem in FL. Current PFL methods can be mainly divided into meta-learning-based methods [1, 8] , fine-tuning-based methods [5, 17] , clustering-based methods [3, 31] , model-regularization-based methods [23, 33] , personalized-aggregation-based methods [14, 40] , and parameter-partition-based methods. Among these methods, the parameter-partition-based method has attracted a lot of attention due to its simplicity and effectiveness.Parameter-partition-based method. The core idea of this kind of method is to share part of the original model's parameters while personalizing the other part. Representative works include selecting specific layers for personalization, such as FedPer [2] , Fe-dRep [7] , and GPFL [38] proposing to personalize classifiers. FedBN [24] and MTFL [29] suggest to personalize the Batch Normalization (BN) layers. LG-FedAvg [25] and FedGH [37] propose to personalize feature extractor. Other works employ Deep Reinforcement Learning (DRL) or hypernetworks technologies to automate the selection of specific layers for personalization [27, 32] . Still, some other research no longer selects personalized parameters based on layers but on each individual parameter, making more fine-grained choices to personalize parameters sensitive to non-IID data [34] . In recent years, some studies propose another kind of personalized parameter partitioning method. Unlike the previous method, the core idea of this method is to add additional personalized layers to the original model. For example, ChannelFed [42] introduces a personalized attention layer to redistribute weights for different channels in a personalized manner. [30] proposes to add a bottleneck module for personalization after each feedforward layer.Parameter-decomposition-based method. A few PFL works also utilize parameter decomposition techniques. For instance, Fedpara [15] decomposes the personalized model parameter matrix into the Hadamard product of two low-rank matrices. It has been proven that the parameter matrix after the Hadamard product still possesses a high rank, thus not sacrificing model capacity. This way, it only requires uploading the low-rank matrix with fewer parameters during training, thereby reducing communication overhead. Factorized-FL [16] decomposes the model parameter matrix into the product of a column vector and a row vector. During training, it Step 1: Freeze and train .Step 4: Initialize with .Step 3: Aggregate a global model .Step 2: Freeze and train . personalizes the row vector while sharing the column vector. This is essentially a low-rank decomposition technique aimed at reducing communication overhead by only uploading the column vector. FedSLR [13] , on the other hand, performs a low-rank decomposition of the parameter matrix when the server distributes the model, thus reducing the communication overhead of the downlink.It is evident that our approach differs significantly from the current decomposition-based methods, from objectives to methodologies. The current methods mainly focus on the communication issue in PFL by reducing the amount of communication between clients and the server through low-rank decomposition of parameters. Our paper, however, focuses on the decoupling and extraction of knowledge in PFL. Through additive decomposition, it decouples the learning of general knowledge and client-specific knowledge. By constraining the personalized matrix to be low-rank, it coordinates the relationship between general and client-specific knowledge. 
method 3.1 overview of feddecomp: 
 We first give an overview of FedDecomp. As illustrated in Fig. 3 , each layer of client 𝑖's personalized model is decomposed into the sum of a full-rank matrix and a low-rank matrix. The training process in each communication round can be summarized as follows: 1) each client 𝑖 freezes its full-rank matrices 𝝈 𝑖 and updates the low-rank matrices 𝝉 𝑖 . 2) Then, each client 𝑖 turns to update 𝝈 𝑖 and freeze 𝝉 𝑖 . After local updating, all clients upload the full-rank part to the server while keeping the low-rank part private. 3) The server receives clients' full-rank matrices and aggregates them to generate a global model 𝝈. After doing this, the server sends 𝝈 back to all clients. 4) Each client receives the global model and uses it to initialize the full-rank matrices.PFL, in contrast to traditional FL algorithms that train a general model for all clients, strives to develop a personalized model for each client 𝑖, denoted as 𝑤 𝑖 , specializing in capturing the unique characteristics of its local data distribution 𝐷 𝑖 . In recent PFL research, there is a consensus that the knowledge acquired by individual clients comprises both general knowledge and client-specific knowledge. In non-IID scenarios, since different clients have distinct data distributions (i.e., 𝐷 𝑖 ≠ 𝐷 𝑗 , 𝑖 ≠ 𝑗), it is difficult to extract general knowledge and thus brings challenges to client collaboration. To address this problem, PFL decouples 𝑤 𝑖 into a shared part 𝜎 and a personalized part 𝜏 𝑖 to learn general knowledge and clientspecific knowledge respectively. Formally, the training objective can be formulated as minEQUATIONwhere 𝐿 𝑖 (𝜎; 𝜏 𝑖 ; 𝐷 𝑖 ) denotes the loss function of client 𝑖 and 𝑁 is the total number of clients. To optimize the target function in Eq. ( 1), recent studies have put forth various PFL methods to partition 𝜏 𝑖 and 𝜎. While these endeavors have shown promise, the question of how to further refine the decomposition of these two parameter components still presents an unresolved challenge.We observe that shared parameters responsible for extracting general knowledge benefit from a high model capacity. In contrast, personalized parameters are tasked with learning knowledge that complements the general understanding for specific local tasks (i.e., client-specific knowledge), therefore, it is sufficient to use a low-rank matrix to represent these personalized parameters. Based on this observation, we propose FedDecomp, an additive low-rank decomposition technique. Details about this method are as follows.Additive Decomposition of Personalized Models: Assume that each personalized model has a set of weights 𝜽 𝒊 = {𝜃 𝑘 𝑖 } 𝐿 𝑘=1 , where 𝜃 𝑘 𝑖 is the weights for the 𝑘-th layer and 𝐿 is the total layer number. Each weight matrix 𝜃 𝑘 𝑖 is originally full-rank. In FedDecomp, we decompose 𝜃 𝑘 𝑖 asEQUATIONwhere 𝜎 𝑘 𝑖 , 𝑘 ∈ [1, 𝐿] is a full-rank parameter matrix that is shared across all clients, and 𝜏 𝑘 𝑖 , 𝑘 ∈ [1, 𝐿] is a personalized low-rank parameter matrix. In the following, we employ the notation 𝜽 𝑖 , 𝝈 𝑖 and 𝝉 𝑖 to denote the complete model parameter set, the full-rank parameter set, and the low-rank parameter set specific to client 𝑖, respectively. Additionally, we use 𝜃 𝑘 𝑖 , 𝜎 𝑘 𝑖 , and 𝜏 𝑘 𝑖 to represent the parameter matrices for layer 𝑘 within client 𝑖.Next, we present the methods for imposing low-rank constraints on 𝝉 𝑖 .Low-rank Decomposition of Fully-Connected Layers: For fully-connected layers, the dimension of 𝜏 𝑘 𝑖 is 𝐼 × 𝑂, where 𝐼 and 𝑂 represent the input and output dimensions. We constrain 𝜏 𝑘 𝑖 through a low-rank decomposition as follows:EQUATIONwhere𝐵 𝑘 𝑖 ∈ R 𝐼 × (𝑅 𝑙 •min(𝐼,𝑂 ) ) and 𝐴 𝑘 𝑖 ∈ R (𝑅 𝑙 •min(𝐼,𝑂 ) ) ×𝑂 .The 𝑅 𝑙 serves as a hyper-parameter designed to regulate the rank of 𝜏 𝑘 𝑖 within fully-connected layers. Its value falls within the range of 0 < 𝑅 𝑙 ≤ 1.Low-rank Decomposition of Convolutional Layers: In contrast to fully-connected layers, convolutional layers involve multiple kernels, resulting in 𝜏 𝑘 𝑖 ∈ 𝐼 ×𝑂 ×𝐾 ×𝐾 dimensions. However, we can still apply a low-rank decomposition to constrain 𝜏 𝑘 𝑖 as follows:EQUATION𝐵 𝑘 𝑖 ∈ R (𝐼 •𝐾 ) × (𝑅 𝑐 •min(𝐼,𝑂 ) •𝐾 ) and 𝐴 𝑘 𝑖 ∈ R (𝑅 𝑐 •min(𝐼,𝑂 ) •𝐾 ) × (𝑂 •𝐾 ) , 𝜏 𝑘 𝑖 = Reshape(𝜏 𝑘 * 𝑖 ) ∈ R 𝐼 ×𝑂 ×𝐾 ×𝐾 .The 𝑅 𝑐 is a hyper-parameter used to control the rank of 𝜏 𝑘 𝑖 within convolutional layers. Its value is within the range of 0 < 𝑅 𝑐 ≤ 1.During training, both 𝐵 and 𝐴 serve as trainable parameter matrices. We initialize 𝐴 with random Gaussian values and 𝐵 with zeros, which means 𝜏 𝑘 𝑖 starts as zero at the beginning of training. The hyper-parameters 𝑅 𝑙 and 𝑅 𝑐 play crucial roles in controlling the rank of parameters within fully connected and convolutional layers, respectively. As the rank increases, the learning capacity of personalized parameters within the model gradually improves. However, if the rank is set too low, 𝝉 𝑖 may struggle to effectively capture client-specific knowledge, making 𝝈 𝑖 highly susceptible to non-IID data distributions. This, in turn, negatively impacts collaboration among clients. In contrast, if the rank is too large, 𝝉 𝑖 may start to absorb some of the general knowledge that should be learned by 𝝈 𝑖 , diminishing the level of collaboration among clients. For simplicity, in the FedDecomp approach, we apply the same 𝑅 𝑐 to all convolutional layers and the same 𝑅 𝑙 to all fully-connected layers. This simplification streamlines the model architecture and hyper-parameter tuning process.To better extract general knowledge, in contrast to the common practice where personalized and shared parameters are trained simultaneously, we find that a more effective strategy is to initially train the low-rank parameters. This alternating approach helps mitigate the impact of non-IID data before proceeding to train the full-rank parameters. Formally, in each communication round 𝑡 ∈ [1,𝑇 ], we first optimize the low-rank parameters 𝝉 𝑖 for 𝐸 lora epochs byEQUATIONThen optimize the full-rank parameters 𝝈 𝑖 for 𝐸 global epochs byEQUATIONWe set 𝐸 lora + 𝐸 global = 𝐸, where 𝐸 is the total number of local update epochs in one round. These hyper-parameters play an important role in balancing the learning dynamics between two key components, 𝝈 𝑖 and 𝝉 𝑖 . When 𝐸 lora is set higher, it results in 𝝈 𝑖 learning less knowledge. Consequently, the degree of knowledge sharing among clients diminishes. In contrast, if 𝐸 lora is set too low, 𝝈 𝑖 ends up acquiring a significant amount of client-specific knowledge. This scenario increases the risk of clients sharing knowledge that is more susceptible to non-IID data. In special cases, when 𝐸 lora = 0,EQUATIONand sends it back to clients. The detailed training process is summarized in the Algorithm 1.To explain our intuition for proposing alternating training, we employ a toy example to illustrate the local update phase of each client's personalized model within the parameter space. As shown in Fig. 4 , the yellow ★ and red △ denote the optimum points of the global model on all clients' data (global loss minimum point) and the personalized model on the client's data (local loss minimum point), respectively. Under the influence of non-IID, there is a big difference between global knowledge and local knowledge of clients. This makes the local minimum point far away from the global minimumInput: Each client's initial personalized parameter matrices 𝝉 (7) . Sending 𝝈 𝑡 +1 to each client 𝑖. end for point. The client's personalized model 𝜽 is decomposed into the sum of a shared part 𝝈 and a personalized part 𝝉. Since we first train the personalized part, the client-specific knowledge is mostly learned by 𝝉 and the shift of 𝜽 to the local minimum point is mainly done by 𝝉. Therefore, when training 𝝈, it moves less towards the local minimum point (i.e., less affected by non-IID data), so it can better extract general knowledge. In Section 4.4, we conduct an experiment to further validate this intuition.In this section, we analyze the memory usage, computation cost, and communication cost of FedDecomp in each client 𝑖 compared to the baseline method FedAvg.Memory usage: FedAvg needs to maintain a set of full-rank parameter set 𝜽 𝒊 . According to Eq. ( 2), Eq. (3) and Eq. ( 4), FedDecomp needs to maintain a full-rank parameter set 𝝈 𝒊 whose number of trainable parameters is equivalent to 𝜽 𝒊 , and a low-rank parameter set 𝝉 𝒊 whose number of trainable parameters is much fewer than 𝜽 𝒊 . Therefore, the memory required by FedDecomp is slightly higher than that of FedAvg.Computation cost: In one round, FedAvg updates 𝜽 𝒊 for 𝐸 local epochs. According to Eq. ( 5) and Eq. ( 6), FedDecomp updates 𝝉 𝑖 for 𝐸 lora epochs and update 𝝈 𝑖 for 𝐸 -𝐸 lora epochs. Because 𝝉 𝑖 has fewer trainable parameters than 𝜽 𝒊 , the computation cost of FedDecomp is lower than that of FedAvg.Communication cost: In one communication round, FedAvg needs to upload 𝜽 𝒊 while FedDecomp needs to upload 𝝈 𝑖 . As 𝝈 𝑖 has the same number of trainable parameters as 𝜽 𝒊 , FedDecomp has the same communication cost as FedAvg. 
experiments 4.1 experiment setup: 
 Dataset. Our main experiments are conducted on three datasets: CIFAR-10 [19] , CIFAR-100 [18] , and Tiny ImageNet [21] . Experiments on larger datasets involving text modalities are included in the supplemental material. To evaluate the effectiveness of our approach in various scenarios, we adopt the Dirichlet non-IID setting, a commonly used framework in current FL research [12, 26, 35] . In this setup, each client's data is generated from a Dirichlet distribution represented as 𝐷𝑖𝑟 (𝛼). As the value of 𝛼 increases, the level of class imbalance in each client's dataset gradually decreases. Consequently, the Dirichlet non-IID setting allows us to test the performance of our methods across a wide range of diverse non-IID scenarios. For a more intuitive understanding of this concept, we offer a visualization of the data partitioning in the supplemental material.Baseline methods. To verify the efficacy of FedDecomp, we compare it with eight state-of-the-art (SOTA) methods: FedAMP [14] , FedRep [7] , FedBN [24] , FedPer [2] , FedRoD [6] , pFedSD [17] , pFedGate [5] , and FedCAC [34] . Among these methods, FedAMP forces clients with similar data distributions to learn from each other. FedBN, FedPer, FedRep, FedRoD, and FedCAC are parameterpartition-based methods that partially personalize parameters. pFedSD and pFedGate are fine-tuning-based methods, whose goal is to adapt the global model to the client's local data. These methods cover the latest advancements in various directions of PFL.Selection for hyper-parameters. We utilize the hyper-parameters specified in the respective papers for each SOTA method. For the FL general hyper-parameters, we set the client number 𝑁 = 40, the local update epochs 𝐸 = 5, the batch size 𝐵 = 100, and the total communication round 𝑇 = 300. Each client is assigned 500 training samples and 100 test samples with the same data distribution. We select the best mean accuracy across all clients as the performance metric. Each experiment is repeated using three seeds, and the mean and standard deviation are reported. We adopt the ResNet [11] network structure. Specifically, we utilize ResNet-8 for CIFAR-10 and ResNet-10 for CIFAR-100 and Tiny ImageNet. In FedDecomp, we adopt the SGD optimizer with learning rate equals 0.1.In this section, we compare our FedDecomp with several SOTA methods. To ensure a comprehensive evaluation, we consider three different non-IID degrees (i.e., 𝛼 ∈ {0.1, 0.5, 1.0}) on CIFAR-10, CIFAR-100, and Tiny Imagenet.The results in Table 1 demonstrate that the performance of FedAMP is comparable to other SOTA methods on the CIFAR-10 dataset, but experiences a notable decline on CIFAR-100 and Tiny Imagenet. This is primarily because of its limited capacity to leverage collaboration among clients with diverse data distributions. In contrast, mainstream model partition methods such as FedRep, FedBN, FedPer, FedRoD, and FedCAC enhance collaboration among clients by personalizing parameters sensitive to non-IID data while sharing others. Among these methods, FedRoD distinguishes itself by introducing a balanced global classifier to facilitate comprehensive knowledge exchange, underscoring the potential for improvements in client collaboration within current model partition strategies. On the other hand, fine-tuning-based approaches like pFedSD and pFedGate enable all clients to collaboratively train a global model, fostering extensive knowledge exchange. However, this approach can lead to performance degradation in certain non-IID scenarios due to mutual interference during joint training. Notably, FedDecomp significantly outperforms all baseline methods in the majority of scenarios, particularly as 𝛼 increases. Fed-Decomp achieves this by effectively decoupling general and clientspecific knowledge through parameter decomposition and mitigating the impact of non-IID through alternating training of full-rank and low-rank matrices.Effect of 𝑅 𝑙 and 𝑅 𝑐 . As we discuss in Section 3.3, 𝑅 𝑙 and 𝑅 𝑐 individually denote the ratio of the low-rank matrix's rank to the full-rank matrix's rank in convolutional and fully-connected layers, respectively. They are two important hyper-parameters to control the learning ability of the low-rank matrices. In this section, we evaluate the effect of 𝑅 𝑙 and 𝑅 𝑐 on model accuracy. We choose 𝑅 𝑙 and 𝑅 𝑐 from {20%, 40%, 60%, 80%, 100%}.The experimental results are presented in Table 2 . Firstly, we observe that the optimal combinations of (𝑅 𝑐 , 𝑅 𝑙 ) are (60%, 60%) for CIFAR-10, (80%, 40%) for CIFAR-100, (80%, 40%) for Tiny Imagenet. This underscores the importance of setting the personalized parameter matrices to low rank. Secondly, regarding the optimal combination as the focal point, model accuracy gradually decreases as the rank increases. This occurs because, after this point, the personalized matrices gain more learning capacity and begin to acquire some of the general knowledge. As a result, collaboration among clients on the shared matrices diminishes. As the rank decreases, model accuracy also gradually declines. This is because the personalized matrices fail to capture sufficient client-specific knowledge. This aligns with our expectations. Thirdly, experimental results highlight that model accuracy is more sensitive to changes in 𝑅 𝑙 than 𝑅 𝑐 . This suggests that the acquisition of client-specific knowledge has a stronger correlation with the classifier than the feature extractor, consistent with prior research such as FedPer, FedRep, and FedRoD.Effect of 𝐸 lora and 𝐸 global . In this section, we verify the effect of 𝐸 lora and 𝐸 global on model accuracy. For simplicity, we set 𝐸 global = 𝐸 -𝐸 lora and only adjust the value of 𝐸 lora . We conduct experiments on three datasets under Dirichlet non-IID with 𝛼 = 0.1 and sample 𝐸 lora ∈ [0, 𝐸]. The experimental results are depicted in Fig. 5 . When the 𝐸 lora = 0, FedDecomp essentially degenerates to FedAvg, and the accuracy closely resembles the FedAvg accuracy presented in Table 1 , as expected. As 𝐸 lora increases, the accuracy initially rises and then declines. When 𝐸 lora = 5, FedDecomp degenerates to local training with low-rank parameter matrices. However, due to the constraints imposed by these low-rank matrices on the model's learning capacity, FedDecomp performs less effectively compared to the Local as shown in Table 1 .Effect of Alternating training. As we discussed in Section 3.4, different from previous work that trains personalized and shared components simultaneously, we propose to train the personalized part first and then the global part to reduce the impact of non-IID and better extract general knowledge. To evaluate this idea, in this experiment, we compare the performance of two training methods.The experimental results on three datasets are shown in Table 3. We can see that when the learning task is simple (e.g., a 10-classification task on CIFAR-10), the performance of alternating training and simultaneous training of two matrices is similar. As the learning task becomes increasingly difficult, the performance improvement brought about by alternating training becomes more apparent. This is because, in the case of a simple learning task, the variations in tasks among clients are relatively minor, which facilitates the extraction of general knowledge. However, as the learning task complexity increases, the differences in tasks among clients gradually expand, rendering the extraction of general knowledge more susceptible to non-IID effects. In such scenarios, the utilization Effect of Model Capacity. In FedDecomp, we employ an additive decomposition technique on the model. In theory, this approach does not change the model's capacity. However, in practical implementation, the decomposed model introduces low-rank matrices, thereby increasing the number of trainable parameters. This raises questions about whether the decomposed model genuinely enhances the model's capacity and whether the observed performance improvement is primarily a result of the increased number of trainable parameters. To address these concerns, we conducted an experiment to assess the impact on model capacity.We conduct experiments using two configurations: CIFAR-10 with the ResNet-8 model and CIFAR-100 with the ResNet-10 model. We established two controlled scenarios: 1) 'Local' and 'Local w/ Low-Rank' indicate models without and with low-rank matrices that are exclusively trained locally. 2) 'FedAvg' and 'FedAvg w/ Low-Rank' indicate models without and with low-rank matrices trained using the FedAvg algorithm. The experimental results are shown in Table 4 . Notably, we observe that, in comparison to the original model, the model enhanced with low-rank matrices exhibits only minimal performance improvement. This outcome underscores that our utilization of parameter decomposition does not bring about significant alterations to the model's capacity. Hence, the performance gains achieved by FedDecomp are not solely attributed to modifications in the model itself.Effect of Personalize Low-rank Matrices and Sharing Fullrank Matrices. As discussed in section 3.3, the shared parameters require a high capacity to maintain general knowledge among clients, while personalized parameters only need to preserve clientspecific knowledge as a supplement to the general knowledge. To validate this intuition, we evaluate 'FedDecomp_Reverse' which shares low-rank matrices and personalizes full-rank matrices. Other training strategy is the same as FedDecomp and the comparison results on CIFAR-100 are shown in Table 5 .We can see that the performance of 'FedDecomp_Reverse' is not as good as FedDecomp. Moreover, combining the results in Table 1 , we can see that the performance of 'FedDecomp_Reverse' is similar to local training. This indicates that in 'FedDecomp_Reverse', the knowledge is mainly preserved in personalized parameters and the knowledge sharing among clients is very limited. This supports our intuition of personalizing low-rank matrices and sharing full-rank matrices.As we discussed in Section 3.4 and Fig. 4 , the primary objective of alternating training is to mitigate the impact of data heterogeneity on 6 . It is evident from the data that, across both datasets, the utilization of alternating training significantly diminishes the differences in the shared parameters among clients. This is consistent with our intuition and analysis.In FedDecomp, we suppose that the client-specific knowledge is learned by the low-rank matrices. To validate this assumption, we initialize 𝝈 with pre-trained weights. In this case, the general knowledge is well extracted. If our idea holds, then the 𝝉 should be trained more to learn client-specific knowledge and 𝝈 should be trained less (i.e., Δ𝝈 should be much smaller than Δ𝝉).Specifically, We initialize the 𝝈 𝑖 , 𝑖 ∈ [1, 𝑁 ] with ImageNet pretrained weights and conduct an experiment on the CIFAR-100 dataset in the Dirichlet non-IID scenario with 𝛼 = 0.1. We calculate the Δ𝝈 by ||𝝈 𝑇 -𝝈 1 || 2 and the Δ𝝉 by 1 𝑁 𝑁 𝑖=1 ||𝝉 𝑇 𝑖 -𝝉 1 𝑖 || 2 . We control the value of 𝐸 lora at different levels (higher 𝐸 lora means update 𝝉 more often, and results in larger Δ𝝉), the results are summarized in the Table 6 .From the table, we can conclude that when Δ𝝉 is larger than Δ𝝈, FedDecomp achieves better results. This indicates that with the pre-trained weights, the 𝝉 should be updated more often than 𝝈 to learn client-specific knowledge. This aligns with our expectations. In this section, we analyze the privacy protection capability of the FedDecomp. To this end, we adopt the Deep Leakage from Gradient (DLG) method [9, 43] as the attack scheme. DLG is a common attack against FL, and its main idea is: 1) The attacker steals the gradients calculated by each client using local data; 2) The attacker finds the optimal input through iterative optimization, such that the gradient computed with this input is as close as possible to the actual gradient.In our specific experimental setup, we choose the CIFAR-10 dataset with 20 clients, and the data distribution of each client follows a Dirichlet distribution with 𝛼 = 0.1. For each algorithm, we assume that the gradients of shared parameters can be obtained by the attacker. At training rounds 10, 20, 30, 40, and 50, we attempt to recover each client's 5 training images using the DLG method. To measure the quality of image recovery, we use the Peak Signal-to-Noise Ratio (PSNR), which is defined as 𝑃𝑆𝑁 𝑅 = -10 • log 10( | |𝑥 -𝑥 * | | 2 2 𝑚•𝑛 ),where 𝑥 * is the target image to be recovered, 𝑥 is the image being optimized for recovery, and 𝑚, 𝑛 are the width and height of the image, respectively. A higher PSNR indicates that the recovered image is closer to the original image, which in turn implies weaker privacy protection by the algorithm. Table 7 shows our experimental results, including the average PSNR and the maximum PSNR of the attack results. The results show that our proposed FedDecomp method achieved lower PSNR compared to FedAvg and other personalized methods. This means that using the FedDecomp method, both on average and in the most extreme cases, better privacy protection can be achieved compared to previous methods. We believe that this is because our method can effectively decouple general knowledge from personalized knowledge and keep personalized knowledge well-preserved in the local low-rank branch. As such, it becomes difficult for the DLG method to recover the original image using only the shared part of the gradients. 
conclusion: 
 In this paper, we propose a new PFL method named FedDecomp. FedDecomp decomposes each model parameter matrix into a shared full-rank matrix and a personalized low-rank matrix. To further enhance the acquisition of general knowledge, we devise a training strategy that prioritizes the training of the low-rank matrix to absorb the influence of non-IID during local training. Our extensive experimental evaluations, conducted across multiple datasets characterized by varying degrees of non-IID, unequivocally demonstrate the superior performance of our FedDecomp method when compared to SOTA methods. 
 
 Appendix:  


Decision: This paper studies the personalized federated learning problem. The authors propose to decompose the parameters into two sets with one representing shared knowledge and the other representing local knowledge. The proposed solution is interesting. The evaluation is comprehensive, especially considering the results from the rebuttal. The performance improvement is relatively significant. Therefore, after carefully reviewing the reviews and rebuttals, the AC thinks that, despite lacking of theoretical analysis, the authors addressed most of the points identified by the reviewers, and suggests acceptance (poster). Please seriously consider the comments and carefully incorporate them into the final version.

Reviews: review: paper_summary: paper_summary: The paper introduces FedDecomp, a novel approach to Personalized Federated Learning (PFL) that addresses the challenge of data heterogeneity across clients.
strength_weakness: strength_weakness: 1.The paper proposes FedDecomp, which utilizes parameter additive decomposition to separate general knowledge shared among clients and client-specific knowledge. This is a significant advancement over traditional PFL methods that use parameter partitioning, which struggle to effectively distinguish between the two types of knowledge.

2.The paper presents a new training strategy where the personalized low-rank part is trained first, followed by the shared full-rank part. This order is intended to mitigate the influence of non-IID data and improve the extraction of general knowledge.

3.Experimental results across multiple datasets and varying degrees of data heterogeneity demonstrate that FedDecomp outperforms state-of-the-art methods up to 4.9%.
limitations: limitations: 1.There could be a need for a deeper theoretical foundation or analysis of the proposed method.

2.While the paper touches on privacy, a more comprehensive analysis could strengthen the paper.

3.The paper could potentially benefit from discussing how FedDecomp might be applied in real-world scenarios.
paper_summary: paper_summary: This paper introduces FedDecomp, a novel method in Personalized Federated Learning (PFL) that decouples general and client-specific knowledge via additive parameter decomposition. Each parameter is split into a shared and a low-rank personalized part, ensuring clearer knowledge separation. An alternating training strategy mitigates the influence of non-IID data. Experiments show that FedDecomp outperforms state-of-the-art methods by up to 4.9%, improving model personalization and client collaboration. This approach offers significant advancements in handling data heterogeneity in federated learning.
strength_weakness: strength_weakness: S1: The layout is well clear, with clearly defined notations and related cost analysis, which are mostly overlooked in existing federated work.
S2: FedDecomp reduces the computational burden while preserving essential client-specific knowledge, enhancing efficiency.
limitations: limitations: W1: Vital relevant work is overlooked, e.g., DBE.
W2: It is only applicable for vision or image data, without the further study for language data.
W3: In terms of the experiment results, the performance on CIFAR-10 dataset reported in Table 3 seems to be significantly lower than the results reported in other related works, e.g., FedRoD.
W4: The convergence impact analysis is deficient for the decomposition model.
W5: The privacy analysis is important, but this presentation is less rigorous as an academic evidence.
W6: the low-rank decomposition method is not novel or has more convincing strength for personalized learning.
W7: the effectiveness of with Low rank(by the results of Table 4) is not evidently improved.
W8: computation efficiency is not empirically validated.


[1] Zhang J, Hua Y, Cao J, et al. Eliminating domain bias for federated learning in representation space[J]. Advances in Neural Information Processing Systems, 2024, 36.
paper_summary: paper_summary: This paper presents a novel approach for Personalized Federated Learning (PFL) that addresses the challenge of effectively separating general and client-specific knowledge within model parameters. This paper utilizes a parameter additive decomposition strategy, splitting each parameter into two components: a shared parameter and a personalized one. This method allows for a clearer distinction between general and client-specific knowledge. Additionally, the adoption of a low-rank matrix for personalized parameters acknowledges that less model capacity is needed for client-specific learning compared to general knowledge. An alternating training strategy is also introduced to enhance performance.
strength_weakness: strength_weakness: 1. This method effectively distinguishes between general knowledge and client-specific knowledge more efficiently than traditional parameter partitioning approaches.
2. The paper proposes a new alternating training strategy that further improves the model's performance. This strategy optimizes the model's adaptability to different clients' data distributions, enhancing its applicability and robustness.
3. The authors have tested their method on multiple datasets and compared it across various non-IID data distributions.
4. The figures in the paper are well-drawn and clear.
limitations: limitations: 1. It seems that the author didn't proofread the paper very carefully, e.g., the text extends beyond the defined margins on lines 538 and 542.
2. The appendix provided in the supplementary material is single-column, which does not fit the double-column format of ACM MM.
3. No code is provided in the supplementary to reproduce the experiments.
4. The performance improvement is not very high, e.g., only 0.8\% of accuracy improvement on the CIFAR-10 dataset when the non-IID setting is 0.5, and less effective when the non-IID setting is 0.1.
5. The experiments provide a limited amount of information. For example, Table 2 is excessively large relative to the amount of information it conveys, authors can consider changing the table to a figure to save space and include the experiments in the appendix into the main manuscript.
paper_summary: paper_summary: In this paper,  the authors introduce FedDecomp, a simple but effective PFL paradigm that employs parameter additive decomposition. Experimental results demonstrate that FedDecomp outperforms state-of-the-art methods up to 4.9%.
strength_weakness: strength_weakness: (1) The paper is easy to follow and well-organized. I enjoy reading this paper.

(2) The methodology is novel and technically sound.

(3) The authors conduct extensive experiments to illustrate the proposed method obtains SOTA performance.
limitations: limitations: (1) I strongly encourage the authors to discuss the convergence analysis (either theoretically or empirically) for the proposed FedDecomp.

(2) I strongly encourage the authors to discuss is it possible that we first train on $\sigma$ then on $\tau$? Could it enhance or deteriorate the model performance?