DualFed: Enjoying both Generalization and Personalization in Federated Learning via Hierachical Representations

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 10:22:06

Abstract: In personalized federated learning (PFL), it is widely recognized that achieving both high model generalization and effective personalization poses a significant challenge due to their conflicting nature. As a result, existing PFL methods can only manage a trade-off between these two objectives. This raises an interesting question: Is it feasible to develop a model capable of achieving both objectives simultaneously?  Our paper presents an affirmative answer, and the key lies in the observation that deep models inherently exhibit hierarchical architectures, which produce representations with various levels of generalization and personalization at different stages. 
A straightforward approach stemming from this observation is to select multiple representations from these layers and combine them to concurrently achieve generalization and personalization. However, the number of candidate representations is commonly huge, which makes this method infeasible due to high computational costs. To address this problem, we propose DualFed, a new method that can directly yield dual representations correspond to generalization and personalization respectively, thereby simplifying the optimization task. 
Specifically, DualFed inserts a personalized projection network between the encoder and classifier. The pre-projection representations are able to capture generalized information shareable across clients, and the post-projection representations are effective to capture task-specific information on local clients. This design minimizes the mutual interference between generalization and personalization, thereby achieving a win-win situation. Extensive experiments show that DualFed can outperform other FL methods.

Main paper: 
 introduction: 
 Federated learning (FL) [41] is an emerging machine learning paradigm that enables multiple clients to collaboratively train a model while preserving their data privacy. In real-world applications, data distributions across clients are often non-independent and identically distributed (Non-IID). For instance, in video surveillance, the data collected by distributed cameras can vary significantly due to differences in weather and lighting conditions [7, 19, 28, 42] . This Non-IID data distributions can significantly degrade the FL model performance [64, 67] . Currently, there are primarily two objectives to mitigate this issue: improving model generalization to accommodate more clients or enhancing model personalization to better adapt local data distributions. However, since local data distributions often differ from the global distribution in Non-IID FL, these two optimized objectives are typically in conflict. Personalized federated learning (PFL), which aims to balance model generalization with personalization, serves as an effective approach to address the challenges posed by Non-IID data. Earlier PFL approaches suggest sharing the classifier or encoder, while personalizing the other [1, 12, 35] . This strategy aims to strike a balance between client collaboration and local adaptation, as presented in Figure 1 (a) and (b). However, these approaches can only ensure that the encoder to generate either the generalized or personalized representations. Thereby, some PFL methods suggest personalizing specific parameters within the encoder, allowing it to extract the representations that exhibit both generalization and personalization [33, 50, 52] . Additionally, some PFL techniques concurrently use global and personalized classifiers for predictions [6, 63] to harmonize generalization and personalization. Nevertheless, these methods inherently involve a trade-off between model generalization and personalization. This leads to an interesting question: Is it feasible to create a model that can achieve both of these objectives concurrently in Non-IID FL?In fact, the dilemma in existed PFL methods primarily because they rely solely on post-encoder representations for decision-making. This design presents a significant hurdle as it necessitates the postencoder representations to simultaneously exhibit both high generalization and personalization -objectives that are inherently contradictory in Non-IID FL. It is well known that deep models naturally produce hierarchical representations, as evidenced in studies such as [2, 18, 40, 44, 47, 51, 56, 59, 61] . The shallow layers capture general patterns that are transferable across different data distributions. As we delve into deeper layers, the representations become more specified for the downstream task. This implies that both the generalization and personalization that PFL seeks for are already existed within the model. These observations do shed some lights on us: Can we leverage the hierachical representations within the deep model to achieve both high model generalization and personalization simulatenously?In this paper, we provide a positive response to the question posed earlier. A straightforward method for leveraging hierarchical representations involves directly selecting both generalized and personalized representations from them. However, this approach can incur substantial computational costs, owing to the volume of the candidate representations [48] . To address this problem, we introduce DualFed, a new PFL approach that not only straightforward to implement but also effectively decouple these two types of representations. As shown in Figure 1 (c), in DualFed, we modify the commonly used encoder-classifier architecture by inserting a projection network between the encoder and classifier. This modification generates representations at two distinct stages, aligning with the objectives of generalization and personalization, respectively. Specifically, the pre-projection representations generated before the projection network, are isolated from local tasks, making them more transferable across clients. Conversely, the post-projection representations produced after the projection network are closer to the decision layers, being more discriminative and personalized to local data distributions. To align with the objectives of these two representations, we maintain a shared encoder while localizing the projection network. A global classifier and a personalized classifier are trained using the pre-projection and post-projection representations, respectively. During inference, the outputs from these two classifier are combined to yield the final predictions, effectively benefiting from collaboration across clients and local adaptation.We conduct extensive experiments on multiple datasets to demonstrate the effectiveness of DualFed. The experimental results show that DualFed can outperform state-of-the-art (SOTA) FL methods. 
related work: 
 Federated Learning. FL [22, 30] can be categorized into general FL (GFL) [23, 31, 41] and personalized FL (PFL) [1, 12, 33, 35, 52, 53] . GFL aims to develop a generalized model that can be shared across clients. However, in Non-IID FL, it becomes challenging for a global model to satisfy the diverse needs of multiple clients, often leading to significant performance degradation [64, 67] . Consequently, PFL has emerged as an effective solution for these Non-IID situations by introducing model personalization to better align with local data distributions. There are various approaches to implement PFL, including model clustering [3, 4, 16, 49] , and the personalization of specific parameters within the model [1, 12, 33, 35, 52] . However, these PFL methods can only manage a trade-off between these two objectives, as they expect the post-encoder representations to achieve the conflicting objectives.Representation Learning in Deep Models. Since advanced deep learning models are typically organized as hierachical layers, analysing how representations evolve during the representation extraction process has been an established field [40, 44, 56, 59, 61] . Previous research indicates that deep models start by extracting generalized features and progressively filter out irrelevant components, retaining only those crucial for downstream tasks [40, 59] . This has inspired numerous studies that leverage intermediate representations, in domains like object detection [36] , image classification [14] , and speech processing [11, 27] . However, selecting the optimal representations for each specific problem is computationally challenging [48] . In response, SimCLR [8] proposes to use a scalable projection network during training and discard it afterwards. This design has become a common practice in both supervised learning [15, 24, 57] and self-supervised learning [5, 9, 10, 17, 60] . Since then, numerous studies have explored the projector's role in model training from empirical [2, 32, 48, 57] and theoretical perspectives [21, 56, 58] . The common explanation is that the projection network differentiates the representations of the pre-training and downstream tasks, thereby enhancing the model transferability [57] . This situation is especially significant when the pre-training and downstream tasks are misaligned [2] . Nevertheless, the effects of projection network within FL are still not fully understood.Federated Learning within Representation Space. The primary contribution of these methods is the regularization of the representation space to mitigate data heterogeneity [37, 38, 46, 54, 62, 63, 65, 66] . A straightforward strategy in these approaches involves directly calibrating the representation space. For instance, CCVR [38] post-calibrates the classifier after federated training using virtual representations. Another research direction links performance degradation to the misalignment of representation spaces across clients [65, 66] . In response, various methods have been developed to explicitly align the representation space across clients. Notably, FedProto [54] , AlignFed [66] , and FedFA [65] use class-wise representation centers for representation alignment. Additionally, some methods achieve alignment by implementing a fixed classifier. For instance, FedBABU [43] employs a randomly initialized classifier, SphereFed [13] introduces an orthogonal classifier, while FedETF [34] implements an ETF (Equiangular Tight Frame) classifier during model training. However, these representation alignment methods primarily focus on extracting generalized representations shareable across clients, often overlooking the personalized representations specific to local tasks. Consequently, recent studies have concentrated on balancing both model generalization and personalization. For example, Fed-RoD [6] achieving this goal combining the predictions of personalized and global classifiers. Yet, these methods face challenges, as they rely solely on representations at the same stage. Expecting the single-stage representations to exhibit both generalization and personalization is often intertwined. 
preliminaries: 
 In this section, we present some preliminaries related to DualFed, including the PFL framework and its motivation. L 𝑚 (Θ 𝑚 ), (1) where L (Θ 1 , . . . , Θ 𝑀 ) represents the overall optimization objective for the PFL system, L 𝑚 (Θ 𝑚 ) denotes the empirical risk for client 𝑚. In PFL, directly optimizing L (Θ 1 , . . . , Θ 𝑀 ) is commonly infeasible as the clients cannot access the data on other clients. Therefore, a PFL training procedure typically involves the independently local updating performed on participating clients utilizing their own empirical risk and the model aggregation performed on the server. Specifically, for client 𝑚, its empirical risk is defined as:EQUATIONwith ŷ𝑖 𝑚 = 𝜓 𝑚 (𝒙 𝑖 𝑚 ; Θ 𝑚 ) representing the model's prediction for 𝒙 𝑖 𝑚 , and ℓ : Y × Y → R being the loss function that quantifies the prediction error (e.g., cross-entropy loss).Once the local training on clients is completed, the participating clients upload their updated global parameters within the model to the server. The server then averages the parameters at corresponding positions to generate new global parameters. These global parameters are subsequently distributed to the clients for the next round of local updating. By iteratively performing local training and model aggregation, PFL facilitates collaborative model training without the need to share raw data from the clients.For the sake of brevity, we occasionally omit the superscript denoting the sample index in subsequent sections of this paper. Additionally, we sometimes denote personalized parameters with the superscrip 𝑝 (e.g., Θ 𝑝 𝑚 ), and global parameters with the superscript 𝑠 (e.g., Θ 𝑠 𝑚 ), to clarify the expressions in the following sections.As shown Figure 1 (a) and (b), in previous studies of PFL, the model Θ 𝑚 is commonly divided into an encoder 𝑓 𝑚 (•) and a classifier ℎ 𝑚 (•) [1, 12, 43, 65, 66] , parameterized by 𝜃 𝑓 𝑚 and 𝜃 ℎ 𝑚 , respectively. The encoder 𝑓 𝑚 (•) : X 𝑚 → Z 𝑚 generally consists of a series of stacked convolutional layers. It maps the raw input 𝒙 𝑚 fromX 𝑚 ⊆ R 𝑛 into a representation space Z 𝑚 ⊆ R 𝑘 , which is denoted as 𝒛 𝑚 = 𝑓 (𝒙 𝑚 ; 𝜃 𝑓 𝑚 ).Here, 𝒛 𝑚 ∈ Z 𝑚 denotes the representation generated from 𝒙 𝑚 utilizing the encoder 𝑓 𝑚 (•). Practically, the dimension of this representation is significantly smaller than that of the raw input, which implies that 𝑘 ≪ 𝑛. The classifier, ℎ 𝑚 (•) : Z 𝑚 → Y 𝑚 , generally includes a fully connected (FC) layer and a softmax layer. It generates the normalized predictions ŷ𝑚 based on the representation 𝒛 𝑚 , which is indicated as ŷ𝑚 = ℎ 𝑚 (𝒛 𝑚 ; 𝜃 ℎ 𝑚 ). Nonetheless, the widely used encoder-classifier architecture proves to be problematic in PFL. In this framework, each client's model has dual objectives: collaborating with other clients to improve its generalization and adapting to its own local data distribution for better personalization. Within the encoder-classifier architecture, only the representations after the encoder, referred to as the post-encoder representations, are used for decision-making. This approach can lead to a dilemma in PFL, as generalization and personalization are contradictory objectives, particularly in Non-IID scenarios. More specifically, to enhance model generalization, the post-encoder representations should capture shared information across varying data distributions among clients. On the other hand, enhancing model personalization requires these representations to capture specific information aligned with each client's local data distribution. When the data distribution varies significantly across clients, these two types of information can be vastly different. Consequently, in this encoder-classifier architecture, ensuring that the post-encoder representations simultaneously meet these two conflicting objectives is a challenging task.To address the dilemma mentioned earlier, we shift our focus on the process of representation extraction within the deep models. Advanced deep models are typically organized in a hierachical architecture. As shown in previous studies, these models initially extract generalized representations that are transferable across various data distributions [2, 18, 40, 44, 47, 51, 56, 59, 61] . As the model progresses to deeper layers, it gradually discards irrelevant components and retains only information relevant to the specific task. In other words, both the generalized and personalized representations that PFL seeks for are already existed within the model. By leveraging these hidden generalized and personalized representations, we can achieve both high generalization and personalization in PFL. However, directly extracting these specific representations during the representation extraction phase is computationally challenging [48] . Therefore, DualFed adopts a simpler strategy by incorporating a personalized projection network, which effectively decouples the generalized and personalized representations. Further details on this approach are discussed in the subsequent sections. 
method: 
 In this section, we present a detailed discussion of our proposed DualFed. First, we provide a framework overview of DualFed. Following that, we present the procedure of local training on the clients and model aggregation implemented on the server, respectively. Functionally, this projection network, 𝑔 𝑝 𝑚 (•) : Z 𝑚 → U 𝑚 is usually a MLP (multi-layer perceptron). By inserting this projection network, the representations produced by the encoder are not directly inputted into the classifier for prediction. Instead, they first pass through the projection network, which remaps them to a personalized representation space U 𝑚 ⊆ R 𝑑 . Formally, we represent this process as 𝒖 𝑚 = 𝑔(𝒛 𝑚 ; 𝜃 𝑔,𝑝 𝑚 ). For clarity, we term the representation before the projection network (i.e., 𝒛 𝑚 ) as the pre-projection representations, and the representation after the projection network (i.e., 𝒖 𝑚 ) as the post-projection representations. Drawing on the hierarchical nature of deep model representation extraction, the pre-projection and post-projection representations in our framework exhibit distinct characteristics, aligning with the generalized and personalized objectives of PFL, respectively. Specifically, the pre-projection representations are separated from the final outputs by the projector network, meaning that they are not directly tied to the local tasks on each client. As previously studies have shown, these pre-projection representations are easier transferred across different data distributions [48, 57] . Therefore, in DualFed, the post-projection representations are fed into a global classifier ℎ 𝑠 𝑚 (•), which is parameterized by 𝜃 ℎ,𝑠 𝑚 . Additionally, to encourage the encoder to extract more generalized information, we let the encoder be shared among clients in DualFed. The predictions from this global classifier is expressed as:EQUATIONConversely, the post-projection representations are more closely aligned with the final outputs. This implies that these representations are more pertinent to accomplishing tasks related to the local data distribution. In DualFed, to effectively adapt to these local distributions, we utilize a personalized classifier ℎ 𝑝 𝑚 (•) for each client, parameterized by 𝜃 ℎ,𝑝 𝑚 , to adapt to the local data distribution. For a given input 𝒙 𝑚 , the prediction generated by this personalized classifier can be expressed as follows:EQUATIONDuring inference, the final predictions are derived by ensembling the outputs from both the global classifier and the personalized classifier. This process is expressed as follows:EQUATIONBy integrating a personalized projection network between the encoder and the classifier, DualFed effectively separates the contradictory optimization objectives inherent in PFL into distinct stages within the model. This approach resolves the conflict of pursuing contradictory objectives within the representations in the same stage, thereby can achieve a win-win situation between the model generalization and personalization.In DualFed, each client updates the model for 𝐸 rounds using its own datasets after revceiving the global models from the sever. In EQUATIONwhere 𝒚 𝑖,𝑐 𝑚 denotes the value at 𝑐 𝑡ℎ class of the one-hot ground-truth label of the 𝑖 𝑡ℎ sample on client 𝑚, ŷ𝑝,𝑖,𝑐 𝑚 represents the normalized prediction probability of 𝑐 𝑡ℎ classes of the 𝑖 𝑡ℎ sample on client 𝑚 from the personalized classifier.As the post-projection representations are tailored to adapt to the local data distribution, we further enhance its discrimination by implementing supervised contrastive loss [24] , as demonstrated in the following equation:EQUATIONwhere 𝐴 is the full set of samples, 𝐴(𝑖) consists of samples in 𝐴 that belong to the same class as 𝒙 𝑖 𝑚 , ⊙ is the cosine similarity, and 𝜏 ∈ R + is the temperature coefficient.The optimization objective at this stage is then defined as:EQUATIONwhere 𝜆 denotes the hyperparameter used for balancing these two loss terms, 𝑡 denotes the local updating epochs. After updating Θ 𝑚 , we freeze its parameters and train the global classifier using the pre-projection representations to fulfill the local task, as represented in the following equation:EQUATIONwhere 𝒚 𝑖,𝑐 𝑚 denotes the value at 𝑐 class of the one-hot ground-truth label of the 𝑖 𝑡ℎ sample on client 𝑚, ŷ𝑠,𝑖,𝑐 𝑚 represents the normalized prediction probability of 𝑐 classes of the 𝑖 𝑡ℎ sample on client 𝑚 from the global classifier.The optimization objective in this stage can be expressed as:EQUATIONIn DualFed, both optimization objectives, as described in Eqs. ( 8) and (10) , are optimized using mini-batch stochastic gradient descent (SGD). As evidenced by our experiments, this stage-wise optimization strategy diminishes the impact of local tasks on the pre-projection representations, thereby effectively preserving its generalization.Once the local updating process is complete, the clients send their global encoder and classifier parameters to the server. The server then aggregates these parameters using the following equation:EQUATIONFollowing the model aggregation, the server broadcast the updated model back to the clients to initiate the subsequent round of local training. 
experiment: 
 In this section, we conduct extensive experiments to showcase the effectiveness of DualFed, mainly including the comparative results with existed FL methods, and the additional analysis of DualFed.Our experiments are conducted on PACS [29] , DomainNet [45] , and OfficeHome [55] . The PACS dataset includes 4 distinct domains: Photo (P), Art Painting (A), Cartoon (C), and Sketch (S), each featuring images from 7 common categories. The DomainNet dataset encompasses 6 distinct domains: Clipart (C), Infograph (I), Painting (P), Quickdraw (Q), Real (R), and Sketch (S). Initially, each domain comprises 345 classes, but for our study, we narrow this down to 10 commonly used classes to create our experimental dataset. The Office-Home dataset contains images from 4 distinct domains: Art (A), Clipart (C), Product (P), and Real-World (R), each containing 65 classes. We retain all classes to conduct a comprehensive evaluation of DualFed on a larger-scale dataset.For these datasets, we select the images from a single domain to form the dataset of an individual client. In both PACS and Domain-Net, we choose a subset of 500 training images per client from the same domain for the training dataset. For Office-Home, we set the number of training samples to 2, 000 for the Clipart, Product, and Real-World domains. In the case of the Art domain, the number is limited to 1, 942, matching the total number of samples available in this domain. All the images from the test dataset are reserved for evaluation for these datasets. We apply random flipping and rotational augmentations to these images during the training.We perform a comparative analysis against the following methods, including FedAvg [41] , FedProx [31] , FedPer [1] , FedRep [12] , LG-FedAvg [35] , FedBN [33] , FedProto [54] , SphereFed [13] , Fed-RoD [6] , FedETF [34] . Additionally, the SingleSet method, where separate models are trained and tested for each client using only their private data, is also used for comparison in our experiments.The adopted encoder is from the one of the ResNet18 model pretrained on the ImageNet dataset [20] . It is followed by a projector network, which consists of an FC network with the architecture: [Linear(512, 256) -ReLU -BN -Linear(256, 512) -BN]. To ensure uniform model capacity, all compared methods employ this Encoder-Projector architecture for representation extraction.The learning rate is set 0.01, with a momentum of 0.5, for all methods except SphereFed . For SphereFed, we set the learning rate to 1.0 for Office-Home and to 0.1 for both DomainNet and PACS. During local updating, a batch size of 256 is consistent across all methods. The epoch of local updating is set to 1 for all methods except FedRep. For FedRep, it has a total of 5 local epochs, with the initial 4 epochs focusing on classifier optimization and the last epoch on encoder and projector optimization. The total global rounds is set to 300.The other hyperparameters for different methods are selected by grid searching. To mitigate cross-domain interference and potential privacy issues related to BN layers, we localize the running-mean and running-var components within these layers for all methods.To ensure the reliability of our results, each experiment is repeated 5 times with different random seeds: {0, 1, 2, 3, 4}. The subsequent sections will detail the mean and standard deviation of the highest test accuracy achieved during FL training. More implementation details are provided in Appendix.Tables 1 -3 showcase the experimental results of our proposed DualFed alongside other FL methods on the PACS, DomainNet, and Office-Home datasets, respectively. Notably, DualFed presents a significant performance gain in comparison to these SOTA methods. Interestingly, the SingleSet model stands out as a strong benchmark, despite not collaborating with other clients. This is particularly evident in simpler domains, such as the Quickdraw domain in the DomainNet dataset. The underlying reason is that these simpler domains requires less complex semantic information for downstream tasks. In these cases, a personalized encoder's representations are sufficient, and collaboration for extensive semantic However, as the complexity within a domain increases, such as in the Infograph domain of DomainNet, the benefits of sharing the encoder among clients become apparent. This collaborative approach allows the encoder to extract more nuanced semantic information from the raw data, improving overall model performance, as demonstrated by the results of FedAvg and FedProx. FedRep and FedPer, employing a personalized classifier to adapt the representations from the global encoder, often outperform FedAvg and FedProx. However, these methods primarily leverage the global encoder's representations and do not fully utilize personalized information to cater to the local data distribution on individual clients.FedProto significantly improves model performance by aligning representations from different clients within a unified representation space. Nonetheless, this alignment can result in a loss of semantic information pertinent to local tasks due to varying data distributions across clients. This issue is even more pronounced in models like SphereFed and FedETF, which employ a predefined classifier for representation alignment and lack specific semantic information about local data. Fed-RoD adopts an architecture similar to ours, utilizing both global and personalized classifiers to capture generalized and personalized information. However, it attempts to utilize representations at the same stage, posing challenges in simultaneously meeting these two contradictory objectives. In contrast, our proposed method strategically separates these two conflicting objectives into different stages of the model. This division allows us to achieve both generalization and personalization more effectively, ultimately resulting in superior performance across a wider range of scenarios.Comparison of global and personalized classifiers. To gain a deeper understanding of the behavior of the global and personalized classifiers, we compare their accuracy, individually and in combination, during training. Figure 3 shows the corresponding experimental results on DomainNet. It is evident that personalized classifier significantly surpasses the global one, owing to its better alignment with local data distributions. Nevertheless, the accuracy of the local classifier can be significantly improved by combining its predictions with those from the global classifier. This enhancement is particularly notable in complex domains, such as Infograph. Conversely, in simpler domains like Quickdraw and Sketch, the benefit of combining classifiers becomes less pronounced. This occurs because, in simpler domains, the representations extracted by the personalized projection network are sufficient for each client's local tasks, thereby reducing the necessity for more diverse representations from the global encoder.Visualization of generalized and personalized representations. To intuitively understand the generalized and personalized representations, we utilize t-SNE [39] for visualization. Figure 4 illustrates the visualization of both the generalized and personalized representations on DomainNet dataset. In these visualizations, different colors indicate different classes. It is noticeable that the personalized representations are more discriminative than the generalized ones, yet they exhibit lower consistency across clients. This demonstrates that DualFed can effectively separate the representation extraction process into two distinct stages, each characterized by high levels of generalization and personalization, respectively. Quantitative evaluation of generalized and personalized representations. We employ two metrics to quantitatively evaluate the evolution of generalized and personalized representations during training. To quantify the personaliztion of representations on clients, we adopt the class-wise separation in [25] . Additionally, we adopt the linear centered kernel alignment (CKA) [26] , to measure the generalization ability of representation. Figure 5 presents the varying of 𝑠𝑒𝑝𝑎𝑟𝑎𝑡𝑖𝑜𝑛 during the training. The personalized representations can achieve higher class separation compared with the generalized representations. However, as shown in Figure 6 , the similarity between clients of generalized representations is significant higher that that of the personalized representations. the pre-projection representations to align more closely with the local task, thereby reducing their generalization. Effect of Projector Architecture. We investigate the impact of the architecture of the projection network in three key aspects: the model depth (𝐷), the dimension of hidden layers (𝐻 ), the impact of BN layers. The corresponding results are shown in Table 5 . While increasing 𝐷 can lead to more generalized pre-projection representations, it simultaneously reduces their discriminative power. Therefore, it is advisable to select an optimal 𝐷 that maintains a balance in the discriminative and generalized ability of the preprojection representations. Increasing 𝐻 can enhance the model performance in most times. The importance of BN layers becomes more pronounced as the scale of the dataset increases. Effect of Position of Global Classifier. In DualFed, we employ a global classifier for generalized representations and a personalized classifier for personalized representations. Here we conduct experiments when placing the global classifier after the projector. In these experiments, we maintained a shared encoder and investigated two configurations: sharing the projection network (DualFed-G) and personalizing it (DualFed-P). As indicated in Table 6 , removing the global classifier to the same stage as the personalized classifier Communication Costs. We assess the communication costs by using the total number of model parameters transferred to reach a predefined target accuracy during training. For PACS, DomainNet, and Office-Home, the target accuracies are set to 85%, 75%, and 70%, respectively. As illustrated in Table 8 , DualFed outperforms other methods by achieving the same target accuracy with lower communication costs, showcasing its practical efficiency. the loss balance coefficient (𝜆), and the number of local epochs (𝐸). As depicted in Figure 7 , we observe that as 𝜏 increases, its effectiveness in distinguishing between different classes diminishes, thereby losing the advantage of contrastive loss. Figure 8 presents the test accuracy with varying 𝜆. Setting 𝜆 to 0 is equivalent to training the model solely with cross-entropy loss. Increasing 𝜆 enhances the distinctiveness and relevance of personalized representations to the local task, which, in turn, improves model performance. However, as 𝜆 value exceeding a certain threshold might cause training failures. Figure 9 shows the test accuracy with different local epochs, it illustrates that DualFed consistently surpasses FedAvg across various local epochs, demonstrating its robustness to local epochs. 
conclusion: 
 In this paper, we have developed a new PFL approach called DualFed. DualFed decouples the objectives of generalization and personalization in PFL by a personalized projection network. This modification reduce the mutual interference between the conflicting optimization objectives in traditional PFL, thereby can achieve a win-win situation of both generalization and personalization in Non-IID FL.Our experiments across various datasets have shown that DualFed performs better than other FL methods, proving its effectiveness in handling the unique demands of PFL.  
 
 Appendix:  


Decision: The paper presents DualFed, a novel federated learning approach aimed at balancing model generalization and personalization by introducing a personalized projection network in the encoder-classifier architecture. Reviewers acknowledged the innovative method and its potential to address the computational overhead in federated learning, as well as the clear and coherent presentation. However, concerns were raised regarding the novelty of the method compared to existing approaches, the clarity of certain methodological details, and the non-IID experimental settings. While some reviewers leaned towards acceptance due to the method's potential and thorough experimental validation, others highlighted significant unresolved issues, leading to a mixed consensus。

After carefully reading the paper and the comments, AC recommends acceptance since the organizational issue and inherent FL problems pointed out by some reviewers are not flaws of the paper, which slightly influence the rating. AC encourages the authors to read all comments and further improve the work.

Reviews: review: paper_summary: paper_summary: This paper proposes a new federated learning method DualFed. It aims to address the computational overhead issues of previous techniques that require choosing between universal or personalized representations. DualFed generates two types of representations: a general one before projection and a more personalized one after passing through a localized projection network in the encoder-classifier architecture. By using a shared encoder and training global and personalized classifiers concurrently, while combining their outputs at inference, DualFed can leverage both cross-client collaboration and personalization. Some experimental results are included in the paper to demonstrate the performance of the proposed method.
strength_weakness: strength_weakness: 1. The method is simple and effective, able to meet the personalized needs of local clients and the generalized requirements of a central server.
2. Evaluations on mainstream datasets effectively demonstrate the effectiveness of the proposed method.
3. The overall organization of the paper is coherent and lucid, facilitating ease of comprehension.
limitations: limitations: 1. It is crucial to clarify the differentiation between this paper and DBE[1]. Both of them use only linear layers to improve personalize task performance and DBE does not require two-stage training. It raises questions regarding the novelty of this work.
2. Lines 139-143 claimed that the method needs to maintain a shared encoder. However, some key information was not discussed. Such as which layers are personalized or generalized oriented, and how to determine the location where the projection layer is inserted?  The authors are encouraged to provide further explanation to enhance clarity.
3. The authors are advised to provide further explanation about the non-IID setting in experiments. 

[1] Zhang J, Hua Y, Cao J, et al. Eliminating domain bias for federated learning in representation space[J]. Advances in Neural Information Processing Systems, 2024, 36.
paper_summary: paper_summary: This paper introduces novel federated learning method that improves generalization and personalization at the same time via hierarchical representations. Extensive experiments demonstrate the proposed method is effective.
strength_weakness: strength_weakness: 1.	Improving or balancing generalization and personalization is an important research problem in deep learning, it is novel to develop a method that tackles this problem in the challenging Non-IID FL setting.
2.	Leveraging the hierarchical representation to achieve both high model generalization and personalization at the same time is an interesting direction.
limitations: limitations: In general, the paper is well written, but I still have some questions regarding to this paper.
1.	There is one personalized classifier for each client, and one global classifier. What is the purpose of the global classifier? In what cases the output of the global classifier is different from the personalized classifier?
2.	In Eq. (7), the authors choose to use supervised contrastive loss. What if it used conventional supervised loss instead of supervised contrastive loss? Did you perform ablation studies to evaluate the difference?
paper_summary: paper_summary: This paper presents DualFed, a novel approach in personalized federated learning (PFL) that addresses the challenge of balancing model generalization and personalization. By leveraging the hierarchical nature of deep models, DualFed introduces a personalized projection network between the encoder and classifier, effectively decoupling generalized and personalized representations. This design ensures minimal interference between these conflicting objectives, leading to improved performance. Extensive experiments on datasets such as PACS, DomainNet, and Office-Home demonstrate that DualFed significantly outperforms state-of-the-art FL methods, proving its effectiveness in handling Non-IID data distributions while maintaining efficiency in communication costs.
strength_weakness: strength_weakness: The strengths of the paper lie in its innovative approach to overcoming the conflict between generalization and personalization in PFL. By introducing a personalized projection network, DualFed effectively decouples these two objectives, leveraging hierarchical representations to optimize performance. This approach not only enhances the model's ability to generalize across clients but also allows for better adaptation to local data distributions. The extensive experimental results on multiple datasets, such as PACS, DomainNet, and Office-Home, showcase DualFed's superior performance compared to other state-of-the-art methods. Additionally, DualFed demonstrates robustness in handling varying local epochs and achieves these improvements with lower communication costs, making it a highly efficient and effective solution for federated learning in Non-IID scenarios.
limitations: limitations: 1. Why not directly list the contributions in the introduction?

2. The paper organization is a little bit confusing. Why do you write the motivation in the preliminaries? When I read the introduction, I thought you have finished the motivation writing.

3. In figure 2, the forzen paratmers notations is different from the one in your picture.

4. Can you note the pre-projection representations and post-projection representations in Fig. 2?

5. According to the experiment, what is the relationship between your method and multimedia? Why not add some multimedia dataset in your experiment part?

6. It is hard to follow Table 5 because there are too many variables like D, H and BN. 

7. In Fig. 8a,  AS C, A,S still perform better as the $\lambda$ get larger. This also happens in DomainNet. It is different from what you say as $\lambda$ value exceeding a certain threshold might cause training failures

7. Why, in Fig. 9, the local epochs get larger, the accuary performs worse?
paper_summary: paper_summary: This paper introduces DualFed, aimed at enhancing both personalization and generalization in personalized federated learning. DualFed comprises pre-projection and post-projection representation stages. In the stage-wise training procedure for local clients, both the main branch and the global classifier undergo continuous training with global part aggregation, resulting in superior performance compared to other federated learning methods.
strength_weakness: strength_weakness: The paper devises a dual training process to distinctly address the generalization and personalization aspects of the model, effectively reconciling the conflicting objectives inherent in the task. Furthermore, the supplementary analysis section delves deeper into the architectural design and explores the incurred communication costs, providing a comprehensive understanding of the model's capabilities.
limitations: limitations: In this paper, certain design choices may require reevaluation or additional experimentation.  

For instance, in the training strategy, the decision to keep the encoder trainable during pre- projection representation but frozen during post-projection representation may lack justification or require empirical validation. Additionally, exploring alternative ensembling methods for both the global classifier and personalized classifier, rather than directly summing them up, might need further investigation.