Title: Adaptive Latent-Space Constraints in Personalized Federated Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Federated Learning

Maximum Mean Discrepancy

Data Heterogeneity

Deep Learning

Medical Imaging
Abstract: Federated learning (FL) is an effective and widely used approach to training deep learning models on decentralized datasets held by distinct clients. FL also strengthens both security and privacy protections for training data. Common challenges associated with statistical heterogeneity between distributed datasets have spurred significant interest in personalized FL (pFL) methods, where models combine aspects of global learning with local modeling specific to each client’s unique characteristics. This work investigates the efficacy of theoretically supported, adaptive MMD measures in pFL, primarily focusing on the Ditto framework, a state-of-the-art technique for distributed data heterogeneity. The use of such measures significantly improves model performance across a variety of tasks, especially those with pronounced feature heterogeneity. Additional experiments demonstrate that such measures are directly applicable to other pFL techniques and yield similar improvements across a number of datasets. Finally, the results motivate the use of constraints tailored to the various kinds of heterogeneity expected in FL systems.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an improvement to the Ditto framework for personalized federated learning (pFL) by incorporating adaptive latent space constraints using statistical distance measures — specifically MK-MMD and MMD-D — instead of the traditional ℓ2 norm on weights. These constraints aim to better handle feature heterogeneity by minimizing distributional drift in the latent representation space rather than simple weight differences. The authors conduct experiments on diverse datasets (CIFAR-10, Synthetic, Fed-ISIC2019, RxRx1) and show that the proposed methods offer consistent improvements, particularly under significant heterogeneity. The approach is theoretically grounded and broadly applicable to other pFL frameworks beyond Ditto.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths：\n 1. Clear Problem Focus and Innovation: The paper precisely identifies feature heterogeneity as a crucial challenge in Personalized Federated Learning (pFL). It proposes to address this by introducing adaptive, theoretically supported MMD constraints within an existing advanced framework (Ditto). This approach, building upon a strong baseline, is more convincing.\n2. Constraints in Latent Space: Compared to imposing constraints in the raw data space, applying constraints in the latent space is a more advanced and flexible way to handle feature heterogeneity. It operates on abstract representations of data, promising to capture deeper distributional relationships.\n3. Modularity and Extensibility: Although the research focuses on the Ditto framework, the paper suggests that these adaptive MMD constraints can be directly applied to other pFL settings. This indicates a modular design and good generality, facilitating broader adoption.\n\nWeaknesses:\nSee Questions below.\n\nQUESTIONS:\n1. More Experiments: Although the paper claims that the method has wide applicability, the current validation is limited to the Ditto framework. Experimental validation on other mainstream or different types of pFL frameworks would be more convincing.\n2. Privacy Considerations: The reliance on feature embeddings for regularization raises potential privacy concerns. Specifically, intermediate activations can retain semantic information about raw inputs, which—if combined with model inversion attacks or a compromised global model—may lead to input reconstruction or data leakage. \n3. Complexity Concern: How do the local computation cost and storage usage compare with Ditto and FedAvg? Is there a trade-off here?\n4. Novelty Concern: As a mature analysis method, MMD is very likely to improve PFL. Although the authors have made some improvements, I am still concerned about the novelty of the insight and technical contribution. I hope the authors can explain their novelty more clearly.\n\nI will consider increasing my score if my questions are answered.\n\nLIMITATIONS:\nSee Questions above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces adaptive statistical constraints - specially Multi Kernel Maximum-mean Discrepeancies (MK-MMD) and deep MMD (MMD-D) - to the panelty-based (e.g., Ditto) personalized federated learning (PFL) framework. It empirically demonstrates that the proposed methods improve performance in tasks with feature and label heterogeneity compared to existing feature-space and weight-based constraints. \n\nHowever, this paper’s contribution seems incremental. It essentially applies well-established statistical distance measures (MK-MMD, MMD-D) to the latent-space distributions within Ditto, without clear theoretical innovation. The justification to choose MMD is not established, and empirical results shows that performance gain from using MMD is not significant.\n\nSTRENGTHS AND WEAKNESSES:\n- Strenghts\n  - 1. This paper considers important problems in PFL, feature-drift. \n  - 2. This paper is well written and hence is easy to follow. \n\n- Weaknesses\n  - 1. The major concern of the reviewer is the novelty of the paper. MMD constraints are not new, nor is their application in deep learning contexts unusual. Merely applying these known measures to Ditto feels straightforward rather than innovative. The authors must articulate precisely what novel insight or substantial methodological development distinguishes this work from straightforward integration.\n  - 2. The authors rely heavily on empirical validation without providing theoretical conditions for why MMD constraints are the best or good to be combined with the penalty-based PFL such as Ditto. \n  - 3. Even in the emprifical validation, performance gain from using MMD is not significant. Especially, in table 2, comparing to Ditto and the proposed schemes, the gain is less than 1% in most cases (only synthetic_0.5 has the large gain).\n  - 4. While the proposed schemes has extra communications and computation (to exchange  $f_i(x;\\bar{\\theta_i})$ for all $i$ compared to the baseline (e.g., Ditto), this paper lacks computational and communication efficiency analysis, which weakens claims of practical utility.\n\nQUESTIONS:\n- 1. What is the difference of the proposed methods from the prototype based PFL? As the prototype-based PFL also apply constraints in the latent space, discussion about this should be included in the related works. \n- 2. Why should MMD be combined to the penalty-based pFL (e.g. Ditto). It seems that applying MMD is orthogonal to the penalty-based constraints. \n- 3.  In section 3.1, the authors claim that this paper proposes an unpaired approach. In MMD-D, however, feature-drift constraints depicted in figure 1 seems to be paired. Please correct me if I missed something. \n- 4. What is commputational and communication costs of the proposed schemes to additionally apply MMD on top of Ditto?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper extends the Ditto framework for personalized Federated Learning (pFL), by replacing or augmenting the way of penalizing divergence of the local model from the global model, with adaptive latent-space regularization using Maximum Mean Discrepancies (MK-MMD and MMD-D). The paper shows that using these alternative measures is beneficial in scenarios with high feature heterogeneity between clients.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n1) Building upon existing literature, the paper formally and clearly characterizes the Ditto and MMD measures used to discourage divergent model weights or feature maps in pFL.\n\n2) The proposed framework is simple and generalizable to pFL algorithms other than Ditto.\n\n### Weaknesses\n\n1) The paper proposes the application of existing measures, taking inspiration from Ditto. However, the use of MMD measures in FL applied to latent feature maps is not entirely new in the literature [1, 2], and the relationship between the proposed method and existing literature has not been sufficiently discussed.\n\n2) The related works section could be improved. Specifically, an overview of techniques for pFL based on methods other than regularization techniques, and how the proposed framework positions with respect to these methods, is absent. Due to page limitations, you can also consider expanding the related works in the appendix.\n\n3) The computational and communication costs of the proposed combinations of MMD penalty-based pFL methods have not been discussed.\n\n4) The experiment section is not convincing. Specifically, the settings considered are overly simplistic, using too few clients and vanilla models. It would be beneficial to include experiments with other widely used networks in FL, such as ResNet20 or MobileNetV2. These networks are lightweight yet well-established in the existing literature. Additionally, increasing the number of clients, ideally to at least hundreds, would provide a more realistic representation of real-world scenarios.\n\n[1] Yao, Xin, et al. \"Federated learning with additional mechanisms on clients to reduce communication costs.\"\n\n[2] Hu, Kai, et al. \"FedMMD: a federated weighting algorithm considering non-IID and local model deviation.\"\n\nQUESTIONS:\n### Questions\n\n1) In the paper, it is stated that the effectiveness of MMD measures is highly sensitive to the choice of kernel; however, only the RBF kernel is evaluated in the experimental section. Would it be possible to provide more insights on the choice of kernels other than RBF?\n\n2)  ⁠Why have Ditto results not been included in Table 1? A comparison with Ditto may be beneficial to interpret results.\n\n3) Could the authors please elaborate more on what they mean by the increasing advantages of using MMD measures in an iterative training process in the sentence in rows 178-179?\n\n### Minor suggestions\n\n1) The introduction should give more relevance to MMD-D and MK-MMD. Additionally, references to other algorithms in rows 39-45 may be included and better characterized in the related work section, while symbols should be introduced in the methodology section.\n\n2) The methodology section provides an overview of the Ditto algorithm and the MMD measures, which is acceptable. However, paragraph 3.5 should provide a more thorough description of the main innovative contributions of the proposed approach, including how it could be adapted or integrated into pFL methods, as well as its advantages.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes and investigates the application of adaptive Maximum Mean Discrepancy (MMD) measures, specifically Multi-Kernel MMD (MK-MMD) and MMD-D, as regularization terms within the Ditto framework for personalized FL. The primary contribution lies in exploring these theoretically supported statistical distance measures to constrain feature representation divergence between local and global models. The paper conducts extensive experiments across various datasets, including CIFAR-10, a synthetic dataset, Fed-ISIC2019, and RxRx1, demonstrating that these MMD-based penalties, either alone or combined with the traditional Ditto penalty, yield improved model performance, particularly in settings with pronounced feature heterogeneity.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper presents a technically sound exploration of adaptive MMD measures in personalized FL. The methods are clearly articulated, which provides mathematical formulations for MMD, MK-MMD, and MMD-D, and details their integration into the Ditto algorithm. \n\n2. The experimental setup is comprehensive. It utilizes both synthetic and real-world clinical datasets with varying levels of heterogeneity, which adequately tests the proposed methods. \n\n3. The results are systematically presented and analyzed. This highlights the conditions under which the MMD-based constraints are most effective. \n\n4. Ablation studies support the claims regarding kernel optimization and the benefits of a single constraint layer.   \n\nWeaknesses:\n\n1- The computational overhead associated with the iterative optimization of kernels for MMD measures. \n\n2- The paper lacks a theoretical framework that quantifies the specific conditions under which the proposed MMD measures are most effective. \n\n3. The paper primarily focuses on the Ditto framework, and the general applicability of these MMD measures to other personalized FL methods, such as FedProx and MR-MTL, is suggested but not empirically validated in detail.   \n\n4. The experimental results are reported as the average of only three runs. This would limit the ability to fully assess the statistical significance and variability of the reported performance improvements.\n\nQUESTIONS:\n1. Could you elaborate on the criteria used for selecting the optimal depth for MMD constraints, particularly for datasets other than CIFAR-10?  This is mainly because the paper states that for CIFAR-10, the optimal depth was uniformly 1. A more detailed analysis or discussion on why deeper constraints did not yield better results, and whether this observation generalizes to all datasets, would be valuable.   \n\n2. The paper mentions that for some datasets (Synthetic and RxRx1), including the Ditto penalty with MMD-based measures, degrades accuracy or hurts performance across the board. Could you provide a more in-depth explanation or theoretical intuition for this observation? \n\n3. Regarding the computational overhead of kernel optimization procedures, the paper acknowledges this as a limitation. Could the authors suggest specific directions for future work to reduce this overhead? \n\n4. Could the authors consider including standard deviations for the reported average performance metrics in tables, especially for key results, as this would give a better sense of the consistency of the improvements?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}