Title: Learning Reconfigurable Representations for Multimodal Federated Learning with Missing Data
Venue: NeurIPS 2025
Accepted: yes
Keywords: Federated Learning

Modality Missing

Multimodal Learning
Abstract: Multimodal federated learning in real-world settings often encounters incomplete and heterogeneous data across clients. This results in misaligned local feature representations that limit the effectiveness of model aggregation. Unlike prior work that assumes either differing modality sets without missing input features or a shared modality set with missing features across clients, we consider a more general and realistic setting where each client observes a different subset of modalities and might also have missing input features within each modality. To address the resulting misalignment in learned representations, we propose a new federated learning framework featuring locally adaptive representations based on learnable client-side embedding controls that encode each client’s data-missing patterns.

These embeddings serve as reconfiguration signals that align the globally aggregated representation with each client's local context, enabling more effective use of shared information. Furthermore, the embedding controls can be algorithmically aggregated across clients with similar data-missing patterns to enhance the robustness of reconfiguration signals in adapting the global representation. Empirical results on multiple federated multimodal benchmarks with diverse data-missing patterns across clients demonstrate the efficacy of the proposed method, achieving up to 36.45\% performance improvement under severe data incompleteness. The method is also supported by a theoretical analysis with an explicit performance bound that matches our empirical observations. Our source codes are provided at [https://github.com/nmduonggg/PEPSY](https://github.com/nmduonggg/PEPSY)


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of multimodal federated learning (MMFL) in the presence of incomplete and heterogeneous data, where clients may have access to different subsets of modalities and suffer from partial feature missingness within each modality. The authors propose PEPSY, a novel federated learning framework that introduces learnable client-specific embedding controls to capture and communicate each client’s unique data-missing patterns. These embeddings serve as reconfiguration signals that align the global model with each client's local context, enabling more effective and robust representation learning.\n\nThe framework operates by:\n1.\tExtracting modality-specific and data-specific features on the client side.\n2.\tUsing a query-key mechanism to select relevant embeddings from a shared pool that reflect the missing patterns.\n3.\tAggregating embedding controls across clients via a probabilistic clustering-based alignment on the server.\n\nTheoretical analysis provides performance bounds showing that PEPSY’s design minimizes prediction error due to missing modalities. Empirical evaluations on PTBXL and Sleep-EDF datasets under IID and Non-IID settings demonstrate significant improvements over baselines, achieving up to 36.45% accuracy gains in high missingness scenarios.\n\nSTRENGTHS AND WEAKNESSES:\n## **Strengths**\n\n### **Quality**\n- The method addresses an important and realistic challenge in multimodal federated learning (MMFL): handling both *missing modalities across clients* and *missing features within modalities*. The proposed PEPSY framework is designed to tackle this dual challenge and is evaluated through solid empirical experiments.\n\n### **Clarity**\n- The paper clearly motivates the real-world relevance of MMFL under partial observability and articulates the limitations of prior work. The distinction between different types of \"missing-ness\" (modality-level and feature-level) is helpful for framing the contribution.\n\n### **Significance**\n- The addressed problem is important for real-world applications involving distributed sensing and privacy-preserving learning, particularly in domains like healthcare and IoT. The strong empirical gains and general design framework make this work a promising direction for advancing robust and flexible MMFL.\n\n### **Originality**\n- While the paper proposes a novel solution, there is some conceptual proximity to existing work:\n  - **Personalized Federated Learning (PFL)**: Many PFL approaches use local embeddings or client-specific signals to personalize global models. PEPSY similarly leverages client-side latent information via embedding control pools.\n  - **Domain Adaptation**: The idea of learning latent codes to adapt shared models to local contexts mirrors approaches in domain adaptation, especially under distribution shifts.\n---\n\n## **Weaknesses**\n\n### **Clarity**\n\nWhile the paper presents a compelling high-level idea, the technical exposition—especially in **Section 2**—is difficult to follow. Several core concepts and mathematical formulations lack clarity or motivation, making it hard to understand the method in depth. Specific issues include:\n\n- **Data-Missing Profile Ambiguity (Section 2.1–2.2.2)**  \n  It is unclear how the data-missing profile $\\Psi$ is computed or updated, whether it is global or client-specific, and how it's initialized. The absence of a client index in the notation adds confusion, especially given the claim that the profile reflects local missingness patterns.\n\n- **Equation Transitions (Section 2.1)**  \n  The transition from Eq. (1) to Eq. (3) introduces a new utility term $u_k$ without sufficient justification. Its role in the optimization problem, and why the objective is now a difference of two terms, is not explained intuitively or mathematically.\n\n- **Inconsistent Notation (Section 2.1)**  \n  In the expression for $u_k(\\theta, \\Psi)$, the right-hand side omits $\\theta$, though it appears to depend on model parameters via the relevance function. This creates notational confusion in a key part of the formulation.\n\n- **Unclear Objectives (Section 2.1)**  \n  The claim that minimizing $\\ell_k(\\theta, \\Psi)$ helps extract data-missing features, and that maximizing $u_k(\\theta, \\Psi)$ helps $\\Psi$ adapt to local context, is not justified or supported by theory or experiments.\n\n- **Vague Terminology (Section 2.1–2.2.2)**  \n  Phrases like “reconfigured by $\\Psi$” or “diluting learned data-missing profile” are conceptually vague and lack clear operational definitions, making the method harder to interpret.\n\n- **Embedding Controls Design (Section 2.2.2)**  \n  The identity function is used as the key function $k(\\cdot)$ in a query-key matching setup, with no justification for its effectiveness. Also, the decision to limit relevance to only $\\kappa$ embeddings is explained in abstract terms without empirical motivation or ablation.\n\n- **Contrastive Learning Justification (Section 2.2.3)**  \n  The use of contrastive loss for reconfiguration regularization is introduced abruptly. It is unclear why this choice is suitable for ensuring “data completeness,” and the sampling of positives and negatives is not specified.\n\n- **Definition of $h_{di}$ (Section 2.2.1)**  \n  The modality-specific feature $h_{di}$ is mentioned as a key component in representation construction, but its computation is unspecified. There’s no mention of what neural layers produce it or how it fits into the overall architecture.\n\n- **Server Aggregation with PFPT (Section 2.3)**  \n  The server uses PFPT (probabilistic federated prompt tuning) for aggregating data-missing profiles, but this method is not described. Readers unfamiliar with PFPT are left without context on its mechanism, benefits, or why it was chosen over alternatives.\n\n### **Quality**\n\n- **Computational Complexity**  \n  The method comprises several sophisticated components—embedding control pools, query-key matching, contrastive and alignment losses, and probabilistic clustering (PFPT). These likely incur nontrivial computational and communication overhead, but the paper does not discuss the system cost or efficiency.\n\n- **Limited Dataset Variety**  \n  Although performance on PTBXL and Sleep-EDF is promising, both datasets are from biosignal/health domains. Broader validation on more diverse multimodal settings (e.g., video+audio, language+vision) would better support the generality of the approach.\n\n### **Significance**\n\n- **Strong Theoretical Assumptions (Section 3)**  \n  Theorem 3.1 assumes that the model is $\\mu$-Lipschitz continuous—a strong condition not typically satisfied by deep networks. The paper does not discuss whether this assumption is realistic or how sensitive the bound is to its violation, limiting the practical applicability of the theoretical result.\n\n### **Originality**\n\n- **Conceptual Overlap with Prior Work (Section 2.1–2.2.2)**  \n  While PEPSY presents a technically novel mechanism, its conceptual structure shares similarities with:\n  - **Personalized Federated Learning (PFL):** where client-specific embeddings or local statistics are used to personalize global models.\n  - **Domain Adaptation:** where shared models are adapted using domain-specific latent codes.\n\n  To more clearly distinguish the contribution, the authors could:\n  - Emphasize the novelty of *cross-client collaboration via clustering* of embedding profiles.\n  - Provide explicit empirical and theoretical comparisons to relevant PFL and domain adaptation baselines.\n\nQUESTIONS:\nBefore addressing specific questions, I strongly encourage the authors to review the **Clarity** section under **Weaknesses**, particularly issues raised in Sections 2.1–2.3. Several definitions, notational choices, and design rationales lack sufficient explanation. Addressing these concerns would significantly improve the comprehension of the paper and may positively influence the overall assessment.\n\nBelow are the key questions that, if clarified, could strengthen the paper further:\n1. **Clarify the Construction and Scope of the Data-Missing Profile ($\\Psi$)**  \n   How is the data-missing profile $\\Psi$ initialized and updated during training? Is there a single global profile, or does each client maintain a local version? Why is there no client index in the notation if the profiles encode client-specific missingness patterns?\n\n2. **Explain and Justify the Optimization Formulation (Eq. 3)**  \n   Why is the original federated loss in Eq. (1) replaced with a difference involving the utility term $u_k(\\theta, \\Psi)$? What does this subtraction represent, and how should it be interpreted? \n\n3. **Clarify the Use of Contrastive Learning for Reconfiguration (Section 2.2.3)**  \n   Why is contrastive loss suitable for aligning partial and full-modality representations? What exactly are the positive and negative sample pairs used during training?\n\n4. **Describe PFPT and Its Role in Server Aggregation (Section 2.3)**  \n   What are the main characteristics of PFPT, and why is it an appropriate choice for aggregating the data-missing profile across clients?\n\n5. **Discuss the Lipschitz Continuity Assumption in Theorem 3.1**  \n   Given that neural networks rarely satisfy strict Lipschitz conditions, how practical is this assumption? Can the bound be extended or relaxed under more realistic constraints?\n\n6. **Clarify the Definition of \"Modality\" Used in the Dataset Description**  \n   The paper states that PTBXL has 12 modalities and Sleep-EDF has 5, but these datasets are generally not considered to contain that many distinct modalities under standard definitions (e.g., text, audio, image). Could the authors clarify what they count as a modality in this context? \n\n7. **Provide an Analysis of Computational Complexity**  \n   The proposed method introduces multiple components—embedding control pools, query-key relevance selection, contrastive regularization, and PFPT-based aggregation. Can the authors provide an estimate or breakdown of computational and communication overhead introduced by each component?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
The paper studies multimodal federated learning under both missing modalities and missing features. The authors proposed a reconfiguration based framework with a data-missing embedding profile. The missing data will be carefully reconstructed based on the profile, which will be then fused with the available data. There is also a theoretical analysis on the convergence behavior of the model under missing data. Comprehensive experiments agains SOTA baselines show accuracy improvements under severe data incompleteness.

STRENGTHS AND WEAKNESSES:
Strength:
* The paper is organized clearly and easy to follow
* The proposed idea of using learnable client-side embedding controls makes sense and is interesting
* Comprehensive experiments are conducted under different missing statistics.

Weakness:
* The overhead of the method compared to existing baselines are not analyzed
* The sensitivity of parameter k, which controls the size of the neighborhood in embedding control, is not analyzed

QUESTIONS:
* What is the overhead of the method compared to existing baselines?
* How will k, which controls the number of relevant controls selected for each instance, affect the performance?

LIMITATIONS:
Would be good to discuss the limitations in the main paper instead in the appendix
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to dynamically reconstruct the representation of each client through embedding controls, enabling the global model to maintain stable performance when faced with multiple complex data missing patterns (missing modality、missing feature).\n\nSTRENGTHS AND WEAKNESSES:\nStrength：\n\n（1）This paper addresses the issues of modality missing and feature missing under multimodal federated learning.\n\n（2）Model missing data as structured vectors and enable their sharing within a federated framework.\n\n（3）This paper theoretically analyzed the impact of data missing on prediction output and proposed a clear upper bound of error, enhancing the credibility of the method.\n\nWeakness：\n\n（1）\tAlthough the paper uses PFPT to aggregate the embedding controls of each client, the order and semantics of the controllers learned by each client are not consistent. Does this paper perform semantic interpretability or consistency checks on the embeddings after clustering? If different clients represent different missing semantics, would blind aggregation compromise representation quality?\n\n（2）\tThe controller selection mechanism is based on cosine similarity matching. Does this similarity matching seek to find a specific controller that matches the current client's missing pattern?\n\n（3）\tWithout providing the code, it is incredible that performance can be maintained at 16% even with 80% missing data. Which module do you think can maintain model performance in a severely missing data scenario and provide the corresponding basis?\n\n（4）\tIn the study of multimodal federated learning scenarios, there has been some work on cross-modal and missing modality, but this paper does not provide a thorough review of related work. To my knowledge, there is also some FL+ missing modality work that considers the use of parameters from other clients.\n\nQUESTIONS:\nPlease refer to the weakness.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the issue of incomplete and heterogeneous data in multimodal federated learning (MMFL), where clients may possess different modality subsets and suffer from missing features within modalities. The authors propose a novel framework called PEPSY, which leverages learnable client-side embedding controls to encode data-missing patterns and generate locally adaptive representations. These embeddings serve as reconfiguration signals to align globally aggregated representations with each client’s local context, and embeddings from clients with similar missing patterns are algorithmically aggregated to enhance the robustness of reconfiguration signals for global representation adaptation. The key contributions include: developing the PEPSY framework to handle both modality and feature missing scenarios by integrating missing-pattern information into representation reconfiguration without raw data sharing; establishing a theoretical performance bound via theorem proof to demonstrate PEPSY’s stability under random missing patterns by minimizing prediction discrepancies between complete and missing modality cases; and empirically validating that PEPSY achieves up to 36.45% performance improvement over baselines on federated multimodal benchmarks (PTBXL and SleepEDF) under severe data incompleteness, showcasing robust performance in both IID and Non-IID settings. This work provides an effective solution for missing data in MMFL, addressing the gap in existing methods that fail to handle combined modality and feature missing challenges.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**  \n1. The PEPSY framework is proposed to address the co-occurrence of modality and feature missing in multimodal federated learning. It encodes data-missing patterns through client-side embedding controls, enabling global aggregated representations to adapt to the local context of each client, which fills the gap of traditional methods in handling mixed missing scenarios.  \n2. Experiments validate the effectiveness of PEPSY on PTBXL and SleepEDF datasets, achieving up to 36.45% improvement over baselines under severe data incompleteness. It demonstrates robustness to heterogeneous missing patterns in both IID and Non-IID settings, with theoretical analysis consistent with empirical results.  \n\n**Weaknesses**  \n1. The theoretical analysis relies on the assumption of μ-Lipschitz continuity of the model, failing to cover non-smooth data distributions. It also does not clarify the performance bounds in extreme missing scenarios (e.g., only single modality available), and the correlation between the dimension of embedding controls and the complexity of missing patterns lacks theoretical interpretation.  \n2. The dynamic selection of client-side embedding controls and server-side PFPT probabilistic clustering may lead to a surge in communication overhead in large-scale federated scenarios (e.g., thousands of clients). The weight coefficients of contrastive losses (λ, η) require manual tuning, and the cross-domain application lacks an automated adaptation mechanism.  \n3. The experiments generate binary missing matrices with fixed probabilities, failing to simulate continuous missing in time-series data or inter-modality dependencies (such as synchronous audio-visual missing). Additionally, it does not verify the adaptability to newly added modalities after training, limiting the expandability in dynamic IoT scenarios.\n\nQUESTIONS:\n1. The theoretical analysis in the paper is based on the assumption of μ-Lipschitz continuity of the model, which may not fully cover real-world scenarios with complex and irregular data distributions. Additionally, the performance boundaries in extreme missing scenarios (such as when only a single modality is available) are not clearly demonstrated. It is recommended that the authors relax the theoretical assumptions by introducing more universal continuity conditions or validate the model's performance in extreme missing scenarios through experiments simulating complex data distributions, so as to enhance the practical applicability of the theory.  \n\n2. The dynamic selection of client-side embedding controls and the server-side PFPT probabilistic clustering algorithm are likely to cause a significant increase in communication overhead in large-scale federated learning scenarios with numerous clients, but the paper does not provide a detailed analysis of the computational efficiency in this regard. It is suggested that the authors supplement the communication complexity assessment in large-scale scenarios or propose targeted lightweight optimization strategies, such as sparse embedding selection or more efficient clustering algorithms, to improve the feasibility of model deployment in practical large-scale environments.  \n\n3. The current experiment uses a binary missing matrix generated with fixed probabilities, which fails to fully simulate real-world continuous missing patterns (such as prolonged sensor failures) or inter-modality dependencies (where the absence of one modality may affect the validity of another). It is recommended that the authors introduce more realistic complex missing pattern generation mechanisms in future experiments, such as simulating continuous missing based on temporal dependencies or considering potential correlations between modalities, to verify the model's adaptability in more authentic scenarios.  \n\n4. The need for manual tuning of contrastive loss weights (λ, η) may severely limit the model's automated deployment and application across different domains. It is suggested that the authors explore integrating automated tuning mechanisms, such as Bayesian optimization or model-based meta-learning, and optimize the tuning strategy through testing on cross-domain datasets to enhance the model's convenience and performance stability in various application domains.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}