Title: Enhancing Privacy in Multimodal Federated Learning with Information Theory
Venue: NeurIPS 2025
Accepted: yes
Keywords: Federated Learning

Multimodal Learning

Privacy Protection

Information Theory
Abstract: Multimodal federated learning (MMFL) has gained increasing popularity due to its ability to leverage the correlation between various modalities, meanwhile preserving data privacy for different clients. However, recent studies show that correlation between modalities increase the vulnerability of federated learning against Gradient Inversion Attack (GIA). The complicated situation of MMFL privacy preserving can be summarized as follows: 1) different modality transmits different amounts of information, thus requires various protection strength; 2) correlation between modalities should be taken into account. This paper introduces an information theory perspective to analyze the leaked privacy in process of MMFL, and tries to propose a more reasonable protection method \textbf{Sec-MMFL} based on assessing different information leakage possibilities of each modality by conditional mutual information and adjust the corresponding protection strength. Moreover, we use mutual information to reduce the cross-modality information leakage in MMFL. Experiments have proven that our method can bring more balanced and comprehensive protection at an acceptable cost.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work mitigates modality-specific vulnerabilities in MMFL through risk assessment based on conditional mutual information, adaptive noise allocation, and cross-modal decoupling, with multi-dataset experiments demonstrating significant privacy gains.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Novel risk-aware protection framework grounded in information theory.  \n- Effective cross-modal leakage suppression methodology.  \n- Empirically validated across heterogeneous datasets and privacy budgets.\n\nWeaknesses:\n- The privacy protection effectiveness is mainly measured by PSNR and TRR, which might not fully capture the privacy implications in complex real-world scenarios. More comprehensive metrics could strengthen the evaluation.\n- There is limited exploration of the impact of different model architectures on the performance of Sec-MMFL. The framework's effectiveness might vary depending on the choice of encoders and fusion mechanisms.\n-  Omits discussion about potential risks: adaptive noise profiles could themselves leak modality sensitivity, creating new attack surfaces.\n- Inconsistent terminology appears in:\n  - Line 268: \"information leakage risk among different modes\" (should be \"among different modalities\")\n  - Line 286: \"which will make the text mode protection too strong\" (should be \"which will make the text modality protection too strong\")\n  - Line 288: \"make the protection effect of the two modes more balanced\" (should be \"make the protection effect of the two modalities more balanced\")\n\nQUESTIONS:\n1. The paper mentions the use of early and late fusion strategies in the experiments. Could you explain how the proposed method's effectiveness might vary depending on the fusion strategy used?\n2. The paper mainly uses LeNet-5 and TextCNN as encoders. How would the choice of more complex or different model architectures affect the performance of Sec-MMFL in terms of both privacy protection and model utility?\n3. Could Sec-MMFL's adaptive noise profiles be reverse-engineered to identify high-risk modalities? If so, what countermeasures would prevent attackers from exploiting this meta-information?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSec-MMFL introduces an information theory based framework that enhances privacy in multimodal federated learning by dynamically quantifying leakage risks of each modality via conditional mutual information, adaptively scaling noise budgets, and reducing dependencies across different modalities, with validation across diverse datasets showing improved privacy-utility tradeoffs.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. Establishes novel modality-specific protection, validated through evidence of text/image vulnerability asymmetry.\n\n2. Conditional mutual information provides a rigorous theoretical foundation for risk quantification correlated with attack outcomes.\n\n3. Multiple datasets testing under varied privacy budgets demonstrates consistent improvements with clear workflow visualization.\n\n\n\nWeaknesses:\n\n1. Lack of details on the implementation of the conditional mutual information calculation,  such as the specific algorithms or approximations used to estimate the conditional mutual information.\n\n2. While the framework shows promising results for image-text systems, its applicability to other modality combinations like audio-video remains unexplored. \n\n3. The paper does not provide detailed comparisons with other state-of-the-art privacy protection methods specifically designed for multimodal federated learning, making it difficult to fully assess its relative advantages.\n\nMinor suggestions:\n\nline 125:\" We consider each client k can only access to his local private dataset\" -> \"We consider that each client k can only access their local private dataset\".\n\nline 138: \"GIA exploit the information encoded in the gradients\"  -> \"GIA exploits the information encoded in the gradients\".\n\nThe complexity of the sentence \"The whole process of data of each modality in MMFL flowing to the attacker through GIA is shown in Figure 2.\" in line 145 could present readability challenges. Suggest simplifying it as follows: \"Figure 2 illustrates the flow of data from each MMFL modality to attackers via GIA.\"\n\nQUESTIONS:\n1. Could you provide more details on the specific implementation of the conditional mutual information calculation? For example, what specific algorithms or approximations are used to estimate the conditional mutual information between data of each modality and the gradients?\n2. How might Sec-MMFL be adapted for other modality combinations, like audio-video modalities? Is it possible to outline a potential approach using existing datasets like CREMA-D?\n3. Would it be possible to include brief benchmarking against recent MMFL defenses like FedMultimodal?\n4. For GIA attacks, how are pseudo inputs initialized? When applying LDP noise, did you clip gradients before or after adding noise?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nRecognizing that different modalities exhibit varying vulnerabilities to adversarial attacks and that inter-modal data correlations can expand the scope of privacy leakage across modalities, this paper proposes an information-theoretic framework for optimal privacy budget allocation across modalities while mitigating privacy risks arising from cross-modal dependencies.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n1. The topic of privacy protection in multimodal federated learning is timely, and the integration of information theory is interesting.\n2. The experimental evaluation provides comprehensive comparisons that demonstrate the effectiveness of the proposed method.\n3. The paper is well-written with clear and thorough introductions to the background and related work, making it accessible to readers.\n\nWeakness:\n1. The paper lacks sufficient mathematical rigor in key derivations. For instance, there is no clear explanation of how the allocation scheme in Equation (10) guarantees that the total privacy budget equals $\\epsilon$, which is a fundamental requirement for differential privacy.\n2. Several technical details are inadequately explained, leading to ambiguity in the methodology. Specifically, given that different modalities employ distinct encoder architectures, it remains unclear how the gradient mutual information in Equation (12) is computed across heterogeneous network structures.\n3. The evaluation relies on somewhat outdated attack methods, which may not reflect the current threat landscape. The paper would benefit from more comprehensive experiments incorporating state-of-the-art privacy attacks to strengthen the validity of the proposed defense.\n4. From Table 2, it seems that the improvement over the existing approaches is rather marginal. The improvement in privacy is obtained with a degradation in test accuracy.\n5. There are some typos. For example, in lines 253-254, the text encoder should be TextCNN, and the image encoder LeNet-5.\n\nQUESTIONS:\n1.\tHow is the entropy of gradients computed in Equation (8)? Given that encoders across different modalities have distinct architectures, the resulting gradients will have different dimensions. How are these heterogeneous gradients aggregated or represented when computing entropy across all modalities?\n2.\tHow does the allocation scheme in Equation (10) guarantee that the total privacy budget equals $\\epsilon$?\n3.\tSimilar to Question 1, given the difference in gradient dimensions across modalities, how is the mutual information between different modalities calculated in Equation (12)? What preprocessing or alignment steps are applied to enable meaningful mutual information computation?\n4.\tWhat is the rationale for selecting the hyperparameter $\\lambda$? The paper appears to lack discussion on $\\lambda$'s sensitivity analysis or selection criteria in the experimental section.\n5.\tWhy were the most recent attack methods, such as those referenced in [40], not included in the experimental evaluation? The current attack methods appear to be from publications approximately five years ago, which may not adequately represent the evolving threat landscape in federated learning privacy attacks.\n\nLIMITATIONS:\nThe limitations are not explicitly discussed, but the authors mentioned some of them in the checklist.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an information-theoretic approach to analyze privacy leakage in the process of MMFL. The proposed Sec-MMFL uses conditional mutual information to quantify the information leakage risk of each modality and dynamically adjusts the corresponding protection strength. Experiments on benchmark datasets demonstrate that Sec-MMFL outperforms traditional methods.\n\nSTRENGTHS AND WEAKNESSES:\nThis paper focuses on the issue of MMFL and cross-modal leakage, which is an interesting topic. The proposed privacy-preserving approach is tailored to the characteristics of MMFL, with clear research objectives. However, the current comparative methods do not adequately cover multimodal baselines. The claimed solutions are not clearly validated or demonstrated qualitatively.\n\nQUESTIONS:\nThe paper highlights that \"different modalities require varying protection strengths\" as a key motivation, but what is the specific solution and validation of this problem? Can conditional mutual information guarantee such a balance across modalities? More qualitative analyses on this issue would likely facilitate an intuitive understanding of the proposed model.\n\nThe current comparative methods do not adequately cover multimodal baselines. For instance, methods like LDP-Fed included in the comparisons are not designed for multimodal scenarios, making it difficult to effectively validate the advantages of the proposed method in MMFL settings. Although this issue is acknowledged in the limitations, mere textual mention cannot compensate for the lack of experimental evidence. This weakens the support for the claim that the proposed method concentrates on multimodal problems. It is recommended to supplement comparative experiments with mainstream multimodal methods to clarify the performance boundaries and advantages of the proposed approach.\n\nThe model framework (Figure 3) is hard to follow. It is claimed that \"the Noise Scale Adjustment module adjusts the privacy budget assigned to each modality, achieving a better balance between privacy protection and model effectiveness,\" yet it does not seem to illustrate how this balance is achieved.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}