Enhancing Privacy in Multimodal Federated Learning with Information Theory
Authors: Tianzhe Xiao
Yichen Li
Yining Qi
YI LIU
wangshi.ww
Haozhao Wang
Yi Wang
Ruixuan Li
Venue: NeurIPS 2025 poster ()
Keywords: Federated Learning
Multimodal Learning
Privacy Protection
Information Theory
Primary area: other

Abstract: Multimodal federated learning (MMFL) has gained increasing popularity due to its ability to leverage the correlation between various modalities, meanwhile preserving data privacy for different clients. However, recent studies show that correlation between modalities increase the vulnerability of federated learning against Gradient Inversion Attack (GIA). The complicated situation of MMFL privacy preserving can be summarized as follows: 1) different modality transmits different amounts of information, thus requires various protection strength; 2) correlation between modalities should be taken into account. This paper introduces an information theory perspective to analyze the leaked privacy in process of MMFL, and tries to propose a more reasonable protection method \textbf{Sec-MMFL} based on assessing different information leakage possibilities of each modality by conditional mutual information and adjust the corresponding protection strength. Moreover, we use mutual information to reduce the cross-modality information leakage in MMFL. Experiments have proven that our method can bring more balanced and comprehensive protection at an acceptable cost.

Enhancing Privacy in Multimodal Federated Learning with Information Theory
Tianzhe Xiao', Yichen Li Yining Qi', Yi Liu?, Wei Wang?,
Haozhao Wang , Yi Wang". Ruixuan Li School of Computer Science and Technology. Huazhong University of Science and Technology, Wuhan, China CChongiing Ant Consumer Finance Co., Ltd, Ant Group, Chongqing, China (d202381469, ycli0204, qiyining, hz_wang, rxli)(hust.eddi
Abstract
Multimodal federated learning (MMFL) has gained increasing popularity due to its ability to leverage the correlation between various modalities, meanwhile preserving data privacy for different clients. However, recent studies show that correlation between modalities increase the vulnerability of federated learning against Gradient Inversion Attack (GIA). The complicated situation of MMFL privacy preserving can be summarized as follows: 1) different modality transmits different amounts of information, thus requires various protection strength; correlation between modalities should be taken into account. This paper introduces information theory perspective to analyze the leaked privacy process of MMFL, and tries to propose more reasonable protection method Sec-MMFL. based on assessing different information leakage possibilities of each modality by conditional mutual information and adjust the corresponding protection strength. Moreover, we use mutual information to reduce the cross-modality information leakage in MMFL. Experiments have proven that our method can bring more balanced and comprehensive protection at an acceptable cost.
Introduction
Federated Learning (FL) has emerged as a fundamental paradigm that enables collaborative model training among multiple parties via parameter aggregation without sharing private datasets [31
27 14.
Owing to its privacy-preserving and communication--fficci
nature, FL. has been widely
deployed in diverse applications, including smart healthcare [ 33 financial analysis [50] 30
and recommendation system (22). Some of the major research directions in FL also include efficient
aggregation (45,, communication compression 20 19 [8]. continual learning 24 25 2 23 and
knowledge distillation 44
However, FL is not immune to security threats. A notable attack, Gradient Inversion Attack (GIA), aims to infer sensitive information from the shared model updates (gradients) [26]. Malicious participants can exploit this reconstruct private data infer its properties, thereby breaching local
privacy 55 56]
To counter this risk, several defense methods have been proposed in FL 46 15]. LDP-Fed 43
optimizes local differential privacy (LDP) for FL, ensuring lightweight and quantifiable privacy measure. In (7), regularization and sparsification techniques are employed to alleviate performance degradation with user-level DP. FedDPA (5I) explores differential privacy personalized FL through dynamic Fisher personalization and adaptive constraints, while PrivateRec (28) focuses on federated recommendation, aiming to improve model utility under DP guarantees.
Previous FL approaches are mainly trained using modal data, yet real-world data is often multi- modal. For example, videos typically come with audio tracks and text subtitles, and internet content
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

like social media posts and news articles often blend text, images. videos, and audio. Multi-modal federated learning (MMFL) tasks such as image annotation, visual question answering, and image- text retrieval leverage complementary information from different modalities, resulting in global model that outperforms uni-modal counterparts
(9
While existing defense approaches may work well for uni-modal FL, they struggle in MMFL for two key reasons:
First, applying the same pro- tection strength across differ- ent modalities is inappropri- ate. Different modalities have distinct representation formats and reconstruction challenges. For instance, image data is typically represented as high- dimensional pixel matrices (c.g.,
Image
Insipendent Modalty
Cross Modally
300 attack tterations
Image
Image
image
sublerfugg loses pistal school survived
A cattle the image
A cattie the imago
224x224 image contains roughly 150,000 parameters) with continuous, smoothly vary- ing features that require coordi- nated recovery of multiple pixels to restore semantic content. In contrast, text data is represented by discrete words or embed- dings (e.g., Word2 Vec or BERT), where recovering key words usu- ally suffices to retain most se- mantic information. As shown in Figure [ after 300 rounds of attack starting from random pseudo data, the recovered text is completely consistent with the original, whereas the recovered image remains blurry. Uniform protection can therefore either over-distort one information and reducing accuracy under-protect another, leaving gradients vulnerable leakage. This discrepancy in data representation demands a nuanced approach to protection. By carefully calibrating the level of noise perturbation for each modality, vation and performance. In practice, determining the optimal protection strength requires deep understanding of each modality's inherent characteristics.
Text
Initial Peeudo Text
Recovard
Figure I: Different recovery rates of data from various modali- ties under attack. For each independent modality, starting from random pseudo data, after 300 rounds under the same setting, the recovered text perfectly matches the original, while the recovered image remains blurry. However, when we let the information of each modality interact to carry out cross modality attacks, under the same attack iterations, the images can be recovered more simi- lar using the information recovered from text modality.
the fusion of useful multi-
better balance privacy preser-
Second, inter-modal data correlation can intensify the impuct of GIAs. In MMFL, training data for different modalities paired, leading to "barrel effect" where the most vulnerable modality breached first, accelerating leakage across all modalities. Models typically align features from different modalities (c.g., images and text) into common space capture their relationships. This alignment increases cross-modal semantic associations in the gradients, enabling attackers to infer more precise information and even deduce the content of one modality from another 29 Furthermore, the close coupling of modalities means that even a minor breach in one channel may compromise the security of the entire system. Understanding and mitigating these risks is essential developing robust privacy-preservin techniques in MMFL. Such cross-modal alignment thus broadens the scope of privacy leakage.
To address these challenges, we propose Sec-MMFL, which secures MMFL both effectively and efficiently. We begin by analyzing the causes and flow of training information leakage in MMFL. Integrating information theory, we propose a method to measure leakage risk across modalities, allowing for modality-specific protection strengths. Additionally, correlations on privacy leakage and develop methods to mitigate the heightened risk of cross-modal gradient inversion attacks without sacrificing training accuracy.
study the impact of inter-modal
We analyze the causes of training information leakage in MMFL and propose a novel information-theorett framework to quantify leakage risk across modalities, allowing to assign appropriate protection based on each modality's vulnerability to gradient inversion attacks.

---

We introduce Sec-MMFL, an adaptive protection framework that assigns modality-speciiff protection strengths to mitigate privacy risks from inter-modal correlations while maintaining high model performance.
Extensive experiments on benchmark datasets like CIFAR-IO, CIFAR-100, Hateful-Memet and CrisisMMD demonstrate that Sec-MMFL. outperforms traditional methods under equiv- alent privacy guarantees, effectively balancing privacy preservation and model utility.
2 Related Work
2.1 Multimodal Federated Learning
Multimodal federated learning (MMFL) has emerged promising paradigm that synergizes the nature of federated learning [8] [6] with the representational power of multimodal
learning 33 34 This framework has demonstrated significant potential in real-world applications
ranging from affective computing [ 33 to distributed healthcare systems [42 [3], particularly through its deployment in IoT sensor networks [35 37 [58]. The fusion methodology MMFL typically operates through three principal approaches: early fusion that combines raw feature rep- resentations, late fusion that aggregates model outputs, and hybrid strategies that integrate both paradigms 2 10 12]. Our investigation focuses on the fundamental dichotomy between early and mechanisms.
late fusion to elucidate their distinct impacts on
2.2 Gradient Inversion Attack
The security vulnerabilities of federated learning systems have been extensively documented, with gra- dient inversion attacks [49] representing one of the most potent threats to data privacy. These attacks exploit the mathematical properties of shared gradients to reconstruct private training data through iterative optimization of pseudo-inputs [59]. Recent methodological advancements have significantly enhanced attack efficacy through innovations in prior-informed initialization [17]. ground-truth label recovery [57], and specialized regularization techniques [13 53), enabling successful breaches even against large-batch training scenarios and complex architectures like vision transformers [53] This evolving threat landscape underscores the critical need for robust defensive countermeasures.
2.3 Privacy Protection in Distributed Learning
Contemporary privacy-preserving techniques employ multi-layered protection strategies. Differ- ential privacy mechanisms inject calibrated noise into gradient updates 43 47 and similar noise perturbation strategies have also been explored to defend against backdoor attacks Secure aggregation protocols enable encrypted parameter aggregation without exposing individual updates Homomorphic encryption further extends protection by permitting computations on ciphertexts Although recent work by [11 proposes information-theorete framework for privacy leakage assessment in unimodal settings, the unique challenges posed by multimodal data interactions in MMFL remain largely unaddressed, highlighting a crucial gap in existing literature.
3
55
3 Preliminaries and Problem Statement
3.1 FL Procedures.
We aim to collaboratively train a global model for K total clients in FL. We consider each client can only access to his local private dataset Da := {a;, yi), where x; is the i-th input data sample and y: E (1, 2, .... ,C, C} is the corresponding label of x; with C classes. Specifically in MMFL, I; [2] , dataset is considered as the composition of all local datasets Σk= Dj. The objective of the FL learning system is to learn global model w that minimizes the total empirical loss over the entire dataset D:
rT"" where x!!" is the i-th input data sample of the m-th modality. The global
K DEl Cx(w), where [x(w)
Del Co
min L(w) =
(1)
D
D
3

---

Logils
Logts
Classifier
Look
Loss
1F
Malicious attacker
Fusion
Gradient:G
Gradienti
Fusion
Ground Trus
Encoder
Encoder
Encoder n
Encoder
Encoder
Encoder
cattle
cattle
In the image
image
D image
D2: text
D audio
D image
text
auddio
The information flow of GIA in MMFL. Information from the benign client's ' training data flows into the embedding after being processed by different modality encoders. Following fusion and classification, it is then incorporated into the gradient through the loss function. A malicious attacker can recover the information from the different modalities of the training data by eavesdropping on the gradient.
Figure 2:
where (u(w) is the local loss in the k-th client and Lce is the cross-entropy loss function that measures the difference between the prediction and the ground truth labels.
Each client updates its model parameters using gradient descent:
wft
1, (wr)).
(2)
where 1) is the learning rate, and V (w) represents the gradient of the loss function with respect to the parameters, denoted as G².
The central server aggregates the gradients from all clients to update the global model parameters:
K
Dal D
u++1 Wb GG, where G,
G
3.2 GIA Procedures.
GIA exploit the information encoded the gradients to reconstruct the original data D. The attacker's process can be summarized as follows:
1. Initialize a random guess D for the data.
2. Iteratively refine ò by minimizing the difference between the gradients computed from ò and the observed gradients G:
D D-D-nV (
3. The refined î converges to an estimate of the original data.
This process highlights how gradients can leak sensitive information, and the way attackers obtain information about training data from it. The whole process of data of each modality MMFL flowing to the attacker through GIA is shown in Figure
3.3 CntrmattinnThhoredde Preliminaries
The conditional entropy of X given Y represents the remaining uncertainty of X after observing Y:
H(XIY) = Σ
p(æ, y)
xEX
The mutual information between X and Y quantifies the amount of information shared between the two random variables:
I(X;Y) H(X) H(X(Y)
(6)

---

Leakage Risk Esitimator
Mulsimoaal Doitseets
Adjustr
H[G D
Pettubed
Gradient
R I(G D:
Nomaatzatoo
Uplood
Allootte
H(G, mi
[[G D2)
Gradiont
D
Aggregator
Mukual Infemmaiio
Server
LDP
Cross reduction
Client
Client
Distribute
The overall architecture of Sec-MMFL. After calculating the gradient of the multimodal model, the local training data processed by the Leakage Risk Estimator compute the conditional mutual information between the original data of each modality and the gradient. This is used to assess the risk of data leakage for each modality through the gradient. Based on this, the Noise Scale Adjustment module adjusts the privacy budget assigned to each modality, achieving a better balance between privacy protection and model effectiveness. Meanwhile, the Cross Modality MI Reduction module reduces the mutual information between gradients of different modalities, thereby mitigating cross-modal privacy leakage risks.
Figure 3:
3.4 Local Differential Privacy
perturbation algorithm M satisfies (c, 6)-Local Differential Privacy ((c, 8)-LDP) if, for any pair of adjacent datasets D and D', and for all possible output subsets S, the following inequality holds:
Pr(M(D) E S S e Pr(M(DD) E E +
where is the privacy budget of M, which quantifies the privacy protection level, and 6 is the probability of the privacy guarantee being violated. smaller value for indicates smaller gap between two probabilities and thus a stronger privacy.
4 Proposed Method
The workflow of the proposed framework shown in Algorithm T and Fig. Mcc-MMFL approach.
3
illustrates the
4.1 Leakage Risk Assessment
While assessing the information leakage of the gradient through GIA and applying corresponding protection before sending gradients to servers and evaluating the similarity between reconstructed data and real data offers direct insights, is computationally expensive due to its iterative nature, often requiring thousands of iterations to converge. Mutual information serves approach to quantify the information leakage in scenarios like MMFL; it can not only measure information dependence between the raw data and the gradient but also consider the correlation between the modalities.
more efficient
It is essential to formalize the description of the complete information leakage channel. In the whole process, we consider the dataset D as the sender and the estimated data D obtained through the attacker's iterative optimization as the receiver. The complete information leakage channel can be described as process from the data D to the gradients G, and subsequently from the gradients G to the estimated data D.
5

---

Algorithm 1: Sec-MMFL
Input:
number of communication rounds; K: number clients; y/ learning rate; (Dx) distributed datasets; w: initial model parameters; A: hyperparameter for MI reduction. Output:Traiie Trained global model parameters w Initialize the parameter w;
T: :
2 for / -= to 7 do
Server randomly selects client subset S, and sends them;
for each selected client k E S; do
compute the gradient G of model:
Leakage Risk Assessment: for each modality do
Assess the risk R; of leakage of data from i-th modality using conditional mutual information with 8
end
Noise Scale Adjustment: for each modality do
11
Adjust the scale of noise added for i-th modality with 9 10 11 and 12:
13
end
14
Cross Modality MI reduction:
15
Reduce the mutual information between gradient of encoders across modalities with 13
and [4]
16
Update local model parameters;
17
Send the updated model w back to the server;
18
end
19
At server sider;
Aggregate received models w from clients in S, to update the global model w;
20
end
However, the channel from gradients G back to the estimated data D is subject to the attacker's attack methods and prior knowledge, which cannot fully evaluate and control. Therefore, focus the controllable part the channel, which is from the data D to the gradients G.
When we evaluate the risk of data leakage from each modality, if we directly calculate mutual information between D and it reflects direct relationship between data and gradients without any additional conditions. But in MMFI., the influence of data from one modality on the gradient may be influenced or supplemented by data from other modalities; therefore, we choose conditional mutual information to evaluate the information sent by data De and received by gradients G in the channel which reflects the risk leakage R, of data from that modality:
R I(G:D D ((G D H(G D)
(8)
where (G D-1) is the entropy of the gradients given all modalities except D³, and (G D) is the entropy of the gradients given all modalities.
The conditional mutual information I (G; D' D=') serves as key metric for quantifying the privacy leakage of each modality. higher value indicates that more information about D' is encoded in the gradients G, even when the data from other modalities D- is known. This be particularly concerning if an adversary can access these gradients, as it indicates the potential to reconstruct or infer sensitive attributes of D
4.1.1 Noise Scale Adjustment
To balance privacy preservation and model utility in Sec-MMFL, we allocate noise scales a; for each modality using Rényi Differential Privacy (RDP), leveraging an information- -theoretcc approach to achieve tighter privacy accounting under composition. The leakage risk for modality is given by R. as defined in Eq. We compute normalized risk weight for each modality via softmax:

---

Wi
M E exp(- R2)
where M the total number of modalities. To ensure higher-risk modalities receive stronger protection, we define a scaling factor s, = 1/ /wy, which amplifies noise for modalities with larger R2 (and thus smaller wy). The noise scale for modality i is then set as:
0, Sjs with 8;
(10)
VW
The global scaling factor is determined via binary search to satisfy the target (Etarget 6D-DP guarantee. The total RDP cost at order a is:
M
Pootal ((x) = Σ pi(a); 1=1
where p:(a)) = compute_rdp((1. e 81, St, or)
(11)
Here, qs is the sampling rate, St is the gradient clipping norm, and a 1 is the Rényi order for modality i. This summation is valid for each fixed a > 1 due to the additive composition property of RDP under independent Gaussian mechanisms. We enforce the privacy budget by ensuring:
log(1/0)
min a>1
(a)
Starget: for fixed
(12)
a
This optimization ensures the cumulative RDP across all modalities satisfies the target privacy level. The resulting a; values are applied each modality's PrivacyEnging in Opacus, enabling efficient privacy-preserving training with minimal utility loss.
4.2 Cross Modality MI reduction
In MMFL, the close relationship between different modalities can lead to increased privacy leakage risks. For instance, if images and text pairs are closely related, and the text typically describes the image, then the ability to infer the text information could accelerate and increase the probability of the text being extracted by an attacker. Due the interconnection between different modalities, the mutual information between their encoders' gradients is also high. To address this issue, we propose a method that, in addition to calculating the task loss related to the accuracy (e.g., cross-entropy loss), includes an additional term to reduce the mutual information between the gradients of different modality encoders before propagating the gradients back for model updates.
Let Lusk be the task-specific loss, such as cross-entropy loss, and LMI be the loss term designed to reduce the mutual information between the gradients of different modality encoders. The total loss function Loaul is then given by:
Liotal Lrask + ALMI
(13)
where À a hyperparameter that balances the importance reducing mutual information against the task-specific loss.
The mutual information reduction loss Lmi can be defined based on the gradients of the different modality encoders G' G² G.,..., G as follows:
m-1
m Σ jmit1
L
Σ
GG
(14)
where GG): denotes the mutual information between the gradients of the i-th and j-th modality encoders.
By minimizing JMI, we aim to reduce the dependency between the gradients of different modality encoders, thereby reducing the risk of privacy leakage in MMFL.
7

---

Experiments
5.1
Datasets
We conduct MMFL experiments using both synthesized and native multimodal datasets. For image- text modality studies, we employ CIFAR-10/100 with text descriptions generated from image labels, following standard GIA experimental protocols. Additionally, validate our method on authentic multimodal benchmarks: Hateful-Memes for social media content analysis and CrisisMMD for disaster response.
5.2 Evaluation Metrics
Following 46 26 29 we evaluate: (1) Privacy protection via Peak Signal-to-Noise Ratio (PSNR) between original/reconstrreedd images on CIFAR datasets, and LPIPS for complex images in HaaisMMDD/attefll-MeMeeet
(2) Text Recovery Rate (TRR) measuring semantic similarity be- tween original and recovered texts; (3) Except for employing AUC as the evaluation metric on the Hateful Memes dataset and Fi-score the CrisisMMD dataset, classification accuracy utilized to measure model performance across all other datasets.
5.3 Attack Methods
We implement two gradient inversion attacks: (1) DLG (59 for CIFAR datasets with cross-modal label recovery, following the method [29] and IG [3 enhanced with stable diffusion generators for CrissMMMD/HatettlMMeene
where limited label semantics in these two datasets necessitate accelerated text-guided image reconstruction. Both methods optimize pseudo-inputs by minimizing the cosine distance between real and synthetic gradients through iterative backpropagation.
5.4 Baselines
Our method is compared against: DP-FedAvg 32 applying uniform noise across modalities; NbAFL with client-side parameter noising; LDP-FL [40] using random response mechanisms; and DP-FedAvg-MI adjusting noise scales via mutual information analysis.
48
The model used in the experiment has two encoders and classifier layer. On CIFAR-10 10 and CIFAR- 100, the text encoder is based on LeNet-5 and the image encoder based on TexICNN. On Crisis MMD and Hateful Memes, the text encoder is based Bert and the image encoder based on Resnet-50. For early fusion, the features generated by the encoders are fused by concatenation, and for late fusion, logits will be output after passing through the fully connected layer and fused by averaging at the decision level. We take lc-2 chosen via grid search, ö as le-5 and clipping norm 1.0. The batch size during training is set to 128. The learning rate is set to le-3 and the training is conducted for 200 rounds. We use neural estimators to calculate mutual information. We run experiments on Intel Xeon Gold 6133 CPU, RTX4090 GPU.
Table 1: Conditional mutual information parison.
Fusion Method
CIFAR-10
CIFAR-100
Text Early Fusion
0,2539
0,2439
Text Late Fusion
0.5383
0.3909
Image Early Fusion
0.0100
0.0120
Image Late Fusion
0.0180
0.0390
5.5 Conditional mutual information can help measure leakage risk differences
We compare the text and image conditional mutual information between early fusion and late fusion MMFL on different datasets. can be seen from Table that for the model used the two data sets, the conditional mutual information of text modality is larger than that of the image modality, this trend is consistent with our observation in the actual attack experiment that the speed of text recovery is much faster than that of image recovery, which proves that conditional mutual information can be successfully applied to measure the difference of information leakage risk among different modes. Moreover, the conditional mutual information of each modality of the late fusion is higher than that of the early fusion. This may be because each encoder the late fusion model more independent than that of the carly fusion model, so it has greater impact on the overall gradient of the model

---

Table 2: Defense performance on CIFAR-10 and CIFAR-100.
CIFAR-10
0:25
0.5
< 2
Method
Acct
PSNR]
TRR.
Acct
PSNR.
TRR.
Acct
PSNR.
TRR.
Accf
PSNR.
TRR.
DR-FedAvg
0,5219
6.563
0.57
0.6957
7.845
0.59
0.7336
8.166
0.61
0.8275
10.836
0.83
NbAFL
0,5654
7.039
0.61
0.7845
7.134
0.62
0.8031
8.998
0.72
0.8724
11.794
0.88
LDP-FL
4.341
0.41
6.931
0.9037
De.TedAvg-MI
0.50
7.073
0.7987
0.76
Sec-MMFL
0.7875
4.192
0.39
0.8159
4.824
0.42
0.8624
5.142
0.51
0.9201
6.912
0.69
CIFAR-100
5
10
Method
Acct
PSNR
TRR.
Acct
PSNR]
TRR
Acc
PSNRL
TRR.
Acc
PSNRL
TRR.
DP-FodAvg
0,3811
8,372
0.51
0,5233
12.592
0.74
0.6783
14.674
0.82
0.7538
17.515
0.91
NhAFL
0.4467
8.699
0.53
0.6185
11.482
0.71
0.7225
14.645
0.78
0.8321
18.014
0.89
LDP-FL
0,1089
6.255
0.45
0.4451
9.265
0.63
0.5154
11.197
0.66
0.5218
14.226
0.71
DP-FedAvg-MI
0,3704
7.628
0.49
0.4921
10.243
0.66
0.6561
13.588
0.79
0.7497
15.691
0.78
Sec-MMFL
0.4578
6.387
0.43
0.6759
9.241
0.59
0.8028
10.136
0.64
0.9214
12.891
0.67
and leaks more information, thus the barrel effect will be even stronger. Since it can be seen that the risk of privacy disclosure of all modalities of late fusion MMFL may be greater, we mainly use late fusion MMFL for privacy disclosure risk analysis in the following experiments.
5.6 Conditional mutual information can provide better protection
To measure the capability of Sec-MMFL to pro- vide more balanced and reasonable privacy pro- tection in MMFL, we compare the accuracy similarity of the recovered text and similarity of the recovered image compared to the origi- nal training data. From Table 2 it can be seen that existing methods exhibit fragile protection capabilities because they fail account for the differing information carried by different modal- ities and varying recovery rates. For instance, higher text recovery ability can leak more information to the image modality, leading to higher image similarity - a sign of insufficient privacy protection. The method of using simple mutual information between the data of each modality and the (Dadient(DP-eedAAvg--)
Hateful Memes
Crisis MMD
Method
AUCT
LPIPS1
TRR.
LPIPST
TRR.
De-FedAvg
0.766
0.536
0.573
0.77
0.384
0.572
0.58
LDPFL
0.376
0.33
0.51
0.263
0.583
0.51
DP DP-FedAvg:Mi
0.417
0.871
0.59
0.321
0.627
0.45
MMRL
0.564
0.975
0.47
0.385
0.801
0.39
Table 3: Comparison of defense performance.
can make the protection ability of each mode of MMFL more balanced. However, it does not take into account the relationship between the input of other modalities and the gradient, it sometimes cannot correctly reflect the real leakage risk of each modality, resulting in the protection effect is not balanced enough. Sec-MMFL using conditional mutual information can reasonably measure the risk of privacy breach, make the protection effect of the two modes more balanced, and achieve the minimum loss of accuracy simultancously. In fact, not only when using LDP in MMFL, this method of using conditional mutual information to leakage risk assessing and noise adjusting can also be applied on other defense methods in MMFL such directly adding Gaussian noise source data, which can also help to achieve a more balanced distribution of noise and more reasonable protection. Same trend is shown in Table and Figure
5.7 Task performance loss of Sec-MMFL
In order to explore the task performance cost for more balanced MMFL privacy protection under different settings, we conducted experiments with different client number, privacy budget amount, and different client data heterogeneity distribution. As shown in Figure and that as the number of clients goes up from 5 to 30, the performance loss has not changed much but when the privacy budget tightens from to 0.5, the performance loss increases slightly, we think it may be because smaller privacy budgets will bring more noise and let MMFL model be more sensitive to noise reallocation. As for the performance loss when the data distribution heterogeneity of each client changes, conducted experiments on the two data sets the setting of = 1 CIFAR- 10 and

---

DP-FedAvg
NbAFL
LDP-FL
DP-FedAvg-MI
Sec-MMFL
CIFAR- 100
cattle the image
cattle the image
cattle image
seculars image
seculars chosen image
eumeed seculars capture created image
CrisisMMD
RTT@usinggniillnnd:
rtlands usvirgin @-ii major flooding is st thomasllhurricane #virgin
usvirgin maxinunginis. majoraskvir stiac thomas hurricane flooding mar triumph
evaluated meteorgin+
sunglassesvir viragne nation major communespad chester. us burricanemaria
@+ usgin usferapliavir fh
Major Flooding
Hurria
printers major flood on
pulsing # usmaa ructured Hi
M Maria USS V #Virginlslands
mukherjec hurricane
Original Image/Text
Recovered Image/Text
Figure 4: Attack effect visualization.
Accurecy Compadisn (CIXS-100
SerteeFi (n
90
SocH (a=2)
SucIAVeL (ums)
(e-5)
50
20
04
But
Daortbutinn
(a) CIFAR-10 Scalability
(b) CIFAR-100 Scalability
(c) Data heterogeneity
Figure 5: Parameter Sensitivity Analysis.
<= 5 on CIFAR-100 for 10 clients. It can be seen from Figure 5c that when data heterogeneity is low (i.e., larger Dirichlet parameter a, such 0.9 or 0.7), Sec-MMFL has relatively small impact on performance; as heterogeneity increases (smaller o, down to 0)1), accuracy remains stable on both datasets, with slightly larger fluctuations on CIFAR-100 likely due its more complex task.
6 Conclusion and Future Work
We propose Sec-MMFL, framework that help enhance privacy preservation in MMFL. By leveraging information theory, SeC-MMFL can reasonably assess the privacy leakage risk of each modality in MMFL and properly adjust the privacy protection intensity of each modality, making MMFL more robust against GIA attacks. Extensive experiments conducted on various settings show that Sec-MMFI. strikes the best balance between defensive effect and utility. Although our method has shown significantly more balanced and robust protection MMFL of image and text modalities, there is currently work to perform better privacy protection studies on more modalities, and future work can carried out on more baseline models and other modalities.
Acknowledgment This work is supported by the National Key Research and Development Program of China under grant 20249FC3307900 the National Natural Science Foundation of China under grants 62302184, 62376103, 62436003 and 62206102; Major Science and Technology Project of Hubei Province under grant 2024BAA008 Hubei Science and Technology Talent Service Project under grant 2024DIC078; Ant Group through CCF-Ant Research Fund; and Fundamental Research Funds for the Central Universities under grant YCJJ2025239, The computation is completed the HPC Platform of Huazhong University of Science and Technology.
10

---

References
[1] Rodolfo Stoffel Antunes. Cristiano André da Costa, Ame Küderle, Imrana Abdullahi Yari, and Bjöm Eskofier. Federated learning for healtheare: Systematic review and architccture proposal. ACM Transactions on Intelligent Systems and Technology (TIST), 13(4):1-23, 2022.
[2] George Barnum, Sabera Talukder, and Yisong Yue. On the benefits of early fusion in multimodal representation learning. arXiv preprint arXiv:201 2020.
[3] Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. Practical secure aggregation for learning. In proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pages 1175-1191, 2017.
machine
2017.
[4] Said Yacine Boulahia, Abdenour Amamra, Mohamed Ridha Madi, and Said Daikh. Early, intermediate and late fusion strategies for robust deep learning-basee multimodal action recognition. Machine Vision and Applications, (2(6):121, 2021.
[5] David Byrd and Antigoni Polychroniadou. Differentially private secure multi-party computation for federated learning in financial applications. In Proceedings of the First ACM International Conference on Al in Finance, pages 1-9, 2020.
[6] Liwei Che, Jiaqi Wang, Yao Zhou, and Fenglong Ma. Multimodal federated learning: A survey, Sensors, 23(1)66998, 2023.
[7] Anda Cheng, Peisong Wang, Xi Sheryl Zhang, and Jian Cheng. Differentially private federated learning with local regularization and sparsification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10122-10131, 2022.
[8] Chenyuan Feng, Daquan Feng. Guanxin Huang, 7uozhu Liu, Zhenzhong Wang, and Xiang-Gen Xia. Robust
recommendation systems driven by multimodal federated learning. IEFF Transactions on Neural Networks and Learning Systems, 2024.
[9] Tiantian Feng, Digbalay Bose, Tuo Zhang, Rajat Hebbar, Anil Ramakrishna, Rahul Gupta, Mi Zhang, Salman Avestimehr, and Shrikanth Narayanan. Fedmultimodal: benchmark for multimodal federated learning. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 4035 4045, 2023.
[10] Konrad Gadzicki, Razieh Khamsehashari, and Christoph Zetzsche. Early vs late fusion in multimodal convolutional neural networks. In 2020 IEEE 23rd international conference information fusion (FUSION). pages 1- IEEE, 2020.
[11] Neha Gahlan and Divyashikha Sethia. Federated learning inspired privacy sensitive emotion recognition based multi- modal physiological sensors. Cluster Computing. 27(3)31179-33 2024.
[12] Jing Gao, Peng Li. Zhikui Chen, and Jianing Zhang. A survey deep learning for multimodal data fusion. Neural Computation, 32(5):89-866, 2020.
[13] Jonas Geiping, Hartmut Bauermeister, Hannah Dröge, and Michael Moeller. Inverting gradients-hor how easy is break privacy in federated learning? Advances in Neural Information Processing Systems, 33: 13:16937-16947, 2020.
[14] Ming Hu, Yue Cao, Anran Li, Zhiming Li, Chengwei Liu, Tianlin Li, Mingsong Chen, and Yang Liu. Fedmut: Generalized federated learning via stochastic mutation. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pages 12528-12537, 2024.
[15] Yangsibo Huang, Samyak Gupta, Zhao Song, Kai Li, and Sanjeev Arora. Evaluating gradient inversion attacks and defenses in federated learning. Advances in Neural Information Processing Systems, 34:7232- 7241, 2021.
[16] Tae-Ho Hwang, Jingyao Shi, and Kangyoon Lee. Enhancing privacy-preserving personal identification through federated learning with multimodal vital signs data. IEEE Access, 2023.
[17] Jinwoo Jeon, Kangwook Lee, Sewoong Oh, Jungseul Ok, et al. Gradient inversion with generative image prior. Advances in neural information processing systems. 2228998-29908 2021.
[18] Shiwei Li, Qunwei Li, Haozhao Wang, Ruixuan Li, Jianbin Lin, and Wenliang Zhong. Fedbif: Communication-efficae efficient federated leamning via bits freezing. IEEE Transactions on Parallel and Dis- tributed Systems, 36 36(12):2668 26788,22688228 2025.

---

[19] Shiwei Li. Xiandi Luo, Haozhao Wang, Xing Tang, Shijie Xu. Weihong Luo, Yuhua Li, Xiugiang He, and Ruixuan Li. The panaceas for improving low-rank decomposition in communication--fficienn federated learning. In Forty-second International Conference on Machine Learning, 2025.
[20] Shiwei Li, Wenchao Xu, Haozhao Wang, Xing Tang. Yining Qi, Shijie Xu, Weihong Luo, Yuhua Li, Xi- uqiang He, and Ruixuan Li. Fedbat: communication-effiee federated learning via learnable hinarization. In Proceedings of the 4 st International Conference on Machine Learning, pages 29074-29095, 2024.
[21] Yichen Li, Qunwei Li, Haozhao Wang, Ruixuan Li, Wenliang Zhong, and Guannan Zhang. Towards efficient replay in federated incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages
June 2024.
[22] Yichen Li, Yijing Shan, Yi Liu, Haozhao Wang, Wei Wang. Yi Wang. and Ruixuan Li. Personalized federated recommendation for cold-start users via adaptive knowledge fusion. In Proceedings the ACM on Web Conference 2025, WWW '25, page 2700-2709, New York, NY, USA, 2025, Association for Computing Machinery.
[23] Yichen Li, Haozhao Wang. Yining Qi, Wei Liu, and Ruixuan Li, Re-fed+: A better replay strategy for federated incremental learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
[24] Yichen Li. Haozhao Wang. Wenchao Xu, Tianzhe Xiao, Hong Liu, Minzhu Tu, Yuying Wang. Xin Yang. Rui Zhang, Shui Yu, Song Guo, and Ruixuan Li. Unleashing the power of continual learning on non-centralized devices: A survey, 2024.
[25] Yichen Li, Wenchao Xu, Yining Qi, Haozhao Wang. Ruixuan Li, and Song Guo. Sr-fdil: Synergistic replay for federated domain-incremental learning. IEFE Transactions Parallel and Distributed Systems, 35(11):1879- 2024.
[26] Zhaohua Li, Le Wang, Guangyao Chen, Muhammad Shafq, et al. A survey of image gradient inversion against foderated learning, Authorea Preprints, 2023.
[27] Ji Liu, Jizhou Huang, Yang Zhou, Xuhong Li, Shilei Ji, Haoyi Xiong, and Dejing Dou. From distributed machine learning to federated learning: A survey. Knowledge and Information Systems, 64(4):885-917, 2022.
[28] Ruixuan Liu. Yang Cao. Yanlin Wang, Lingjuan Lyu, Yun Chen, and Hong Chen. Privaterec: Differentially private model training and online serving for federated news recommendation. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 4539-4548, 2023.
[29] Xuan Liu, Sigi Cai, Lin Li, Rui Zhang, and Song Guo. Mgia: mutual gradient inversion attack in multi-modal federated learning (student abstract). In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages
2023.
[30] Guodong Long, Yue Tan, Jing Jiang, and Chengqi Zhang. Federated learning for open hanking. In Federated Learning: Privacy and Incentive, pages 240-254. Springer, 2020.
[31] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication--ffici learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273 1282. PMLR, 2017.
[32] H Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning differentially private recurrent language models. arXiv preprint 17Xiv:1710.06663, 2017.
[33] Arijit Nandi and Fatos Xhafa. A federated learning method for real-time emotion state classification from multi modal streaming. Methods, 204:340 347, 2022.
[34] Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, Andrew Y Ng, et al. Multimodal deep learning. ICML, volume 11, pages 689-696, 2011.
[35] Yuanzhe Peng, Yusen Wu, Jieming Bian, and Jie Xu. Hybrid federated learning for multimodal iot systems. IEEE Internet of Things Journal, 2024.
[36] Adnan Qayyum, Kashif Ahmad, Muhammad Ahtazaz Ahsan, Ala Al-Fuqaha, and Junaid Qadir. Collabora- tive federated learning for healthcare: Multi-modal covid-19 diagnosis at the edge. IEEE Open Journal of the Computer Society, 1844 2022.
[37] Pian Qi. Diletta Chiaro, and Francesco Piccialli. Fl-fd: Federated learning bearning-based fall detection with multimodal data fusion. Information fusion, 10110189,, 2023.
12

---

[38] Md Mahmudur Rahman and Sanjay Purushotham. Fedpseudo: Privacy- pseudo value-based deep learning models for federated survival analysis. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1999-2009, 2023.
[39] Dhanesh Ramachandram and Graham Taylor. Deep multimodal learning: survey on recent advances and trends. IEEE signal processing magazine, 34(6):96 108, 2017.
[40] Lichao Sun, Jianwei Qian, and Xun Chen. Ldp-fl: Practical private aggregation in federated learning with local differential privacy. arXiv preprint urXiv:2007.1578, 2020.
[41] Qi Tan, Qi Li, Yi Zhao, Zhuotao Liu, Xiaobing Guo, and Ke Xu. Defending against data reconstruction attacks in federated learning: An information theory approach. arXiv preprint arXiv:2403.0/268 2024.
[42] Jacob Thrasher, Alina Devkota, Prasiddha Siwakotai, Rohit Chivukula, Pranav Poudel, Chaunbo Hu, Binod Bhattarai, and Prashnna Gyawali. Multimodal federated learning healthcare: a review. arXiv preprint arXiv:2310.096500 2023.
[43] Stacey Truex, Ling Liu, Ka-Ho Chow, Mchmet Emre Gursoy, and Wenqi Wei. Ldp-fed: Federated learning with local differential privacy. In Proceedings of the Third ACM International Workshop Edge Systems, Analytics and Networking, pages 61-66, 2020.
[44] Haozhao Wang. Yichen Li, Wenchao Xu, Ruixuan Li, Yufeng Zhan, and Zhigang Zeng. Dafkd: Domain- aware federated knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20412-20421, 2023.
[45] Haozhao Wang, Haoran Xu, Yichen Li, Yuan Xu, Ruixuan Li, and Tianwei Zhang. FedCDA: Federated learning with cross-rounds divergence-aware aggregation. In The Twelfth International Conference on Learning Representations, 2024.
[46] Junxiao Wang, Song Guo, Xin Xie, and Heng Qi. Protect privacy from gradient leakage attack in federated learning. In IEEE INFOCOM 2022-IEEE Conference on Computer Communications, pages 580-589. IEEE, 2022.
[47] Kang Wei, Jun Li, Ming Ding, Chuan Ma. Howard H Yang. Farhad Farokhi, Shi Jin, Tony QS Quek, and H Vincent Poor. Federated learning with differential privacy: Algorithms and performance analysis. IEEE Transactions on Information Forensics and Security, 15:3454-3469, 2020.
[48] Kang Wei, Jun Li, Ming Ding, Chuan Ma. Howard H Yang, Farhad Farokhi, Shi Jin, Tony QS Quek, and H Vincent Poor. Federated learning with diffcrential privacy: Algorithms and performance analysis. IEEE transactions on information forensics and security, 15:3454-3469, 2020.
[49] Wengi Wei, Ling Liu, Margaret Loper, Ka-Ho Chow, Mehmet Emre Gursoy, Stacey Truex, and Yanzhao Wu. A framework for evaluating gradient leakage attacks in federated learning. arXiv preprint ar04 10397, 2020.
[50] Wensi Yang, Yuhang Zhang, Kejiang Yc, Li Li, and Sheng--hhng, Xu. Fíd: federated learning based method for credit card fraud detection. In Big Data Data-Big BigData 2019 8th International Congress, Held as Part of the Services Conference Federation, SCF 2019, San Diego CA, USA, June 25-30, 2019, Proceedings 8, pages 18-32. Springer, 2019.
[51] Xiyuan Yang, Wenke Huang, and Mang Ye. Dynamic personalized federated learning with adaptive differential privacy. In Thirty-seventhh Conference Neural Information Processing Systems, 2023.
[52] Yanxin Yang, Chentao Jia, DengKe Yan, Ming Hu, Tianlin Li, Xiaofei Xie, Xian Wei, and Mingsong Chen. Sampdetox: Black-box backdoor defense via perturbation- sample detoxification. Advances in Neural Information Processing Systems, 37:1 2212221311222222 2024.
[53] Hongxu Yin, Arun Mallya, Arash Vahdat, Jose M Alvarez, Jan Kautz, and Pavlo Molchanov, See through gradients: Image batch recovery via gradinversion. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 16337-16346, 2021.
[54] Qiying Yu, Yang Liu, Yimu Wang, Ke Xu, and Jingjing Liu. Multimodal federated learning via contrastive representation ensemble. arXiv preprint a Xiv: 08888, 2023.
[55] Chengliang Zhang, Suyi Li, Junzhe Xia, Wei Wang. Feng Yan, and Yang Liu. Batcherypt: Efficientt homomorphic encryption for cross-silo federated learning. In 2020 USENIX annual technical conference (USENIX ATC 20), pages 493-506, 2020.
13

---

[56] Rui Zhang, Song Guo, and Ping Li. Gradfilt: Class-wise targeted data reconstruction from gradients in federated learning. In Companion Proceedings of the ACM on Web Conference 2024, pages 698-701, 2024.
[57] Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. idlg: Improved deep leakage from gradients. arXiv preprint arXiv:2001.022 022600000..0........000100 2020.
[58] Yuchen Zhao, Payam Barnaghi, and Hamed Haddadi. Multimodal federated learning on iot data. In 2022 IEEE/ACM Seventh International Conference on Internet-o Things Design and Implementation (IoTDI), pages 43 54. IEEE, 2022.
[59] Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. Advances neural information processing systems, 32, 2019.
14

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Section and Section 4 eexpicitly state our contributions in developing multi- modal privacy protection through coormation-theoretie gradient regulation. Abstract and introduction accurately reflect the theoretical framework and empirical validation scope.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Section and Section 6 discusses generalization limitations across dataset types, addressing performance degradation under high data heterogeneity scenarios, and applicability on more modalities.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: Our work focuses on empirical validation of privacy protection mechanisms rather than theoretical analysis. All propositions are experimentally validated through gradient inversion attacks.
4. Experimental result reproducibility
Question: Does paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section specifies dataset generation protocols, attack implementations (DLG/IG variants), and evaluation metrics.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: CrisisMMD dataset access is restricted by proprictary agreements. Synthetic CIFAR-text datasets can be regenerated using our described methodology.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section 5 details data splits, optimizer configurations, and hyperparameter search spaces. Training procedures are specified in Algorithm
7. Experiment statistical significance
Does the paper report enror bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Results report mean values over 5 runs. Variance analysis excluded due to consistent cross-run performance patterns observed across different initialization seeds.
15

---

8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: Section Section Specifies that we run all experiments Intel Xeon Gold 6133 CPU, RTX4090 GPU.
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics ttthics//ttss:/neuiiipsss httppp///eaarppp.cccpaaaaaaaaaaa
Answer: [Yes]
Justification: All research complies with NeurlPS Ethics Guidelines. No human subjects or sensitive personal data involved. Benchmark datasets used contain publicly available non-identifiable information.
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Section [6 discusses potential damage of gradient inversion attacks and mitiga- tions against adversarial exploitation through Sec-MMFL techniques.
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our method focuses on defense mechanisms rather than releasing potentially harmful models. No pretrained generative models or high-risk assets distributed.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: CIFAR datasets (CC-BY 4.0) and Hateful Memes (CC- (CC-BY-NC 2.0) licenses acknowledged in References. CrisisMMD usage complies with original data sharing agree- ments.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: No new datasets or models released beyond methodological implementations. Synthetic text descriptions for CIFAR generated through label-to-text transformations.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Study exclusively uses benchmark datasets without human annotation crowdsoureing components.
Institutional review board (IRB) approvals or equivalent for research with human subjects
16

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No human subject research conducted. All data sources consist of publicly available benchmark corpora.
Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [No]
Justification: LLMs not used in methodology development or experimental analysis. Gram- marly employed solely for grammar checking in manuscript preparation.
17


Reviews:
[{"review_id": "CVIbGKLAH9", "reviewer": "Reviewer_X7LP", "summary": "This paper proposes an information-theoretic approach to analyze privacy leakage in the process of MMFL. The proposed Sec-MMFL uses conditional mutual information to quantify the information leakage risk of each modality and dynamically adjusts the corresponding protection strength. Experiments on benchmark datasets demonstrate that Sec-MMFL outperforms traditional methods.", "questions": "The paper highlights that \"different modalities require varying protection strengths\" as a key motivation, but what is the specific solution and validation of this problem? Can conditional mutual information guarantee such a balance across modalities? More qualitative analyses on this issue would likely facilitate an intuitive understanding of the proposed model.\n\nThe current comparative methods do not adequately cover multimodal baselines. For instance, methods like LDP-Fed included in the comparisons are not designed for multimodal scenarios, making it difficult to effectively validate the advantages of the proposed method in MMFL settings. Although this issue is acknowledged in the limitations, mere textual mention cannot compensate for the lack of experimental evidence. This weakens the support for the claim that the proposed method concentrates on multimodal problems. It is recommended to supplement comparative experiments with mainstream multimodal methods to clarify the performance boundaries and advantages of the proposed approach.\n\nThe model framework (Figure 3) is hard to follow. It is claimed that \"the Noise Scale Adjustment module adjusts the privacy budget assigned to each modality, achieving a better balance between privacy protection and model effectiveness,\" yet it does not seem to illustrate how this balance is achieved.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "This paper focuses on the issue of MMFL and cross-modal leakage, which is an interesting topic. The proposed privacy-preserving approach is tailored to the characteristics of MMFL, with clear research objectives. However, the current comparative methods do not adequately cover multimodal baselines. The claimed solutions are not clearly validated or demonstrated qualitatively.", "was_revised": true, "final_justification": "My questions have been fairly well addressed, and I choose to raise the score.", "rebuttal": "**Question1: The paper highlights that \"different modalities require varying protection strengths\" as a key motivation, but what is the specific solution and validation of this problem? Can conditional mutual information guarantee such a balance across modalities? More qualitative analyses on this issue would likely facilitate an intuitive understanding of the proposed model.**\n\n**Response1:** Thank you for highlighting this crucial aspect. Our solution Sec-MMFL directly addresses modality-specific protection through two core innovations: **1)** Using conditional mutual information $I(G;D^i \\mid D^{-i})$ (Eq. 8) to quantify the unique leakage risk per modality, accounting for inter-modal correlations. This allows precise risk assessment where text modality consistently shows higher risk than images, validating its sensitivity to modality characteristics. **2)** Adaptive noise allocation via Eq. 10 that inversely scales privacy budget $\\epsilon_i$ with normalized risk $R'_i$ ensures high-risk modalities (e.g., text) receive stronger protection (lower $\\epsilon_i$) while preserving utility for low-risk modalities. Experimental validation in Table 2 ($\\epsilon=0.5$: TRR↓ 0.67 for text vs PSNR↓ 12.62 for images) demonstrates balanced protection. Qualitative analysis in Figure 5 visually confirms reduced cross-modal leakage, showing attackers cannot leverage recovered text to reconstruct high-fidelity images under Sec-MMFL, whereas uniform protection (LDP-Fed) allows clear image reconstruction when text is compromised. Additional parameter sensitivity analysis (Figure 4) further verifies stability across client counts and heterogeneity levels.\n\n**Question2/Weakness1: The current comparative methods do not adequately cover multimodal baselines. For instance, methods like LDP-Fed included in the comparisons are not designed for multimodal scenarios, making it difficult to effectively validate the advantages of the proposed method in MMFL settings.**\n\n**Response 2**: We appreciate this insightful suggestion. To strengthen validation, we have supplemented experiments with FedMultimodal [9], a state-of-the-art MMFL baseline, on CrisisMMD dataset, and Hateful Memes dataset. The table below presents a comprehensive comparison of all methods in terms of task performance and defense capability across two datasets. While FedMultimodal (specifically designed for MMFL) achieved marginally higher task accuracy on both datasets, it lacks dedicated defense mechanisms. In contrast, our Sec-MMFL method maintains comparable task performance while demonstrating significantly more balanced privacy protection across all modalities. We will add these results to Table 3 and explicitly discuss FedMultimodal comparisons in Section 5.3 of the revised manuscript. Furthermore, we will continue to incorporate additional mainstream multimodal methodologies to rigorously clarify the performance boundaries and comparative advantages of the proposed approach.\n\n| **Defense Method** | **CrisisMMD** | **CrisisMMD** | **CrisisMMD** | **Hateful Memes** | **Hateful Memes** | **Hateful Memes** |\n| ------------------ | ------------- | ------------- | ------------- | ----------------- | ----------------- | ----------------- |\n|                    | **Acc↑**      | **LPIPS↑**    | **TRR↓**      | **Acc↑**          | **LPIPS↑**        | **TRR↓**          |\n| LDP-Fed            | 0.652         | 0.583         | 0.80          | 0.572             | 0.663             | 0.75              |\n| NbAFL              | 0.635         | 0.602         | 0.82          | 0.564             | 0.670             | 0.73              |\n| LDP-FL             | 0.477         | 0.799         | 0.72          | 0.433             | 0.832             | 0.61              |\n| LDP-Fed-MI         | 0.643         | 0.756         | 0.64          | 0.569             | 0.891             | 0.57              |\n| FedMultimodal      | **0.658**     | 0.511         | 0.83          | **0.605**         | 0.622             | 0.78              |\n| Sec-MMFL           | 0.642         | **1.085**     | **0.53**      | 0.575             | **1.116**         | **0.49**          |\n\n\n\n**Question3: The model framework (Figure 3) is hard to follow. It is claimed that \"the Noise Scale Adjustment module adjusts the privacy budget assigned to each modality, achieving a better balance between privacy protection and model effectiveness,\" yet it does not seem to illustrate how this balance is achieved.**\n\n**Response 3**: Thank you for this valuable feedback. We will revise Figure 3 to explicitly visualize the balancing mechanism: 1) Add directional arrows showing the flow from Leakage Risk Estimator (outputting $R_i)$ to Noise Scale Adjustment, where Eq. 9-10 convert $R_i$ into $\\epsilon_i$ values (e.g., text: \\($\\epsilon_{\\text{text}}$=0.2$\\epsilon$), image: $(\\epsilon_{\\text{image}}=0.8\\epsilon)$ when $(R_{\\text{text}} > R_{\\text{image}}$)). 2) Include gradient icons with variable noise magnitudes (larger noise symbols for text, smaller for images).  This clarifies how high-risk modalities induce lower $\\epsilon_i$ (stronger noise), while low-risk modalities retain higher $\\epsilon_i$ (weaker noise) - precisely balancing protection and utility. The empirical evidence of this balance is demonstrated in Table 2 (e.g., at $\\epsilon=1$, text TRR improves by 26% and image PSNR degrades only 6% vs LDP-Fed)."}, {"review_id": "O9Zud8gV4e", "reviewer": "Reviewer_AQiP", "summary": "Recognizing that different modalities exhibit varying vulnerabilities to adversarial attacks and that inter-modal data correlations can expand the scope of privacy leakage across modalities, this paper proposes an information-theoretic framework for optimal privacy budget allocation across modalities while mitigating privacy risks arising from cross-modal dependencies.", "questions": "1.\tHow is the entropy of gradients computed in Equation (8)? Given that encoders across different modalities have distinct architectures, the resulting gradients will have different dimensions. How are these heterogeneous gradients aggregated or represented when computing entropy across all modalities?\n2.\tHow does the allocation scheme in Equation (10) guarantee that the total privacy budget equals $\\epsilon$?\n3.\tSimilar to Question 1, given the difference in gradient dimensions across modalities, how is the mutual information between different modalities calculated in Equation (12)? What preprocessing or alignment steps are applied to enable meaningful mutual information computation?\n4.\tWhat is the rationale for selecting the hyperparameter $\\lambda$? The paper appears to lack discussion on $\\lambda$'s sensitivity analysis or selection criteria in the experimental section.\n5.\tWhy were the most recent attack methods, such as those referenced in [40], not included in the experimental evaluation? The current attack methods appear to be from publications approximately five years ago, which may not adequately represent the evolving threat landscape in federated learning privacy attacks.", "limitations": "The limitations are not explicitly discussed, but the authors mentioned some of them in the checklist.", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths: \n1. The topic of privacy protection in multimodal federated learning is timely, and the integration of information theory is interesting.\n2. The experimental evaluation provides comprehensive comparisons that demonstrate the effectiveness of the proposed method.\n3. The paper is well-written with clear and thorough introductions to the background and related work, making it accessible to readers.\n\nWeakness:\n1. The paper lacks sufficient mathematical rigor in key derivations. For instance, there is no clear explanation of how the allocation scheme in Equation (10) guarantees that the total privacy budget equals $\\epsilon$, which is a fundamental requirement for differential privacy.\n2. Several technical details are inadequately explained, leading to ambiguity in the methodology. Specifically, given that different modalities employ distinct encoder architectures, it remains unclear how the gradient mutual information in Equation (12) is computed across heterogeneous network structures.\n3. The evaluation relies on somewhat outdated attack methods, which may not reflect the current threat landscape. The paper would benefit from more comprehensive experiments incorporating state-of-the-art privacy attacks to strengthen the validity of the proposed defense.\n4. From Table 2, it seems that the improvement over the existing approaches is rather marginal. The improvement in privacy is obtained with a degradation in test accuracy.\n5. There are some typos. For example, in lines 253-254, the text encoder should be TextCNN, and the image encoder LeNet-5.", "was_revised": true, "final_justification": "From the manuscript and response, I feel the authors submitted the paper in a rush with lots of ambiguity. Failing to specify the privacy mechanisms (and how they are implemented) is not acceptable for a differential privacy paper in my opinion. In addition, mixing up the approximate DP and pure DP, as well as utilizing the basic composition theorem instead of the more advanced ones suggests a lack of familiarity with the differential privacy literature. I do not think this can be overlooked. Therefore, I tend to keep my original score.", "rebuttal": "**Weakness1: The paper lacks sufficient mathematical rigor in key derivations. For instance, there is no clear explanation of how the allocation scheme in Equation (10) guarantees that the total privacy budget equals $\\epsilon$.**\n\n**Response1.1:** We sincerely appreciate you for pointing out the mathematical formulation problem in Equation (10). This resulted from a typographical error in the manuscript where the normalized risk weight $R_i'$ was incorrectly placed in the denominator. The allocation scheme and all experiments actually implemented the formulation:$\\epsilon_i = \\epsilon \\times R_i'$. This scheme is mathematically sound because the normalization property of $R_i'$ (defined in Equation (9) as: $R_i' = \\frac{e^{-R_i}}{\\sum_{t=1}^M e^{-R_t}}$inherently satisfies: $\\sum R_i' = 1$ which guarantees: $\\sum \\epsilon_i = \\epsilon$ through the relation: $\\sum_{i=1}^M (\\epsilon \\times R_i') = \\epsilon \\times \\underbrace{\\sum_{i=1}^M R_i'}_{=1} = \\epsilon$. We will correct Equation (10) in the manuscript and explicitly prove budget conservation in Section 4.2. \t\n\n**Weakness2:  How the gradient mutual information in Equation (12) is computed across heterogeneous network structures.**\n\n**Response1.2:** Thanks for raising the concern. We address cross-modal gradient mutual information computation across heterogeneous encoders using Mutual Information Neural Estimation (MINE) method with Donsker-Varadhan representation. The gradients from different modalities are flattened into 1-dimensional vectors, concatenated, and then fed into a three-layer MLP discriminator network (layer sizes [32, 64, 32] with ReLU activation). The model was trained for 2000 epochs using a learning rate of 1e-3. Raw text encoder gradients $G^{text} \\in \\mathbb{R}^{N \\times d*_{text}}$ and image encoder gradients $G^{image} \\in \\mathbb{R}^{N \\times d_*{image}}$ serve as direct inputs. Positive sample pairs ${(G_i^{text}, G_i^{image}) for i=1 to N}$ are constructed from gradients originating from identical data instances to represent the joint distribution $P_{G^{text},G^{image}}$, while negative sample pairs${(G_i^{text}, G_{\\pi(i)}^{image}) for i=1 to N}$ are generated via random permutation $\\pi$ of image gradients to simulate the marginal product distribution $P_{G^{text}} \\otimes P_{G^{image}}$. The mutual information is computed as: $I(G^{\\text{text}};G^{\\text{image}})  \\geq \\sup_\\phi \\big( E_{P_(joint)}[T] - \\log E_{P_(marginal)}[e^T] \\big)$. So this approach handles architectural heterogeneity because the critic network $T_\\phi$ learns relationships in the unified space independently of source architectures. \n\n**Weakness3: The paper would benefit from more comprehensive experiments incorporating state-of-the-art privacy attacks to strengthen the validity of the proposed defense.**\n\n**Response1.3:** Thank you very much for the advice. Regarding the attack methods, we followed the same experimental setup as Mgia in the MMFL security-related paper [20]. Indeed, more attack methods could be incorporated. In the following experimental setup, we compared the results of LDP-Fed and Sec-MMFL under the \"See through gradients\" attack [40], and we will supplement additional experimental results with more configurations and attack methods in future work.\n\nCIFAR-10, $\\epsilon$ =1:\n\n| **Method** | PSNR ↓ | TRR↓ | Acc↑   |\n| ---------- | ------ | ---- | ------ |\n| LDP-Fed    | 19.232 | 0.91 | 0.7927 |\n| Sec-MMFL   | 15.379 | 0.61 | 0.7801 |\n\nCIFAR-100, $\\epsilon$ =20:\n\n| **Method** | PSNR↓  | TRR↓ | Acc↑   |\n| ---------- | ------ | ---- | ------ |\n| LDP-Fed    | 18.716 | 0.82 | 0.3591 |\n| Sec-MMFL   | 13.978 | 0.64 | 0.3270 |\n\nCrisisMMD:\n\n| **Method** | LPIPS↑ | TRR↓ | Acc↑  |\n| ---------- | ------ | ---- | ----- |\n| LDP-Fed    | 0.472  | 0.80 | 0.652 |\n| Sec-MMFL   | 0.835  | 0.53 | 0.642 |\n\n Hateful Memes: \n\n| **Defense** | LPIPS↑ | TRR↓ | Acc↑  |\n| ----------- | ------ | ---- | ----- |\n| LDP-Fed     | 0.588  | 0.75 | 0.572 |\n| Sec-MMFL    | 0.961  | 0.49 | 0.575 |\n\n**Weakness4: From Table 2, it seems that the improvement over the existing approaches is rather marginal. The improvement in privacy is obtained with a degradation in test accuracy.**\n\n**Response1.4**: Thank you so much for raising this concern. We want to clarify that Sec-MMFL's primary contribution is achieving optimized privacy-utility equilibrium in multimodal federated learning, particularly addressing the inherent tension between protecting high-risk modalities (e.g., text) and preserving low-risk modality utility (e.g., images). While absolute accuracy gains may seem limited, our method demonstrates significant qualitative advances: Under equivalent privacy budgets (ε=1), Sec-MMFL reduces Text Recovery Rate (TRR) by 33% (0.61 vs. LDP-Fed's 0.91) while limiting accuracy degradation to just 1.2% (0.7801 vs. 0.7927) on CIFAR-10. This balanced protection is visually evidenced in Figure 1, where uniform defenses fail catastrophically—text is perfectly reconstructed while images remain irrecoverably blurred—whereas our conditional mutual information framework (Eq. 8) dynamically calibrates noise to prevent such asymmetric vulnerabilities. The critical innovation lies in transcending the \"high TRR or low accuracy\" dichotomy: At ε=2, Sec-MMFL maintains near-identical accuracy (90.01% vs. 90.05%) while slashing TRR by 26% (0.69 vs. 0.93), demonstrating unprecedented multimodal harmony. This balance is further validated in CrisisMMD results (Table 3), where our approach simultaneously improves LPIPS by 86% (1.085 vs. 0.583) and TRR by 34% (0.53 vs. 0.80) without compromising classification accuracy.\n\n**Weakness5: There are some typos. For example, in lines 253-254, the text encoder should be TextCNN, and the image encoder LeNet-5.**\n\n**Response1.5**: We sincerely apologize for this oversight. We will correct these typos in the paper and conduct another thorough check.\n\n**Question1: How is the entropy of gradients computed in Equation (8)? Given that encoders across different modalities have distinct architectures, the resulting gradients will have different dimensions. How are these heterogeneous gradients aggregated or represented when computing entropy across all modalities?**\n\n**Response2.1:** We sincerely appreciate the question. The conditional mutual information $I(G;D^{i}|D^{-i})$ in Equation (8) is actually computed using the Neural Estimators for Conditional Mutual Information framework (Molavipour et al., 2021). To address heterogeneous gradient dimensions across modalities, our implementation:   1. Flattens each modality's gradient tensor into a 1D vector   2. Concatenates all modality gradients into a unified representation $\\tilde{G} = \\text{concat}(\\text{vec}(G^{\\text{text}}), \\text{vec}(G^{\\text{img}}), \\cdots)$   3. Processes three distinct inputs  through the MLP classifier $f_\\theta$:     - $\\tilde{G}$ (concatenated gradients)     - $D^i$ (target modality data)     - $D^{-i}$ (complementary modalities data) . The classifier distinguishes: 1.True pairs$(\\tilde{G}, D^i)$ with correct $D^{-i}$ 2.Counterfactual pairs: $(\\tilde{G}, \\tilde{D}^i)$ where $\\tilde{D}^i$ is shuffled within k-nearest neighbors of $D^{-i}$ . The final CMI value is derived as:   $\\hat{I} = \\frac{1}{B}\\sum_{b=1}^B \\log \\frac{f_\\theta(G_b, D_b^i|D_b^{-i})}{f_\\theta(G_b, \\tilde{D}_b^i|D_b^{-i})}$ . This architecture resolves dimensional heterogeneity through gradient flattening and concatenation, and conditional mutual information is estimated using neural networks rather than directly computing the entropy.\n\n**Question2: How does the allocation scheme in Equation (10) guarantee that the total privacy budget equals $\\epsilon$ ?**\n\n**Response2.2:** Thanks for the question. We have already explained in Response1.1 the typographical error of Equation (10) and provided the corresponding detailed proof that guarantees the total privacy budget equals $\\epsilon$ .\n\n**Question3: Similar to Question 1, given the difference in gradient dimensions across modalities, how is the mutual information between different modalities calculated in Equation (12)? What preprocessing or alignment steps are applied to enable meaningful mutual information computation?**\n\n**Response2.3:** Thank you for your question. We have already elaborated on similar computational procedures in Response 1.2. The gradients from different modalities only need to be flattened and fed into the neural network for computation via the Mutual Information Neural Estimation (MINE) method.\n\n**Question4: What is the rationale for selecting the hyperparameter λ? The paper appears to lack discussion on λ's sensitivity analysis or selection criteria in the experimental section.**\n\n**Response2.4:** Thank you for your question. Regarding the selection of hyperparameter λ, we acknowledge that the article lacks detailed discussion. In practice, we employed a grid search approach, testing multiple values (0.01, 0.05, 0.1, 0.3, 0.5, 1.0) and ultimately found that λ = 0.1 achieves an optimal balance between privacy preservation and task performance. Other values, while providing comparable privacy protection, incurred significantly greater accuracy degradation.\n\n**Question5: Why were the most recent attack methods, such as those referenced in [40], not included in the experimental evaluation?** \n\n**Response2.5:** Thank you for your question. We have addressed this question and supplemented relevant experiments in Response 1.3, and we plan to conduct more comprehensive experiments in future work."}, {"review_id": "MhSLYa53e5", "reviewer": "Reviewer_3U4G", "summary": "Sec-MMFL introduces an information theory based framework that enhances privacy in multimodal federated learning by dynamically quantifying leakage risks of each modality via conditional mutual information, adaptively scaling noise budgets, and reducing dependencies across different modalities, with validation across diverse datasets showing improved privacy-utility tradeoffs.", "questions": "1. Could you provide more details on the specific implementation of the conditional mutual information calculation? For example, what specific algorithms or approximations are used to estimate the conditional mutual information between data of each modality and the gradients?\n2. How might Sec-MMFL be adapted for other modality combinations, like audio-video modalities? Is it possible to outline a potential approach using existing datasets like CREMA-D?\n3. Would it be possible to include brief benchmarking against recent MMFL defenses like FedMultimodal?\n4. For GIA attacks, how are pseudo inputs initialized? When applying LDP noise, did you clip gradients before or after adding noise?", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. Establishes novel modality-specific protection, validated through evidence of text/image vulnerability asymmetry.\n\n2. Conditional mutual information provides a rigorous theoretical foundation for risk quantification correlated with attack outcomes.\n\n3. Multiple datasets testing under varied privacy budgets demonstrates consistent improvements with clear workflow visualization.\n\n\n\nWeaknesses:\n\n1. Lack of details on the implementation of the conditional mutual information calculation,  such as the specific algorithms or approximations used to estimate the conditional mutual information.\n\n2. While the framework shows promising results for image-text systems, its applicability to other modality combinations like audio-video remains unexplored. \n\n3. The paper does not provide detailed comparisons with other state-of-the-art privacy protection methods specifically designed for multimodal federated learning, making it difficult to fully assess its relative advantages.\n\nMinor suggestions:\n\nline 125:\" We consider each client k can only access to his local private dataset\" -> \"We consider that each client k can only access their local private dataset\".\n\nline 138: \"GIA exploit the information encoded in the gradients\"  -> \"GIA exploits the information encoded in the gradients\".\n\nThe complexity of the sentence \"The whole process of data of each modality in MMFL flowing to the attacker through GIA is shown in Figure 2.\" in line 145 could present readability challenges. Suggest simplifying it as follows: \"Figure 2 illustrates the flow of data from each MMFL modality to attackers via GIA.\"", "was_revised": true, "final_justification": "Thanks for the response, which fully addresses my concerns. I also note that some reviewers claim that the description of the method is unclear. I consider this is because the authors ignore the some preliminaries on federated learning with differential privacy. This issue may be acceptable to researchers who are familiar with this field. Therefore, I have decided to raise my rating to 5.", "rebuttal": "**Weakness1: Lack of details on the implementation of the conditional mutual information calculation**  \n\n**Response1.1:** We sincerely thank the reviewer for this insightful question regarding our CMI implementation. The conditional mutual information $I(G;D^{i}|D^{-i})$ in Equation (8) is actually computed using the Neural Estimators for Conditional Mutual Information framework (Molavipour et al., 2021). To address heterogeneous gradient dimensions across modalities, our implementation:   1. Flattens each modality's gradient tensor into a 1D vector   2. Concatenates all modality gradients into a unified representation $\\tilde{G} = \\text{concat}(\\text{vec}(G^{\\text{text}}), \\text{vec}(G^{\\text{img}}), \\cdots)$   3. Processes three distinct inputs  through the MLP classifier $f_\\theta$:     - $\\tilde{G}$ (concatenated gradients)     - $D^i$ (target modality data)     - $D^{-i}$ (complementary modalities data) . The classifier distinguishes: 1.True pairs$(\\tilde{G}, D^i)$ with correct $D^{-i}$ 2.Counterfactual pairs: $(\\tilde{G}, \\tilde{D}^i)$ where $\\tilde{D}^i$ is shuffled within k-nearest neighbors of $D^{-i}$ . The final CMI value is derived as:   $\\hat{I} = \\frac{1}{B}\\sum_{b=1}^B \\log \\frac{f_\\theta(G_b, D_b^i|D_b^{-i})}{f_\\theta(G_b, \\tilde{D}_b^i|D_b^{-i})}$ .\n\n**Weakness2: Applicability to audio-video modalities**  \n\n**Response1.2:** We deeply appreciate this important question about modality generalization. Sec-MMFL extends to audio-video systems through its architecture-agnostic framework. For audio inputs, we process log-Mel spectrograms using Resnet-18 encoders, while video inputs utilize 3D-ResNet encoders for temporal feature extraction. The conditional mutual information estimation follows identical neural estimation procedures as in image-text systems, with protection strengths automatically calibrated based on modality-specific leakage risks. Our validation on CREMA-D demonstrates effecness: leakage risk assessment automatically detects audio's higher vulnerability (38% greater $R_i$ than video). \n\n**Weakness3: Lack of SOTA comparisons for MMFL**  \n\n**Response1.3:** Thanks for the question. To strengthen validation, we have supplemented experiments with FedMultimodal [9], a state-of-the-art MMFL baseline, on CrisisMMD dataset, and Hateful Memes dataset. The table below presents a comprehensive comparison of all methods in terms of task performance and defense capability across two datasets. While FedMultimodal (specifically designed for MMFL) achieved marginally higher task accuracy on both datasets, it lacks dedicated defense mechanisms. In contrast, our Sec-MMFL method maintains comparable task performance while demonstrating significantly more balanced privacy protection across all modalities. We will add these results to Table 3 and explicitly discuss FedMultimodal comparisons in Section 5.3 of the revised manuscript.\n\n| **Defense Method** | **CrisisMMD** | **CrisisMMD** | **CrisisMMD** | **Hateful Memes** | **Hateful Memes** | **Hateful Memes** |\n| ------------------ | ------------- | ------------- | ------------- | ----------------- | ----------------- | ----------------- |\n|                    | **Acc↑**      | **LPIPS↑**    | **TRR↓**      | **Acc↑**          | **LPIPS↑**        | **TRR↓**          |\n| LDP-Fed            | 0.652         | 0.583         | 0.80          | 0.572             | 0.663             | 0.75              |\n| NbAFL              | 0.635         | 0.602         | 0.82          | 0.564             | 0.670             | 0.73              |\n| LDP-FL             | 0.477         | 0.799         | 0.72          | 0.433             | 0.832             | 0.61              |\n| LDP-Fed-MI         | 0.643         | 0.756         | 0.64          | 0.569             | 0.891             | 0.57              |\n| FedMultimodal      | **0.658**     | 0.511         | 0.83          | **0.605**         | 0.622             | 0.78              |\n| Sec-MMFL           | 0.642         | **1.085**     | **0.53**      | 0.575             | **1.116**         | **0.49**          |\n\n**Suggestions: Language revisions**  \n\n**Response1.4:** Thank you so much for the suggestions. All suggested language improvements have been implemented: client data access description (line 125) updated to \"their local private dataset\", subject-verb agreement in GIA description (line 138) corrected, and complex sentence (line 145) simplified to \"Figure 2 illustrates the flow of data from each MMFL modality to attackers via GIA\" for enhanced readability.\n\n**Question1:Could you provide more details on the specific implementation of the conditional mutual information calculation?**\n\n**Response2.1:**  Thanks for the question. We have already explained in Response1.1 the  details on the specific implementation of the conditional mutual information calculation using the neural estimators for conditional mutual information method.\n\n**Question2: How might Sec-MMFL be adapted for other modality combinations, like audio-video modalities? Is it possible to outline a potential approach using existing datasets like CREMA-D?**  \n\n**Response2.2:** Thanks for the question. Our framework naturally extends to CREMA-D through modality-specific feature extractors: Resnet-18 for log-Mel spectrograms (audio) and 3D-ResNet for optical flow (video). Details are explained in Response1.2.\n\n**Question3: Would it be possible to include brief benchmarking against recent MMFL defenses like FedMultimodal?**  \n\n**Response2.3:** Thank you for this  inquiry. We have supplemented experiments in Response1.3.\n\n**Question4: For GIA attacks, how are pseudo inputs initialized? When applying LDP noise, did you clip gradients before or after adding noise?**  \n\n**Response2.4:** We appreciate the question. For gradient inversion attacks, pseudo-inputs are initialized with $\\mathcal{N}(0,1)$ for images and random vocabulary tokens for text. Regarding LDP implementation, gradients are clipped (norm=1.0) before adding Gaussian noise . Optimization uses Adam (3000 iterations, lr=0.1) with total variation regularization ($\\lambda=0.$2) to balance attack realism with practical constraints."}, {"review_id": "6QzkEgDc0g", "reviewer": "Reviewer_EEh7", "summary": "This work mitigates modality-specific vulnerabilities in MMFL through risk assessment based on conditional mutual information, adaptive noise allocation, and cross-modal decoupling, with multi-dataset experiments demonstrating significant privacy gains.", "questions": "1. The paper mentions the use of early and late fusion strategies in the experiments. Could you explain how the proposed method's effectiveness might vary depending on the fusion strategy used?\n2. The paper mainly uses LeNet-5 and TextCNN as encoders. How would the choice of more complex or different model architectures affect the performance of Sec-MMFL in terms of both privacy protection and model utility?\n3. Could Sec-MMFL's adaptive noise profiles be reverse-engineered to identify high-risk modalities? If so, what countermeasures would prevent attackers from exploiting this meta-information?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Novel risk-aware protection framework grounded in information theory.  \n- Effective cross-modal leakage suppression methodology.  \n- Empirically validated across heterogeneous datasets and privacy budgets.\n\nWeaknesses:\n- The privacy protection effectiveness is mainly measured by PSNR and TRR, which might not fully capture the privacy implications in complex real-world scenarios. More comprehensive metrics could strengthen the evaluation.\n- There is limited exploration of the impact of different model architectures on the performance of Sec-MMFL. The framework's effectiveness might vary depending on the choice of encoders and fusion mechanisms.\n-  Omits discussion about potential risks: adaptive noise profiles could themselves leak modality sensitivity, creating new attack surfaces.\n- Inconsistent terminology appears in:\n  - Line 268: \"information leakage risk among different modes\" (should be \"among different modalities\")\n  - Line 286: \"which will make the text mode protection too strong\" (should be \"which will make the text modality protection too strong\")\n  - Line 288: \"make the protection effect of the two modes more balanced\" (should be \"make the protection effect of the two modalities more balanced\")", "was_revised": true, "final_justification": "This work proposes a novel risk-aware protection framework to mitigate modality-specific vulnerabilities in MMFL. The authors’ response has addressed my concerns regarding the fusion strategy and model architecture. I find this paper valuable and believe it is worthy of acceptance.", "rebuttal": "**Weakness1: The privacy protection effectiveness is mainly measured by PSNR and TRR, which might not fully capture the privacy implications in complex real-world scenarios. More comprehensive metrics could strengthen the evaluation.**\n\n**Response1.1:** We appreciate this critical observation regarding evaluation metrics. To provide more comprehensive privacy assessment, we will supplement our evaluation additional metrics like SSIM (Structural Similarity Index) and CLIP-space.\n\n**Weakness2: There is limited exploration of the impact of different model architectures on the performance of Sec-MMFL. The framework's effectiveness might vary depending on the choice of encoders and fusion mechanisms.**\n\n**Response1.2:** Thank you for highlighting this important dimension. We will expand Section 5 with comprehensive architecture sensitivity analysis including: 1) Transformer-based encoders (BERT/ViT) versus CNN/RNN architectures; 2) Different fusion mechanisms (concatenation vs attention-based); and 3) Model complexity scaling from 1M to 100M parameters. \n\n**Weakness3/Question3: Omits discussion about potential risks - adaptive noise profiles could themselves leak modality sensitivity, creating new attack surfaces.**\n\n**Response1.3:** We appreciate this important security consideration. The fundamental protection mechanism ensures that reverse-engineering noise profiles is both practically infeasible and mathematically ineffective. Attackers face critical barriers:   1. Attackers only observe the final perturbed gradients $G' = G + \\eta$ (where $\\eta \\sim \\text{Gaussian}(0, \\Delta f/\\epsilon_i)$), lacking access to the original gradients $G$ or the specific noise parameters $\\epsilon_*i$. This obscures the relationship between noise magnitude and modality sensitivity.   2.  Even if attackers could hypothetically estimate relative noise levels, identifying which modality received stronger protection (indicating higher risk) provides no exploitable pathway. The local differential privacy transformation satisfies $(\\epsilon,\\delta)$-LDP guarantees (Eq.7), making the noise-added gradients statistically irreversible.\n\n**Weakness4: Terminology Revisions.**\n\n**Response1.4:**  We thank the reviewer for catching these inconsistencies - all instances will be corrected: Line 268: \"information leakage risk among different modalities\"; Line 286: \"text modality protection\"; Line 288: \"two modalities more balanced\". Comprehensive terminology standardization will be implemented throughout Sections 3-5 to ensure consistent use of \"modality\" when referring to data types.\n\n**Question1: The paper mentions the use of early and late fusion strategies in the experiments. Could you explain how the proposed method's effectiveness might vary depending on the fusion strategy used?**\n\n**Response1.1:** We appreciate this fundamental question. Sec-MMFL's effectiveness varies significantly with fusion strategies due to inherent information flow differences: Late fusion (feature-level integration) exhibits higher initial leakage risks (Table 1: R_text=0.5383 vs 0.2539) but enables precise modality-specific protection through our noise allocation, yielding maximum 12% privacy gains. In contrast, early fusion creates entangled representations where modality-specific protection is less effective but baseline risks are lower. Our method achieves optimal balance in late fusion scenarios where modality-specific vulnerabilities are most pronounced and conditional mutual information can be precisely quantified.\n\n**Question2: The paper mainly uses LeNet-5 and TextCNN as encoders. How would the choice of more complex or different model architectures affect the performance of Sec-MMFL in terms of both privacy protection and model utility?**\n\n**Response1.2:** We appreciate the inquiry. Complex architectures demonstrate two countervailing effects: 1) Increased baseline vulnerability due to richer gradient information (higher $R_i$ values); but 2) Enhanced noise absorption capacity enabling lower noise injection for equivalent protection. Sec-MMFL's  architecture-agnostic design ensures relative improvements over uniform protection actually increase with model complexity."}]

Author rebuttal:

Meta-review: This paper proposes a method for securing multimodal federated learning via differential privacy. Reviewers praised the methodological development and empirical evaluations in the paper, with the major technical concern raised by two reviewers of a lack of comparison with dedicated multimodal baselines being successfully addressed. 

After the rebuttal and discussion, the remaining negative reviewer bases their evaluation largely on an unclear explanation of the differential privacy details and other clarity issues. These concerns, which I have copied below for completeness, were either not shared or viewed as not so significant by the other reviewers. In my estimation, I agree that the clarity issues are not so severe as cannot be fixed in a revision, and so I am tentatively recommending acceptance. However, I encourage the authors to carefully consider and revise according to the clarity suggestions given by the reviewers.

## Reviewer AQiP's discussion comments:

My major concern about this paper is that the authors seem not familiar with the differential privacy literature. For example, they utilized $\sigma_i\propto\sqrt{T\ln(1/\delta)}/\epsilon$, which is a result of moments accountant for aggregating privacy over $T$ communication rounds (and somehow the privacy amplification effect of subsampling is ignored, see the seminar work Abadi, et al., Deep learning with differential privacy). The authors referred to Wei Kang, et al, but the reference considered a different context. However, when composing privacy over modality, they use the basic composition theorem. Limiting the results to only 2 modalities may also hinder the applicability of the proposed method.

The writing in manuscript and response lacks some rigor as well, for example, in their response, they claimed a clipping threshold of $|\mathbf{g}_i|<C$, but $\mathbf{g}_i$ is a vector, and the Gaussian mechanism assumes $|\|\mathbf{g}_i|\|_2<C$ (note that different mechanisms may require different types of norms for clipping. The Laplace mechanism requires $L_1$-norm).

Overall, I feel this paper was prepared in a rush, and a major revamp will significantly improve the clarity of the paper. Without another round of careful review, it is not certain if the paper will be clear enough for a publication in my opinion.
Decision: Accept (poster)