Federated Dialogue-Semantic Diffusion for Emotion Recognition under Incomplete Modalities
Authors: Xihang Qiu
Jiarong Cheng
Yuhao Fang
Wanpeng Zhang
Yao Lu
Ye Zhang
Chun Li
Venue: NeurIPS 2025 poster ()
Keywords: federated learning; multimodal emotion recognition; diffusion model
Primary area: deep_learning

Abstract: Multimodal Emotion Recognition in Conversations (MERC) enhances emotional understanding through the fusion of multimodal signals. However, unpredictable modality absence in real-world scenarios significantly degrades the performance of existing methods. Conventional missing-modality recovery approaches, which depend on training with complete multimodal data, often suffer from semantic distortion under extreme data distributions, such as fixed-modality absence. To address this, we propose the Federated Dialogue-guided and Semantic-Consistent Diffusion (FedDISC) framework, pioneering the integration of federated learning into missing-modality recovery. By federated aggregation of modality-specific diffusion models trained on clients and broadcasting them to clients missing corresponding modalities, FedDISC overcomes single-client reliance on modality completeness. Additionally, the DISC-Diffusion module ensures consistency in context, speaker identity, and semantics between recovered and available modalities, using a Dialogue Graph Network to capture conversational dependencies and a Semantic Conditioning Network to enforce semantic alignment. We further introduce a novel Alternating Frozen Aggregation strategy, which cyclically freezes recovery and classifier modules to facilitate collaborative optimization. Extensive experiments on the IEMOCAP, CMUMOSI, and CMUMOSEI datasets demonstrate that FedDISC achieves superior emotion classification performance across diverse missing modality patterns, outperforming existing approaches.

Federated Dialogue-Semantic Diffusion for Emotion Recognition under Incomplete Modalities
Xihang Qiu Jiarong Chengl.? Yuhao Fang', Wanpeng Zhang',
Yao Lu', Ye Zhangl? Chun Li"
Shenzhen MSU-BIT University Beijing Institude of Technology
@euxhbbit.edu.. cn, =f_cya2778ottoook com, 1ichun20200smbu, edu. cn
Abstract
Multimodal Emotion Recognition in Conversations (MERC) enhances emotional understanding through the fusion of multimodal signals. However, unpredictable modality absence in real-world scenarios significantly degrades the performance of existing methods. Conventional missing-modality recovery approaches, which depend on training with complete multimodal data, often suffer from semantic distortion under extreme data distributions, such as fixed-moallity absence. To address this, we propose the Federated Dialogue stalogue-guided and Cemantic-Consistte Diffusion (FedDISC) framework, pioneering the integration federated learning into missing- missing-modality recovery. By federated aggregation of modality-speciffc diffusion models trained on clients and broadcasting them to clients missing cor- responding modalities, FedDISC overcomes single-client reliance modality completeness. Additionally, the DISC-Diffusion module ensures consistency context, speaker identity, and semantics between recovered and available modal- ities, using a Dialogue Graph Network to capture conversational dependencies and Semantic Conditioning Network enforce semantic alignment. We further introduce a novel Alternating Frozen Aggregation strategy, which cyclically freezes recovery and classifier modules to facilitate collaborative optimization. Extensive experiments on the IEMOCAP, CMUMOSI, and CMUMOSEI datasets demon- strate that FedDISC achieves superior emotion classification performance across diverse missing modality patterns, outperforming existing approaches.
Introduction
Multimodal Emotion Recognition in Conversations (MERC) has emerged as a pivotal tech- nology of affective computing for understanding complex emotion states through synergistic fusion of textual, acoustic, and visual signals remarkable performance under ideal multimodal conditions 7. their efficacy collapses catastroph- ically real-world scenarios plagued by unpredictable missing modalities caused by sensor failures, environmental noise, privacy constraints [8] 10].
While State-of-the-art (SOTA) methods achieve
Missing modality challenges MERC can be categorized into two scenarios random missing protocol and fixed missing protocol. To recovery the missing information, researchers have proposed various recovery methods, primarily falling into two paradigms: Latent space semantic recovery: Lian et al. [12] propose GCNet to construct cross-modal correlations in the latent space through graph neural networks, leveraging temporal and speaker information from available modalities to
"Equal contribution. 'Corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

compensate for missing features. 2) Explicit modality recovery: with score-based diffusion model directly reconstruct missing modality features, with available modalities guiding the reverse diffusion process as conditional inputs.
Wang et al. g leverage IMDer
However, these methods struggle severely in fixed missing protocol scenarios, where modality is entirely absent in local datasets. The lack of distributional priors for the missing modality and cross-modal alignment supervision leads to critical failures [13 [4]. Latent space recovery suffers
from feature confusion due to missing modality-speciffe representations [TS] [6], while generative
modality recovery models fail catastrophically due to the absence of raw data required for the training of generative model. This highlights the inherent contradiction between current modality recovery techniques' reliance on modality completeness and their real-world applicability under extreme incomplete modalities.
This paper proposes the Federated Dialogue-Guided and Coimantic-Consisee Diffusion framework (FedDISCE which innovatively integrates federated learning with generative modality recovery to address challenges posed by incomplete modalities. To alleviate the dependency of generative modality recovery on modality completeness, FedDISC establishes a federated architecture where clients train local modality-specine models are then aggregated into global modality-specific diffusion models on the server and broadcast to clients missing corresponding modalities. This eliminates the need for clients to locally train recovery models for missing modalities, overcoming the limitations of single-client incomplete modalities while enabling zero-shot cross-client modality recovery and safeguarding data privacy.
diffusion models using their local available modalities. These
Additionally, we design the DISC-Diffusion model, which integrates a Dialogue Graph Network (DGN) capture contextual dependencies and speaker relationships through graph structures, and Semantic Conditioning Network (SCN) that extracts semantic information from available modalities via attention mechanisms. The fusion of these two modules ensures tri-dimensional consistency between recovered and available modalities across context, speaker identity, and semantic alignment. Finally, we introduce the Alternating Frozen Strategy (AFS), which cyclically freezes recovery module and classifier module on each client to resolve optimization confiicts between generative and classification objectives during federated collaborative training. The contributions of this work are summarized as follows:
1. Federated Learning for Modality Recovery: FedDISC pioneers the integration of federated learning with generative modality recovery, mitigating single-client limitations caused by incom- plete modalities while preserving data privacy.
2. DISC-Diflusion for Consistent Recovery:
We design DISC-Diffusion with DGN and SCN modules to ensure tri-dimensional consistency (context, speaker identity, semantics) between recovered and available modalities, leveraging dialogue and semantic constraints.
3. Alternating Frozen Strategy: AFS effectively eliminates cross-modal optimization conflicts by alternately freezing recovery and classifier modules during federated updates.
2 Background
2.1 Incomplete Multimodal Learning
Incomplete multi-modal learning, a critical research topic in machine learning, aims to enhance models' capability to handle unpredictable and inevitable modality missing in real-world scenarios 17. can be categorized into two paradigms based on whether to restore missing modalities: non-recovery methods and recovery methods [18]
Non-recovery methods focus on inferring directly from incomplete modal inputs by improving model architectures or optimization strategies. Representative approaches include knowledge distillation- based and correlation maximization techniques. Knowledge distillation methods employ teacher networks to learn modality-speciit predictive models, then distill knowledge from available modali- ties to student networks [19] 20]. Correlation maximization methods aim to maximize cross-modal dependencies and enforce shared low -in-dimensional embeddings across heterogeneous modalities
through covariance constraints or mutual information maximization 2 22
*Code Repository: https: htt///gttttuuuu wtwphhwww.oo///ppp//e/ee..ee.ee

---

Recovery methods aim to estimate and reconstruct missing modalities from available modalities. It can be categorized into latent space semantic recovery and explicit modality recovery. The former recovers missing semantics in latent spaces by mining deep dependencies from multimodal data. Lian et al. 12 uses graph networks to capture temporal and speaker dependencies in dialogues, enabling semantic compensation for incomplete modalities within an invisible latent space. Zhang et al. [23] focuses on processing
signals via graph neural networks, achieving more comprehensive modality recovery through multi-frequency feature alignment compared with [2]. Explicit modality recovery usually employs generative models to directly reconstruct missing modalities. For example: Wang et al. [18] utilizes normalizing flow-based generative model DiCMoR to predict missing modality distributions, exploiting the reversible property and precise density estimation of flow models to ensure distributional consistency. Liu et al. [9] proposes CIF-MMIN, which integrates an invariant natture-giied imagination module and cascaded residual autoencoders to generate missing modality features that maintain semantic coherence with available modalities. However, the inherent dependency of existing methods on data completeness fundamentally restricts their practical effectiveness in real-world scenarios with severe modality missing.
2.2 Incomplete Multimodal Federated Learning
Multimodal federated lear)ing(MFL) enables decentralized cross-modal learning while preserving data privacy. However, the inevitable presence of missing modalities in real-world scenarios poses significant challenges to its effectiveness. Le et al. 24] propose Multimodal Federated Cross Prototype Learning (MFCPL) to addresses the issue of missing modalities through prototype learning. They introduce Cross-Modal Alignment (CMA) to align zero-padded features of missing modalities with existing modality features, reducing noise from zero-padding. However, this method can only deeply mine existing data the prototype level, failing to fundamentally reconstruct missing information. To tackle this, Yin et. al. (25) introduce Stable Diffusion into MFL to recover missing modalities. On image-text datasets with missing image modalities, clients upload text embeddings, and the server generates and extracts image modality features. However, this method deploys a global generative model on the server, unable to generate client-specific local feature image modalities, limiting its generalization. To address these issues, our FedDISC framework fine-tunes modality- specific feature recovery modules for each client's local characteristics.
3 Method
3.1 Dialogue-guided and Semantic-Consistent Module
This chapter focuses generative modality recovery methods under fixed missing protocol scenarios. To ensure consistency between the recovered modality and the available modalities across context, speaker identity, and semantic alignment, FedDISC first pretrain two modules. DGN and SCN, as illustrated in Figure 2 We define a conversation has C utterances: C = {u,){_ and each utterance d; is spoken by speaker Pa(u.). where s(-) is a map between the utterance and its speaker. In fixed missing protocol, each client retains data from at least one modality, resulting in total of (2 1) possible missing patterns, where M is the number of all modalities. Here, we consider
three modalities: language ()), vision (v), and acoustic (a), and we will discuss the case of missing / modality.
3.1.1 Dialogue Graph Network
The core idea of our DGN is to capture context and speaker dependencies from conversational utterances, which have been proven to be essential for dialogue understanding [26] 27 Due to the inconsistent dimensional spaces across different modalities, we first employ a feature extractor composed of pre-trained embedding layers and convolution layers to project the raw data into unified dimensional space H = [h," \e 1- We construct separate speaker graph and context graph for each available modality extract modality-awan dialogue information.
Taking modality a as an example: H. = { utilized as the initial nodes. In graphs, edge weights quantify the importance of connections between nodes, while edge types reflect their interaction patterns. The context graph and speaker graph share the same graph structure, but employ distinct edge types to capture different dependencies. To avoid excessive computational
3

---

DGN
h
Speaker Graph
17
Speaker Graph
Self- A Attention
R-GCN
Z
Conet
Classifier
Context Graph
V
A
hg
Aggregate
h
4
R-GCN
V
Speaker A Speaker B A B to
SCN
Cross- A ttention
Salf' Attertion
A Audio
Embeding
A to B
Cross-
Vision Head
Forward
A ttention
Backward
Speakers
Present
The frame work of DGN and SCN. DGN captures context and speaker dependencies through graph network while SCN captures cross-modal semantic information with attention mechanism.
Figure 1:
overhead from connecting all nodes, we constrain the graph scale by implementing a fixed-size window w. We select w from the set {1, 2, 3), and a node h; can only connect within the node set denote the edge from hę h2 eij E &, We
{N)]je[mez(i-- w.11 (i.1)mmn mmmmmiin..))] Then
w = 2 in Figure !
Speaker Graph: This graph leverages speaker relation to capture speaker-aware dependencies in conversations. Let O4j E A denote the speaker relation identifier of eij. the value of flow from Pa(u.) P=(u))) Therefore, |.A] when the conversation happens between two speakers: {i (,j + i, i).
ozj reflects the
Context Graph: This graph leverages context relation to capture context-aware dependencies in conversations. Let Big E B denote the context relation identifier of eij. the value of Bij can be categorized as (forward, present, backward] based on the relative positions of node h2 and hy the conversation.
Graph learning We employ single-layer R-GCN 28 to aggregate speaker and context information under multiple relations. The calculation formula is shown as follows:
Σ JENJENT
Σ
W hy). of = o < NJ
ITT W W,
(I)
TEB JENT
where of and of are the output Speaker Graph and Context Graph, respectively. N is the set of connected indexes of h; under relation r, W, is the weighted parameters for different types of graph under relation r. ar(-) )) is the activation function, we leverage ReLU function in this paper. After the aggregation, add corresponding nodes together to get the dialogue graph representation for each available modality: 2mhh = u."h + e m E {a, v}. To predict the category of each utterance, concatenate modality-specific graph representations from available modalities, which are then processed through multi-head self-attention layers and a multi-layer perceptron (MLP) to yield classification probabilities: ÿa CCe, is the number of emotion classes. Then can get the loss of DGN with cross-entropy loss for pretraining: LDGN = - Vilog((a). where ys denotes the ground truth of utterances.
3.1.2 Semantic Conditioning Network
propose the SCN to capture cross-modal semantic information from available modalities for semantic alignment. Similar to DGN, we utilize pre-trained embedding layers and projection layers transform raw data from different modalities into a dimensionally aligned feature space: H {h) c 1 gCxd is the shared dimension.

---

Pacoder Module
Recovery Modale
Milssing
Nouse
Cruss-
LosS
Mali-qaasinn Eackk uk
Noise
Chert
Cliver
Vision
specthic Uncooder
Clssstic Modile
Sal-
Fribeding
Uieday aa
Altetoo
DGN +
prowlddyy
good dey. ..
MLP
Text
Cliert
Sevie
Clisst
Text
Embedding
68
DGN + SCN
Audia
Cileckn
Figure 2: The training pipeline of FedDISC, this figure delineates a hierarchical federated learning framework with two alternating phases: the Recovery Module Training Stage (right) and the Classifier Optimization Stage (left).
To capture inter-modal dependencies, we employ cross-attention layers between audio and visual features. For modality pair (h", the cross-aftention mechanism calculates:
204 Norm(h + Cross h²)),, 20 Norm(h" + Cross ,)))),
(2)
where CrossAllni K = V) = -Sf ( OK V). and 2a 1=0 E IRxxxxx, Each fused
feature undergoes self-attention to refine intra-modas representations zzelf: F Then we select the head token of each sequence as the condality-specinfe semantic summary: s" 2 10], All the heads are concatenated and fed into classifier to predict the emotion category Ve- Consistent with DGN, loss of SCN is calculated as: LSCN =
Σi Vklog(Us). Finally, we combine these two loss functions into a joint objective function which is utilized to pretrain the Dialogue-guides and Ceemantic-Consisterr Module: L LDGN LSON.
3.2 Federated Dialogue-Guided and Semantic-Consisterr Diffusion
We consider centralized federated learning architecture similar to FedAvg 29 where a sever coordinates n clients with incomplete local dataset DD = {>/", ys | E Mavvsi where Nr the number of samples in the 7th client, Moooil the set of available modalities, with S Mavail M.
3.2.1 DISC-Diffusion Model
Due to ethical concerns and processing convenience, our DISC-Diffusion model generates latent features for missing modalities rather than raw data.
We leverage the SGN and SCN, both pretrained on local data, to respectively extract dialogue dependency and semantic alignment information from the available modalities. Let 2 = concat(:" ) m E Mavvol denote the concatenation of the graph network outputs corresponding each available modality in the DGN, and و denote the concatenation of the heads corresponding to each available modality in the SCN, where 2D, = E RR- The embedding layer concatenates the two representations and projects the result into the conditional embedding space. The formula is:
Conditional embedding fusion:
(Linear (),
(= ( Nixxp.
Training process: Recent work has demonstrated the effectiveness of conditional diffusion models 30 Inspired by its success, we implement the conditional noise prediction network with U-Net architecture [3I] where cross- aoos-atttn layers are used integrate the intermediate feature of the U-Net with the conditioning signal c:
F(++) = Concat(F Cross Linear(c)),
(4)
where (() the input of the 1h layer of U-Net, Linear(-) projects e the same size of F(), To amplify the condition, we randomly omit the condition with fixed probability p during training to enable unconditional predictions, and during inference we merge the conditional and unconditional outputs with a weighted combination:
& (1+w) t, w wee (2(2))))
(5)
5

---

further, to optimize the network, we can obtain the noise matching objective function:
e~ ((1 + w) co(21, co(2,, t))
(6)
Sampling process:
During the reverse denoising process, we guide the noise prediction with the conditional embedding and define the reverse transition distribution as:
pe (2 (e,cc) = (z--i (ee t,,,,,,),, Be f),
where (i, t, is calculated as:
(r 1
1 az ai
eg( (zr, t,)),
(8)
Var
vI
where a, denote the noise scheduling coefficient. Then sampling is performed using the reparame- terization trick.
3.2.2 Alternating Frozen Strategy
As illustrated in Figure 2 harmonize the optimization of recovery and classification modules, propose the AFS, a two-stage, hierarchical federated learning protocol that systematically freezes and activate model components. Each client's local model is partitioned into three modules: 1) Encoder Module: a pretrained feature extractor, whose parameters remain fixed throughout all training; 2) Recovery Module: employs DISC-Diffusion for modality recovery; 3) Classifier Module: a downstream predictor that is composed of attention layers and a MLP.
Stage I: In the first stage, as the right part of Figure [hhows, only the Recovery Module is active, while the Classifier Module is frozen. This stage has three steps: 1) Local Update: Client trains its local modality-speciff DISC-Diffusion model 8."H using locally available modality m, where
cross-modal data D CT E Mfvasi serves as the conditioning input according to equation 6 2) Server Aggregation: After local epochs, each client uploads its moddlitty-speciiii DISC-Diffusion model to the server. The server then performs modality-specifff aggregation following equation ?? yielding three global diffusion models {0mm) m M one per modality. 3) Broadeast: These aggregated global models are sent back to all clients for the next stage.
Stage II: In the second stage, the left part of Figure 2 shows, invert the freezing scheme: the Recovery Module is frozen, and the Classifier Module becomes active. We first apply the global diffusion model (0m²), }. m miss to recover the feature of the missing modality m after equation thereby obtain a complete-modality representation. The recovered full-modal features pass through cross-attention layer, a self-attention layer, and MLP produce the output ý E R-xxx The loss function for classifier optimization is L Ni Σ Vilog(ū:). Then we leverage equation ?? to aggregate and update the classifier modules of all clients.
By alternating between these two stages-freezing the classifier during recovery aggregation and freezing recovery during classifier refinement-every communication rounds, we prevent gradient interference and ensure that each module converges under a coherent training signal. Empirically, this strategy leads stable optimization and superior generalization for both modality recovery and emotion recognition.
4 Experiments
4.1 Datasets and Implementation Details
Datasets: To verify the effectiveness of FedDISC, we conduct experiments on three benchmark conversational datasets: IEMOCAP 32). CMU-MOSI [33, and CMU-MOSEI (34. IEMOCAP consists of 151 conversations between two speakers. For a fair comparison, we employ two prevalent labeling methods, generating datasets with four classes (5 and six classes [12] CMU-MOSI consists of 2199 utterances, and CMU-MOSEL contains 22856 utterances.
Evaluation metrics: IEMOCAP is labeled in categorical labels, therefore we use 4-class accuracy (ACC4), 6-class aceuracy (ACC6), and weighted average F1-score (WAFI) 35] as our evaluation
metric. Lables of CMU-MOSI and CMU-MOSEI are scored between [-3, 3). This paper focuses

---

Dataset
Available
Basoline
FedDISC
FedDI5C
GCNe [2]
CU--MMIN
28
[8]
[1)
49,6999
68.0688.2
65.4665.2
66.7465.9
58.059.3
66.2866
604604
28.6/28.5
[e]
58, I/57
62.5/60.3
00.1559
55.755.7
53.351.3
56,355..6 6
56.2/54.7
21.9/15.6
IEMOCAIN
(i.v)
67.2/67.3
53.0/50.4
75.875.88
61.2/60.6
73.2/73.0
56.2/56.
56.8/6.3
64.9/65.0
56.3/58.4
73,2/11.0
57,8/57.3
18.8/68.0
67.3/6..3
50.8550.7
37.025.6
340022
[[a]
5165.8
78.0/88.1
1/33333
68, 7468
74.3/75.6
73.0/22.1
65,9666.2
32,5/32.3
[1,0]
60.66603
68. 7/88.0
67. 4/67
601,460888
65. 65.7/66.9
60.2600
65.064488
8/33/
[,,,a]
78.67784
78.6/78.4
78.6788.
78.4/78.3
78.4/78.1
78.5778.1
78.1778.3
78.2778.
{f)
16.445.8
53.6153.6
50.8/50.0
54.7/51.3
34,5/33.8
{e)
2/36.4
56.0/55.5
46. 3446.3
55.7:55.7
39.7/35.2
41,841.0
39 39.3/35.1
33.6/32.4
IEMOCAP6
[(,0]
[n]
49 4:99.9
36 8/79
60.6/59.9
17,4/17.6
$4.8/55. 0
52.2/51
56.8/96.1
49.3/47.8
56.3/58.4
51.392211
$1.6/50.7
60.6/003
39. 39.2/37.5
49.6/49.0
34.4/348
18.6/38.7
[.aa]
51.3:506
59,459.3
58, 18..0
51.9/51.3
53.8/50.1
60.3/00.4
4/50/
17.1
[9.0]
41.1/36.3
66.4/66.3
52.8/52.9
44,04434
56.5/56.9
43.8/41.2
36.5735.9
(...))
61.3161.7
64.3/61.7
64.3661.7
58.6/59.1
61.3/60.2
61.36112
58.7/58.4
55,9/56.2
Table 1: Comparison results IEMOCAP4 and IEMOCAP6 datasets across different available modalities. The best result in each row is highlighted in dark green.
on the negative/positive classification task, scores less than are mapped negative and greater than O are mapped to positive. For both datasets, we choose the accuracy (ACC) and the WAFI as
evaluation metric 33 37
Implementation details: In line with previous works 11 8 We tested different models on two commonly used protocols: 1) random missing protocol, 2) fixed missing protocol. For random missing protocal, we define the missing rate as 17
Σ m, where m, denotes the number NixM
of available modalities for the jth sample, N₂ indicates the sample number of client 1. At the same time, we ensure that at least one modality is available for each sample, s.t. m; 2 1 and v) S M-1 In this paper, we set M to 3, so we have
[0.0, 0.1. 0.7]. For fixed missing protocol, Wee" define that each client completely lacks n modalities. To ensure at least one modality is retained, we set 1 I < 3. Consequently, the possible modality absence patterns for each client can be enumerated as: (U)) {u}. {a), v1, a), (v,a)) a}). For federated learning, we set the number of clients n. 3, each client evenly and randomly allocated all training, validation, and test data. During the training process, we set the local epoch e to 1, and the communication round E to 3, the window size = 2. We perform five-fold cross volidaaaio [2] and report the mean values on the test set. All experiments were conducted on two NVIDIA LOSS GPUs, each equipped with 48 GB of memory.
Dataset
Method
Missing
0.0
0.3
0.4
FedDISC (P)
86,3186.3
85.9/85.8
83,3833.2
81.1/81.0
79.7/79.6
80.2200.2
77,5/775
75.7/75.8
81,2/81
FedDISC
85.9/85.9
84.6/54.6
82.8/82.9
80.6/50.6
79.3/79.3
77.5/77.6
79.0/78.9
GCNet 0+
85.1/
82.3/82.3
79.5779.4
77,2/77.2
74.4/74.3
69.8/70.0
66.7/67.7
65.4/65.7
75.1/75.2
MOSI
CIF-MMIN
38
$4.0/83.6
82.5/84,1
82.6/82.7
80.5/80.9
78.4/78.1
77.3/77.2.2
74.1/74.29
69.8/71.3
78.8/78.9
SDR-GNN'
23
$6.3/86.3
85,0/85.1
$1.9881.9
80,7/80.8
77.9/78.0
76.1/76.2
72.2/72.2
71.1/713
78.9/79.0
IMDER
85.7/85.6
84.98448
81,28110
78.6/78.5
76.2/75.9
74.7/74.0
71.9/71.2
79.6/79.3
DSCMar'
IN
85.688577
83.9839
82.02.11
80,2004
77.7/77.9
76.4776.7
73.07333
70.8/71.1
78.7778.9
DCCAE
77,3/77.4
74.3/74.7
71,871.9 9
67.00 07666
63.6/62.8
62.0/61.
59.6/58.5
58. 18.1/57.4
66,7166
FelDISC
86.8/66.4
86.3/68.1
86.466.3
65.6/84.7
84.5/84.3
82.6/83.2
81,7/82.5
81.5/82.2
FedDISC (I)
85.4/85.2
84.8/84.7
$4.0/84.3
83.4/82.3
82.3/82.9
81.9/50.1
80,2/81.2
$0.5/81.0
$2.8/82.8
GCNet' P
85.1/85.2
82.1/82.3
79.9780.3
76.8/77.5
74.9/76.0
73.2/74.9
72.1/74.1
70.4/73.2
76.8/77.9
CIF-MMIN
85.8/86.2
85.885.5
85.085.3
83.1883.8
82.7/22.5
80A/111
78.5/79.2
77.3/77.4
82.3/82.6
SDR-GNNI
87.3/87.4
86.7786.8
9
84.7884.8
83. 8/840
82.6/52.8
81,3/81.6
10
IMDER
1/85/
$4.8/84.6
82.7/82/1
81.3/80.7
79.3/78.1
79.0/77.4 4
78.0/75.5
77.3/74.6
$0.9/79.8
DiCMar
1/85/
83.5/83.7
79.3/79.8
77.4/78.7
75.8/77.7
73.7/76.7
72.2/75.4
DCCAE
81,2/81.2
78.3/78.4
75.4775.5
72.2/72.3
70.0/20.3
66.4N992
63.2167.6
62.6/66.6
71.2/72.6
Table Comparison results on CMUMOSI and CMUMOSEI datasets under different missing rates. indicates the results come from 23
4.2 Comparison with SOTA Methods
We compare our FedDISC approach with state-of-the-art recovery methods under unified environmen-
tal and dataset settings: GCNet 12), MMIN [40). SDR-GNN 23 IMDer [ and DiCMoR (18)
Also, we consider one non-recovery method with classical correlation maximization DCCAE [39]
to make comprchensive comparison. To evaluate the generalization of our proposed framework.
7

---

Avail:
Avail:
Miss: J,
Avail:
Miss:
GCNa
PedDISC()
FedDISC(P)
Marii
Gort
Ogan
Figure 3: The t-SNE visualization compares the modality recovery performance of different methods under single-min modality availability. the features generated by FedDISC exhibit higher distributional similarity to the original modality features compared to other methods, demonstrating its effectiveness.
we develop DISC-Diffusion with two classical diffusion models: DDPM (timesteps = 1000) and DDIM (timesteps 50), denoted FedDISC (P) and FedDISC (I) separately. The results are reported below.
Fixed missing protocol. As listed in Table our proposed FedDISC (P) achieves the highest accuracy and WAFI under nearly all partial modality incomplete conditions. Specifically, the FedDISC (P) delivers optimal classification results across all scenarios the IEMOCAP4 dataset, outperforming the rest-performing SOTA methods by 1.3% ~ 7.5% accuracy. On IEMOCAP6, the FedDISC (P) slightly lags behind the SDR-GNN by 0.9% 3.2% in certain incomplete modality settings (1)) {1, v), {l, a.). However, considering federated clients assigned only Nj C. samples, while other SOTA methods are full training data accessible, this result highlights our method's capability to generate high-quality missing modality features under few-shot environments. Wilcoxon signed- rank tests with the alternative hypothesis
ne
"greater" show W 21.00. p 0.0156 < 0.05, indicating significant superiority of FedDISC(P). The effect size, computed as rank-biserial correlation r (r (r= 2/VN, N = 6), r Ro 0777, reflecting a large effect (r > 0.5).
Random missing protocol. Table 2 illustrates the comparison results under missing rates range from 0.0 0.7. On the CMUMOSI and CMUMOSEI datasets, our FedDISC exhibits slightly lower accuracy and WAFI compared to SOTA models when the missing rate lower than 0.4. However, achieves the highest accuracy and WAFI under missing rates 0.4. This discrepancy arises because,
Non Condition
DGN4SCN
X
19
Anger
Happinoss
Ogm
Figure 4: The visualized ablation study on the IEMPCAP6 dataset. Compared with unconditional modality recovery, DGN and SCN guide recovery by leveraging dialog and semantic alignment, ensuring category consistency between recovered and original modalities.

---

with low missing rates (e.g., < 0.4), modality recovery models are of low importance in classification tasks. While as the missing rate increases, classification performance becomes more dependent on modality recovery capabilities. As shown in the table, FedDISC is less suffered by rising missing rates than other models. Specifically, when the missing rate increases from 0.0 to 0.7, on CMU-MOSI, the accuracy of the compared models declines about 15.4% ~ 27.7% while our FedDISC shows only 12.3% decline. on CMU-MOSEI, the compared models declines by 7.4% ~ 22.9% while FedDISC declines merelly 6.1%. These comparisons validate the generalization capability of FedDISC's recovery model under random missing protocol. Wilcoxon tests yield W = 45.00, p = 0.0020 <
0.05, confirming significant differences. The effect size is 0.81 (N 8), also indicating large effect.
Visualization experiments. Figure 3 iillustrates the distribution plots of missing modality features generated by our FedDISC and other recovery- models under the fixed missing protocol with single modality availability, compared to ground truth features. We conducted experiments on the IEMOCAP's test set from a single client, projecting these features into 2D space via t-SNE. Observations reveal that FedDISC (P) achieves the highest denoising capability, predicting features that closely align with the original distribution. FedDISC (I) slightly underperforms FedDISC (P) in recovering modalities and v, yet still surpasses other recovery methods. This demonstrates FedDISC's effectiveness in ensuring distributional consistency between recovered and original data.
Available
L
L
AFS
ACC4
WAF1
ACC4
WAFI
ACC4
WAFI
ACC4
WAFI
ACC4
WAFT
ACC4
WAFT
x
52.3
50.8
46,8
47.1
50.2
50.5
61.5
61.2
59.7
58.9
66.0
65.8
68.0
68.2
62.5
60.3
61.2
60.6
75.8
75.8
78.0
78.1
68.7
68.7
Table 3: Ablation study under different modality-missiee scenarios on the IEMOCAP4 dataset. This table shows that AFS enables collaborative training between the recovery module and the classification module under any modality-missing conditions.
4.3 Ablation Study
DGN & SCN. To validate the importance of dialogue and semantic dependencies extracted by DGN and SCN in the proposed DISC-Diffusion model, we design a visualized ablation study, As shown in Figure we visualize the recovered and original modality features of IEMPCAP6 under two conditions: 1) Unconditional recovery (no guidance), 2) Jointly conditioned on DGN and SCN. Specifically, we randomly select 120 samples (20 per class) from the IEMOCAP6 test set and project both generated and original features into a 2D space via t-SNE. We can observe that under unconditional recovery setting, recovered features suffer from semantic ambiguity because of the lack of semantic guidance. However, the modalities recovered through conditional recovery guided by DGN and SCN exhibit strong clustering with the original modalities of the same category, while maintaining distinct separation between classes. This validates the effectiveness of DGN and SCN in capturing comprehensive dialogue and semantic information.
AFS. We design ablation experiments to validate the effectiveness of the proposed AFS. On the IEMOCAP4 dataset, we test the AFS-equipped model and a baseline model (where the recovery and classification modules are co-trained without AFS) under various modality-missing configurations. Table 3 shows that the model with AFS outperforms the baseline in all missing-modality modality scenar- ios. This demonstrates that AFS effectively mitigates optimization conflicts between modules and improves training efficiency.
4.4 Computation and Communication Cost
Computation cost. The primary source of computational overhead in FedDISC lies in the inference time of the liffusion-basee recovery module. In our experiments, using DDPM 1000), the aver- age inference time is approximately 2.13 min/epoch, for DDIM (t = 50), this reduces significantly to 5.7 8/ s/epoch. As a result, the total training time varies with the diffusion backbone. Importantly, our feature recovery module is not restricted to diffusion models-aae any conditional generative model with lower inference latency (e.g., conditional autoencoders or GANs) can be flexibly adopted within our framework.

---

Communication cost. The communication cost per client is 10.57 MB/ B//round when using the conditional DDPM. Compared to the traditional federated learning framework, FedAvg, which incurs a communication cost of 22.64 MB /roond, our proposed hierarchical federated framework successfully reduces the communication overhead by more than half. Notably, the diffusion model parameters account for the majority of this cost approximaaty 85%, and our framework allows for any lighter-weight conditional generative model to used as the recovery module. For example, when using DDIM, the communication cost further decreases to 7.18 M /round.. Moreover, the recent paper AT reports that communication costs deployable in real-world scenarios. This strongly supports the practicality and deployability of our approach.
12 M per round per client are fully
Conclusion and Limitations
In this work, we try challenge MERC under extreme incomplete multimodalities by introducing new modality recovery paradigm: FedDISC. We pioneers the integration of federated learning with generative modality recovery mitigate the dependence of recovery models modality integrity. We design DISC-Diffusion with a DGN and a SCN module to capture dialogue and semantic dependencies separately for diffusion guidance. Additionally, the designed AFA strategy balances collaborative training between recovery models and classifiers via periodic freezing optimization mechanism. Comparative experiments and ablation studies validate the effectiveness of FedDISC in missing-modality recovery and collaborative training.
Limitations: First, its sampling strategy can't balance recovery quality and computational cost (e.g.. iterative denoising requires multi-step inference). Future work may adopt consistency models for faster high-quality generation. Moreover, FedDISC assumes synchronous federated training, while real-world scenarios often involve asynchronous devices and dynamic modality absence. Extending it to asynchronous FL frameworks will enhance practicality.
Acknowledgements
This research was supported by Guangdong Basic and Applied Basic Research Foundation (No. 2024A1515011 1774), the National Key Research and Development Program of China (No. 2022YFC3310300, the National Natural Science Foundation of China (No. 12171036), Shenzhen Sci-Tech Fund (Grant No. RUSC2023111000000000
References
[1] Yusong Wang, Xuanye Fang, Huifeng Yin, Dongyuan Li, Guoqi Li, Qi Xu, Yi Xu, Shuai Zhong, and Mingkun Xu. BIG-FUSION: brain-inspired global-local context fusion framework for multimodal emotion recognition in conversations. In AAAI-25, Sponsored by the Association for the Advancement of Artificial Intelligence, February 25 March 4, 2025, Philadelphia, PA. USA, pages 1574-1582. AAAI Press, 2025.
[2] Hui Ma, Jian Wang. Hongfei Lin, Bo Zhang, Yi-Jia Zhang, and Bo Xu. A transformer- based model with self-distillation for multimodal emotion recognition in conversations. IEEE Trunsuctions on Multimedia, 26:776-788, 2024.
[3] Zebang Cheng, Zhi-Qi Cheng, Jun-Yan He, Kai Wang, Yuxiang Lin, Zheng Lian, Xiaojiang Peng. and Alexander Hauptmann. Emotion-llama: Multimodal emotion recognition and reasoning with instruction tuning. Advances in Neural Information Processing Systems, 37: 1 110805-110853, 2024.
[4] Yazhou Zhang, Yang Yu, Qing Guo, Benyou Wang, Dongming Zhao, Sagar Uprety, Dawei Song, Qiuchi Li, and Jing Qin. CMMA: benchmarking multi-affection detection in chinese multi-modal conversations.
Advances in Neural Information Processing Systems 36: Annual Conference Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
[5] Jiayu Ye, Yanhong Yu, Lin Lu, Hao Wang, Yunshao Zheng, Yang Liu, and Qingxiang Wang. Dep-former: Multimodal depression recognition based on facial expressions and audio features
10

---

via emotional changes. IEEE IEEE Transactions on Circuits and Systems for Video Technology. 35(3):2087-2100, 2025.
[6] AV Geetha, T Mala, D Priyanka, and E Uma. Multimodal emotion recognition with deep learning; Advancements, challenges, and future directions. Information Fusion, 105:102218, 2024.
[7] Yusong Wang, Xuanye Fang, Huifeng Yin, Dongyuan Li, Guoqi Li, Qi Xu, Yi Xu, Shuai Zhong, and Mingkun Xu. BIG-FUSION: brain-inspired global-local context fusion framework for multimodal emotion recognition conversations. In AAAI-25, Sponsored by the Association for the Advancement of Artificial Intelligence, February 25- March 4, 2025, Philadelphia, PA, USA, pages 1574-1582. AAAI Press, 2025.
[8] Yuanzhi Wang. Yong Li. and Zhen Cui. Incomplete multimodaliy-diiftusd emotion recognition. Advances in Neural Information Processing Systems, 36: 36:17117-17125 2023.
(9] Rui Liu, Haolin Zuo, Zheng Lian, Björn W. Schuller, and Haizhou Li. Contrastive learning based modality-invariant feature acquisition for robust multimodal emotion recognition with missing modalities. IEEE Transactions on Affective Computing, 15(4):1856-188 1873, 2024.
[10] Xihang Qiu, Wanyong Qiu, Ye Zhang, Kun Qian, Chun Li, Bin Hu, Björn W. Schuller, and Yoshiharu Yamamoto. FedKDC: Consensus-driven knowledge distillation for personalized federated learning in eeg-based emotion recognition. IEEE Journal of Biomedical and Health Informatics, pages 1-14, 2025.
[11] Changqing Zhang, Yajie Cui, Zongbo Han, Joey Tianyi Zhou, Huazhu Fu, and Qinghua Hu. Deep partial multi-view learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(5):2402-2415, 2020.
[12] Zheng Lian, Lan Chen, Licai Sun, Bin Liu, and Jianhua Tao. Genet: Graph completion network for incomplete multimodal learning in conversation. IEEE Transactions Pattern Analysis and Machine Intelligence, 45(7):8419 8432, 2023.
[13] Zhangfeng Hu, Wenming Zheng, Yuan Zong, Mengting Wei, Xingxun Jiang, and Mengxin Shi. novel decoupled prototype completion network for incomplete multimodal emotion recognition. In 2024 IEEE International Conference on Multimedia and Expo (ICME), pages 1-6. IEEE, 2024.
[14] Rui Liu, Haolin Zuo, Zheng Lian, Bjöm W. Schuller, and Haizhou Li. Contrastive learning based modality-invanl invariant feature acquisition for robust multimodal emotion recognition with missing modalities. IEEE Transactions Affective Computing, 15(4):1856-1877 2024.
[15] Jiandian Zeng, Jiantao Zhou, and Tianyi Liu. Robust multimodal sentiment analysis via tag encoding of uncertain missing modalities. IEEE Transactions on Multimedia, 25:6301-6311 2022.
[16] Lucas Goncalves and Carlos Busso. Robust audiovisual emotion recognition: Aligning modal- ities, capturing temporal information, and handling missing features. IEEE Transactions on Affective Computing, 13(4):2156-2170 2022.
[17] Yuanyue Deng, Jintang Bian, Shisong Wu, Jianhuang Lai, and Xiaohua Xie, Multiplex graph aggregation and feature refinement for unsupervised incomplete multimodal emotion recognition. Information Fusion, 14:110271,, 2025.
[18] Yuanzhi Wang, Zhen Cui, and Yong Li. Distribution-consisten modal recovering for incomplete multimodal learning. In Proceedings of the IEEE/CVF International Conference on Computer 2023.
Vision, pages
[19] Shicai Wei, Chunbo Luo, and Yang Luo. Mmanet: Margin- distillation and modality-aware regularization for incomplete multimodal learning. In IEEF/CVF Conference Computer Vision and Pattenn Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pages 20039-20049. IEEE, 2023.
11

---

[20] Nuno C. Garcia, Pietro Morerio, and Vittorio Murino. Modality distillation with multiple stream networks for action recognition. In European Conference on Computer Vision, volume 11212. pages 106-121. Springer, 2018.
[21] Jiawen Deng and Fuji Ren. Multi-label emotion detection via emotion-specifiee feature ex- traction and emotion correlation learning. IEEE Transactions on Affective Computing, 14(1): 475 486, 2020.
[22] Magdiel Jiménez- Guannerss and Gibran Fuentes- Pineea. Cfda-csf: A multi-modal domain adaptation method for cross-subject emotion recognition. IEEE Transactions on Affective Computing, 1523)1150-11513, 2024.
[23] Fangze Fu, Wei Ai, Fan Yang, Yuntao Shou, Tao Meng, and Kegin Li. SDR-GNN:
spectral do- main reconstruction graph neural network for incomplete multimodal learning in conversational emotion recognition. Knowledge-
Systems, 309: 12825, 2025.
[24] Huy Q. Le, Chu Myaet Thwal, Yu Qiao, Ye Lin Tun, Minh N. H. Nguyen, Fui-Nam Huh, and Choong Scon Hong. Cross-modal prototype based multimodal federated learning under severely missing modality. Information Fusion, 122: 103219, 2025.
[25] Kangning Yin, Zhen Ding, Xinhui Ji, and Zhiguo Wang. Self-attention fusion and adaptive continual updating for multimodal federated learning with heterogeneous data. Neural Networks, 187:107345, 2025.
[26] Ming Zhong, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. Dialoglm: Pre- trained model for long dialogue understanding and summarization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 11765-11773, 2022.
[27] Geng Tu, Bin Liang, Dazhi Jiang, and Ruifeng Xu. Sentiment-ermotionaadd context-guided knowledge selection framework for emotion recognition in conversations. JEEE Transactions on Affective Computing, (4(3):1803-11) 1816, 2022.
[28] Michael Sejr Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. In The Semantic Web - 15th International Conference, ESWC 2018, Heraklion, Crete, Greece, June 3-7, 2018, Proceedings, volume 10843, pages 593 607. Springer, 2018.
[29] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas Communication--fficiee learning of deep networks from decentralized data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, AISTATS 2017, 20-22 April 2017, Fort Lauderdale, FL, USA, volume 54, pages 1273 1282. PMLR, 2017.
[30] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. CoRR, abs/2207. 2022.
[31] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Inter- vention - MICCAI 2015 18th International Conference Munich, Germany, October 5 9, 2015, Proceedings, Part III, volume 9351, pages
Springer, 2015.
[32] Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. Iemocap: Interactive emotional dyadic motion capture database. Language Resources and Evaluation, 42:335-359, 2008
[33] Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. Multimodal sentiment intensity analysis videos: Facial gestures and verbal messages. IEEE Intelligent Systems, 31 (6):82-88, 2016.
[34] AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis- Phillppe Morency. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages
2018.
12

---

[35] Zhenyu Yang, Xiaoyang Li, Yuhu Cheng, Tong Zhang, and Xuesong Wang. Emotion recognition in conversation based on a dynamic complementary graph convolutional network. IEEE Transactions on Affective Computing, 15(3):1567-1579, 2024.
[36] Devamanyu Hazarika, Roger Zimmermann, and Soujanya Poria. MISA: modality-invariann and -specific representations for multimodal sentiment analysis. In MM '20: The 28th ACM International Conference Multimedia, Virtual Event / Seattle, WA. USA, October 12-16, 2020, pages 1122-1131. ACM, 2020.
[37] Zhongkai Sun, Prathusha Kameswara Sarma, William A. Sethares, and Yingyu Liang. Learning relationships between text, audio, and video via deep canonical correlation for multimodal language analysis. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, New York, NY, USA, February 7-12, 2020, pages 8992-8999. AAAI Press, 2020.
[38] Rui Liu, Haolin Zuo, Zheng Lian, Bjöm W. Schuller, and Haizhou Li. Contrastive learning based modality-invand invariant feature acquisition for robust multimodal emotion recognition with missing modalities. IEEE Transactions on Affective Computing, 15(4):88561188)) 2024.
[39] Weiran Wang, Raman Arora, Karen Livescu, and Jeff A. Bilmes. On deep multi-view repre- sentation learning. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37, pages 1083-1092. JMLR.org, 2015.
[40] Jinming Zhao, Ruichen Li, and Qin Jin. Missing modality imagination network for emotion recognition with uncertain missing modalities. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1th International Joint Conference on Natural Language Processing, ACL/UJCLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 2608-2618. Association for Computational Linguistics, 2021.
[41] Daixun Li, Weiying Xie, Zixuan Wang, Yibing Lu, Yunsong Li, and Leyuan Fang. Feddiff: Diffusion model driven federated learning for multi-modal and multi-clients. IEEE Transactions on Circuits and Systems for Video Technology,
2024.
[42] Xuejiao Wang, Zhen Hua, and Jinjiang Li. PACCDU: Pyramid Attention Cross-Convolutional Dual UNet for Infrared and Visible Image Fusion. IEEE Transactions on Instrumentation and Measurement, 71:1-16, 2022.
[43] Pengcheng He, Xiaodong I. Liu, Jianfeng Gao, and Weizhu Chen. Deberta: decoding-enhanced bert with disentangled attention. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[44] Zengqun Zhao, Qingshan Liu, and Shanmin Wang. Learning deep global multi-scale and local attention features for facial expression recognition in the wild. IEEE Transactions on Image Processing. 00:6544-6556, 2021.
[45] Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. wav2vee: Unsupervised pre-training for speech recognition. In Gernot Kubin and 7dravko Kacic, editors, 20th Annual Conference of the International Speech Communication Association, Interspeech 2019, Graz, Austria, September 15-19, 2019, pages 3465-3469. ISCA, 2019.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims of our FedDISC in the abstract and introduction reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitations the work are briefly discussed in the conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Justification:
Every theoretical result has an associated proof in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
The paper comprehensively describes the experimental setup, including methodologies, key parameters, and data processing steps in see 4 and the appendix.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
The datasets used in the experiments are publicly available. Code will be released.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results. See the NeurlPS code and data submission guidelines (https /pips.ccc/pulic/guuui guwdes/ddee///e Coc//ooodeeessseeeeeeo hwtpibbmsssiioooooooiooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We describe the experimental setup in sec and provide further details of the experiments in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Error bars are not provided in the paper, due to the experiments are too costly be run multiple times. However, the model is validated on sufficient experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
16

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The required GPU resources are outlined in sec
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We read the Code of Ethics and follow the code carefully.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification:
The societal impacts are discussed in appendix
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification:
The FedDISC introduced in this paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite both the paper and the license of each model and dataset we use.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The assets introduced in this paper (code) will be released before publication with full documentation.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Although this paper utilizes a publicly available dataset, a detailed description of the dataset is still provided in the appendix.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This study exclusively utilizes publicly available datasets and does not involve any direct experimentation with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
19

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: This paper do not use any LLMS.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non stanstaddard components.
Please refer to our LLM policy for what should or should not be described.
20

---

A Theoretical Analysis
Assumption (Noise-Prediction Error Model) For all clients i and timesteps
E ( -
-0, (ze. t, c)] = 0, Ezo.e [||e - c02 (2t, t, c)]l²]
I &Eastt
(9)
Assumption (Conditioning Embedding Lipschitznes) For the dialogue graph encoder EDGN and semantic encoder EsCN, we require
EDDN (x) - EDGN(x'))| & LDGN I - 2'11
(10)
[s)N(() (scc((')]] S LScx ||x - x'1l-
We then set Liot == LDGN + LSCN- Assumption (Aggregated Parameter Bias) FedAvg after one communication round yields
|10, 0*1 S dagx: 11,,- S dags:
(11)
where {0', 3 minimises F()) + G( on the pooled (non-federated) data.
Assumption (Smoothness + Polyak-Foojssiwiie)) Both blocks satisfy
VVF(0) [F - M ||e 8'1,, | VF(0))2
2 U[F(())F((B))) F(0)))
(12)
VV - VG(o')| S M | o'I1 1/ [V(())|
2 2 G(
Assumption (Bounded SGD Variance) For any client gradient estimator g.
Eig] Blgg vell
0
(13)
Theorem (Convergence of Recovery-Onle Rounds) Run FEDDISC for T global rounds with step-size 1j S 1/M, E local steps, K clients chosen i...d,, then
M 0 11
1 EE
E F(O - F(O*) s
+
+ M 52 agg
(14)
2
K n
1
Proof. One client, one step. By M -moooothssss:
Mn
F(0 ng) < F(0) - 7(VF(0), g) +
11a11
(15)
2
Take expectation over the stochastic gradient and use Assumption
Mn²
EF(()) < F(0) ( v|(
2 (IVF(()) + +
(16)
Since 77 S 1/M, we get:
Mijo²
S =F(B')=F(()
" VF(0))2
(17)
2
2
PL condition Assumption 4
|FF > (F(()))F((')))
(18)
Apply Equ. 17 for E local steps:
(I)( F(O)) Ap Mn2
A. = F() -(())
(19)
2
FedAvg aggregation bias Assumption
2o E
A S (1 A,
+ M 82
(20)
agg
Solve linear recursion. Noting 1 EEE 5 E - and summing over t = O,... TI,
M 1|00 0-1 EE
AT <
+ M 62 agg"
(21)
T
K₁
Add irreducible diffusion gap. The imperfect predictor incurs an additive floor 2 /p. Combining with equ. (21) yields the final bound.
21

---

Theorem (Error Bound for Recovered Latent Vectors) After Teev reverse diffusson steps, the recovered latent vectors satisfy
Ceum Leot Editt + Mattn
(22)
Proof. From the reverse mean formula and Assumption D
li+e - pill = Ce ||eg - E|| S Ce Ediff.
(23)
Conditioning perturbation at step t:
||| c/l Liot Ce Eaitf
(24)
Because the U-Net cross-attention the next noise prediction error.
42 is affine in ct, the bound Equ. (4 re-enters additively into
T S
E
2,
2
Luo Ct Ediff + Matta Ceum Leon Ediff + Mattn
(25)
tm
Theorem (Linear Rate of Alternating-Freest Optimisation) With alternating blocks A : 0- updates, B d-updates, slep-sizes 1JA 1/8 S 1/M, and executing full alternations, have:
Q(OR, o Q(0", . S (1 31 " "1Q(0) Φ0) Q(0", "')],
(26)
where () F(0) G(d).
Proof. Define the block operator
(0, + (0- 7I VFFO), 6- nB VG())).
(27)
One A-update with @ frozen. By the PL condition for F:
F(0+) F(0") S (1 7AHH) [F(0) F(0'))
(28)
One B-update with frozen. Similarly for G, G( G(ø*) S (1 781) [G(Ø) G(D*)]
(29)
Combine. Let Q(0,0) F(0) G(Ø) and define
max 1 NAH, 1 mann < -
(30)
M
Then after one full alternation:
Qt Q- 2--Spp sp - -Q").
(31)
After R alternations. By recursion.
((RR,")) Q- s <p0(0 00) Q-1 = (1 - - 10000 600) -
(32)
22

---

C
The set of utterances Number of utterances in a conversation
C
U4
The i-th utterance Speaker identity of utterance Unified feature space (e.g., H (h("))
conversation
(u))
Feature representation of modality m for utterance
w
Fixed window size for graph neighborhood construction
of, of
Outputs of Speaker Graph and Context Graph for utterance Output of DGN for modality m Local dataset of client [
Di
Ni
Sample number of client I's dataset Set of available modalities for client Outputs from DGN and SCN modules Conditional embedding for diffusion guidance
Mavall
2" 2,
F()
Input features at the /-th layer of the U-Net
Gaussian noise in the diffusion process Noise scheduling coefficients at timestep
Og, Oet gm
Global diffusion model parameters for modality
g
Og
Global classifier parameters
E
Alternating communication interval of AFS
St
Subset of clients selected in communication round
Number of local training epochs per client
Lable of the dataset
Dd: Da:
Predictions of DGN, SCN, and classifier module
Table 4: Main Notations Used in this Work. This table highlights the key notations and symbols used throughout the paper, providing a concise reference for understanding the mathematical formulations and models presented.
B Algorithm Analysis
In Algorithm SCN.
we present the pretraining procedure for the conditional capture networks DGN and !
Algorithm illustrates the training process FedDISC, detailing how AFS alternately freezes recovery and classifier modules to enable collaborative optimization.
C Information Redundancy Analysis
To address potential concerns regarding information redundancy, we further analyze whether the recovered modality features in framework merely replicate the information contained in the existing modalities. Information redundancy occurs when reconstructed features are highly correlated with, simply duplicates of, the available modalities, thereby failing contribute complementary cues for downstream learning. In our design, the DGN and SCN are explicitly introduced to alleviate this issue by jointly modeling local semantic patterns and global dependencies. Consequently, the recovered modalities are not direct copies of the observed ones but context-enriched representations that capture higher-level dependencies. This interpretation is supported by our experimental results in Table I if substantial redundancy existed, the performance of the model with recovered modalities would approximate that of the Baseline. However, both FedDISC(P) and FedDISC(I) achieve statistically significant improvements under all missing- modality conditions cone-sided Wilcoxon signed-rank test. W 0, p 0.03125 0.05 on JEMOCAP4 and IEMOCAP6), confirming that the recovered modalities contribute non-redundant information to the overall framework.
D Data Missing Protocols and Federated Mechanism
As illustrated in Figure (a), missing modality challenges MERC can be categorized into two scenarios [[] random missing protocol and fixed missing protocol. Figure (b) shows brief
23

---

Algorithm 1: Pre-training Process of DGN & SCN Modules
Input:
Client datasets (Di) window size w. pretraining epochs Output: Pretrained DGN parameters ② DGN, SCN parameters ScN Initialize DON and OsNN
for epoch = .....,, do
for each client E [K] in parallel do
DGN Forward: for each available modality m E M do
Build speaker graph 9 and context graph G with window w; Perform R-GCN aggregation via (3.(3) to get (umm, "
2,"
end
(=)" ÿa f Allentiom (Concal( (z }));
SCN Forward: < Encoding( (DT*)) }). m E Mf
Compute cross-attention and self self-attention layers get {sm);
Ú. + MLL(Concat((s" {))
Joint Optimization: Calculate LDON = [C
Σ yo log Úa:
Calculate LSCN = - Σ yh log
Update DGN and Osox via (LDGN + LSON);
end end
return ODGN, Oscx;
Random Missing
Fixed Missing
Server
Server
Spenker
Sperkar R
Speakorr A
Sponker B
Lanquage
Client
Cliend
Client
Client it
Vision
Accuslic
Missing
(b)
Figure 5: (a) illustrates two types of incomplete modalities: random missing protocol and fixed missing protocol. (b) presents the federated generative modality recovery paradigm proposed in this work, designed to alleviate the dependency of generative recovery on data completeness.
describe of our federated learning mechanism. The clients first train local specialized diffusion models tailored their available modalities. Then through server-side aggregation, unified modality models are constructed and distributed to modality-deficient clients. By eliminating the need for clients to locally train recovery models for missing modalities, this framework overcomes the limitations caused by single-client incomplete modalities.
E Dataset Segmentation and Preprocessing
Dataset: As listed in Table 5 IEMOCAP4 includes four types of emotions: anger, happiness (where excitement is merged with happiness), sadness, and neutral (23). We assign 3290, 1000, and 1241 utterances for train, valid, and test. The six-class dataset IEMOCAP6 encompasses: anger, happiness, sadness, neutral, excitement, and frustration. We assign 4810. 1000, and 1623 utterances for train, valid, and test. CMU-MOSI consists 2199 utterances, where 1284, 299, 686 samples are set for train, valid, and test. CMU-MOSEI contains 22856 utterances, where 16326 used for training, 1871 and 4659 samples are used for validation and testing.
Preprocessing: For each modality, employ the corresponding pre-trained network to perform feature extraction. 1) Language: Pre-trained DeBERTa [43 is employed as the language feature extractor. Motivated by its demonstrated superiority in natural language understanding and generation
24

---

Algorithm 2: Federated Training Process of FedDISC
Input:
Client datasets (Di global diffusion models {0TH JMEM, alternative interval Output: Global recovery models {6T ), global classifier 0y Initialize global models f + mm <) 4- Φg0-
for t 1....,T communication rounds do if 1%E odd (Stage I: Recovery Training) then
for each client E S₁ in parallel do
Freeze classifier module oz;
for each available modality m E M. do
Train local recovery model 0f via (q.(8) using Df",; end
Upload { to server; end
Server aggregates f + E1
PP ETP, 0T:
Broadcast {0m)) all clients;
else if t%E is even (Stage II: Classifier Optimization) then
for each client I E S; in parallel do
Freeze recovery modules {0m J
Recover missing modalities 2m < DISC- BISS--ifffssio(PPe (friis Devail); );
Train classifier Φt via = Σ Vs log Ú::
Upload &i server;
end
Server aggregates 6g + -, D2 Broadcast 0g all clients;
01;
Σ
end end
return (0g) }, Ögi
Dataset
Train
Val
Test
Total
IEMOCAP4
3,290
1,000
1,241
5,531
IEMOCAP6
4,810
1,000
1,623
7,433
CMU-MOSI
1,284
229
686
2,199
CMU-MOSEI 16,326
1,871
4,659
22,856
Table 5: Statistical information on IEMOCAP, CMU-MOSI and CMU-MOSEL
tasks, we leverage the DeBERTa-large variant to encode utterance sequences into 1024 424--iiensional representations. Vision: The pre-trained MA-Net 44 serves the visual feature extractor, utilizing global multi-scale and local attention mechanisms to handle occlusions and non-frontal poses. We first apply MTCNN detect and align faces, followed by extracting facial features via pre-trained MA-Net. Frame-level features are then compressed into 1024-dimensional utterance-level representations through average encoding. 3) Acoustic: Pre-trained wav2vec [45) serves as the acoustic feature extractor, leveraging its multi-layer convolutional architecture trained massive unlabeled speech data. Building on its demonstrated success downstream applications like speech recognition, adopt wav2vec-large to extract 51 112-dimensional acoustic features from utterances.
F
Deep Analysis of Visualization Experiments
Figure 6 as supplement to Figure 3 illustrates the distribution of features recovered by different modality recovery methods compared to the ground truth, under all modality incomplete conditions. All experiments were conducted on the IEMOCAP dataset. We can observe that, regardless of the modality incomplete condition, the distribution of the features recovered by FedDISC is closer that of the ground truth compared to SOTA methods.
25

---

Miss:
Avail: 1.4
Miss:
Miss: J.
Avail: Z
Miss:
Avail: P
Miss:
Avaãl: A Missc P.E.
SDR-GNN
MMIN
IMDer
GUNe
FelDISOO)
FadDISC(P)
dgen
Figure 6: t-SNE visualization: comparison of recovered versus original feature distributions by different modality recovery methods under all modality incomplete conditions on IEMOCAP.
G
DGN and SCN
Figure 7 demonstrate how the integration of DGN and SCN modules capture dialogue dependencies and semantic information, validating their effectiveness guiding the conditional diffusion process. In addition to the conclusions observed in sec 4.3) we can further deduce the following: 1) When single module is used, compared to DGN, the semantic information extracted by SCN provides more effective guidance for the modality recovery process. 2) The combination of DGN and SCN modules, as opposed to using a single module, offers more comprehensive dialogue-semanti semantic information, thereby further mitigating semantic confusion in the recovered modalities.
Table extennss the aforementioned ablation experiments. We conducted tests on the IEMOCAP4 dataset under six incomplete modality conditions, evaluating four ablation settings and recording their ACC and WAF1, with highest metries highlighted dark green. The table clearly demonstrates that when both SGN and SCN are used as conditional guidance, our model achieves the highest performance across all incomplete modality scenarios. This further validates that the integration of dialogue dependencies and semantic information provides superior guidance for modality recovery.
26

---

Non Condition
DGN Only
SCN Only
DGN+SCN
Anger O
Happiness
V Sadness
Neutral X Excitement
Frustration
Lori
aori
Vori
ygn
Oyen
gen
Figure 7: t-SNE visualization of feature distributions under different ablation settings IEMCAP6 I) Unconditional diffusion, 2) DGN only guidance, 3) SCN only guidance, 4) DGN+SCN guidance, colors represent different modalities.
Available
v
1,
l
DGN
SCN
ACC
WAFI
ACC
WAFI
ACC
WAFI
ACC
WAFI
ACC
WAFI
ACC
WAFI
X
x
31.2
15.4
35.6
18.7
38.4
24.5
36.5
25.9
38.9
27.6
32.7
22.4
x
/
63.3
63.1
57.4
56.6
52.5
51.2
71.9
71.2
74.3
74.1
63.8
62.9
/
X
62.4
62.6
58.2
55.9
52.4
51.8
69.5
67.9
72.8
72.5
64.4
64.0
/
v
68.0
68.2
62.5
60.3
61.2
60.6
75.8
75.8
78.0
78.1
68.7
68.7
Table 6: Ablation study results comparing model performance with four ablation settings under various modality incomplete scenarios on the IEMOCAP4 dataset.
H Effectiveness of AFS
Figure Bere@Joonn complements the ablation experiments for AFS presented Table illustrating the accuracy variation trends of models with and without AFS under all incomplete modality conditions. The figure clearly shows that, across all modality- miising scenarios, models incorporating AFS consistently achieve higher accuracy than those without AFS. The reasons for this can be attributed to the following:
Differing optimization objectives: The diffusion model and the classifier pursue distinct optimization goals. During simultaneous training (without AFS), their gradient updates may counteract each other introduce noise, impeding the model's convergence optimal solution. Impact of
27

---

Acnil: :1
Ancil: a
Asail a
70-
70-
70-
Acc
20
01
Commanacatooooooomommmo Rounds
Comminaiiaoon Bounds
Cormramccatonn Bounds
Avail: e
Avail: a,
Avail 1,a
00
10
156
Acc 1960
10
-0-
10
415
30
30
10
16
Communiction Roundss
Cpnmuicctionn Rounds
Communication Plcunds
Figure 8: The ACC variation curves models using AFS compared to models without AFS under all incomplete modality conditions.
modality recovery quality: The quality of modality recovery by the diffusion model directly affects the reliability of the input data for the classifier. the diffusion model is inadequately trained, its outputs may contain residual noise or artifacts, compelling the classifier to learn erroneous feature associations.
AFS addresses these challenges by decoupling the optimization objectives through hierarchical training, which alternately freezes the parameters of the two modules, thereby mitigating gradient conflicts.
Metrics of Different Categories
Figure Darlliilluue illustrate the precision and recall, respectively, FedDISC and other methods for different categories under various incomplete modality conditions on the IEMOCAP4 dataset. The results in Precision show that FedDISC (P) achieves superior performance across all subgraphs with the largest enclosed area. Notably, it attains nearly 80% precision the Anger and Happiness classes under unimodal conditions (c.g., text-only or vision-only) while maintaining competitive accuracy for Sadness and Neutral. In multimodal scenarios, FedDISC (P) further demonstrates strong integration capabilities, significantly outperforming baseline methods. These findings highlight FedDISC's robustness under modality incompleteness. The results in Recall show that both FedDISC (I) and FedDISC (P) consistently achieve the highest recall across most emotion categories and all six modality settings, significantly outperforming the three SOTA models in challenging unimodal or bimodal scenarios with missing modalities. These results highlight FedDISC's superior robustness and generalization capabilities under partial modality conditions, underscoring its effectiveness when certain modalities are unavailable.
Broader Impacts
This study integrates federated learning with conditional diffusion models address the dual challenges of modality incompleteness and privacy preservation in multimodal dialogue emotion recognition, enabling technological advancements healtheare and intelligent customer service. By leveraging distributed collaborative training and modality completion mechanisms, our framework ensures user data privacy while enhancing model adaptability partially observed modalities. This provides efficient solutions for depression screening and personalized mental health interventions.
28

---

Prxxisin Argy
Angy
Angy
avnii: 1
avaii a
Hap
Sod
Hapgy
Std
1appy
Nattal
Natui
Nactol
Angy
Argy
Ange
avaii: 1,
avail 1,a
avoal v,a
Sad
Hap
Sad
Hap
Sid
Happy
Recall
avzil
avadd
avsid
Hepyy
Higgyr
Happy
Nextral
Natmi
Nextal
Angy
Argy
Angyy
avail: 1
1
avail 25
Sadd
1appy
Sed
Hapgy
Sed
happy
Natua
OUIE
SOR-GNN
JMDer
Figure 9: The precision and recall of FedDISC and other methods for different categories under various incomplete modality conditions on the IEMOCAP4 dataset.
29

---

Furthermore, the proposed method demonstrates promising applications in cross- institutnnt medical data sharing, effectively mitigating data silos and advancing Al-driven precision medicine.
Despite its benefits, FedDISC may pose potential risks if deployed without careful consideration. Biases in client-speciff training data (e.g., cultural or demographie imbalances in emotion expression) might propagate into the global model, exacerbating fairness issues in cross-client deployments. Moreover, the computational demands of iterative diffusion sampling and federated aggregation could disproportionately exclude resource-constraingg learning ecosystems. Mitigating these risks requires rigorous audits of recovered modality fidelity, fairness-awarr aggregation strategies, and energy-efficient implementations to align with ethical AI practices.
participants, reinforcing inequities in collaborative
30


Reviews:
[{"review_id": "fowFd2F03J", "reviewer": "Reviewer_pKQY", "summary": "This paper addresses the problem of missing modalities in multimodal dialogue emotion recognition by proposing the novel FedDISC framework, which pioneers the integration of federated learning into missing-modality recovery. During recovery, it leverages pretrained Dialogue Graph Network (DGN) and Semantic Conditioning Network (SCN) to enforce consistency between recovered and available modalities along the three dimensions of context, speaker identity, and semantics. Moreover, it introduces an Alternating Frozen Strategy to resolve cross-modal optimization conflicts. The manuscript is logically coherent and rigorously organized, and its advantages are validated through comprehensive experiments.", "questions": "1:  The framework is complex—how much higher is its computational cost compared to other architectures? If this overhead remains within an acceptable range, I believe the framework’s advantages are indeed significant.\n\n2:  Provide a concise, end-to-end overview of Figures 1 and 2; I am confident this will help readers grasp the workflow more clearly.\n\n3:  In the ablation study, only 120 samples were selected for evaluation. Could this sample size be too small and lack sufficient randomness? Furthermore, when designing the ablation experiments for DGN and SCN, is it possible to validate their individual contributions separately?\n\n4:  After recovering the missing modalities, have the authors considered the potential issue of information redundancy?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths: \n1: The paper is highly innovative, successfully introducing federated learning into missing-modality recovery and offering a new approach for multimodal dialogue emotion recognition under incomplete modalities. \n\n2:  The manuscript is logically rigorous in structure, and demonstrates the method’s strengths through experiments with different missing-modality configurations, varying missing rates, and comprehensive ablation studies.\n\n### Weaknesses:\n1:  The framework is complex, and its computational cost relative to other methods remains unclear. \n\n2:  After presenting the methodology in Section 3, a concise overview of the end-to-end workflows shown in Figures 1 and 2 would further aid reader comprehension.\n\n3:  In the ablation study, only 120 samples were selected for evaluation. Could this sample size be too small and lack sufficient randomness? Furthermore, when designing the ablation experiments for DGN and SCN, is it possible to validate their individual contributions separately?", "was_revised": true, "final_justification": "Thank you for your response. After reading comments from other reviewers and the responses from authors, I decide to keep my original score.", "rebuttal": "**Question1 \\& Weakness1：**\n\nThank you for your thoughtful question regarding computational overhead. We provide a detailed clarification as follows:\n\nComputational Cost:\n\nThe primary source of computational overhead in FedDISC lies in the inference time of the diffusion-based recovery module. In our experiments, using DDPM (t=1000), the average inference time is approximately 2.13min/epoch; for DDIM (t=50), this reduces significantly to 5.7s/epoch. As a result, the total training time varies with the diffusion backbone. Importantly, our feature recovery module is not restricted to diffusion models—any conditional generative model with lower inference latency (e.g., conditional autoencoders or GANs) can be flexibly adopted within our framework.\n\nCommunication Cost:\n\nThe communication cost per client is 10.57 MB/round when using the conditional DDPM. Compared to the traditional federated learning framework, FedAvg \"Communication-efficient learning of deep networks from decentralized data.\" ( McMahan, et al. PMLR, 2017.), which incurs a communication cost of 22.64 MB/round, our proposed hierarchical federated framework successfully reduces the communication overhead by more than half. Notably, the diffusion model parameters account for the majority of this cost—approximately 85%, and our framework allows for any lighter-weight conditional generative model to be used as the recovery module. For example, when using DDIM, the communication cost further decreases to 7.18 MB per round. Moreover, the recent paper “FedDiff: Diffusion Model Driven Federated Learning for Multi-Modal and Multi-Clients” (Li et al.,  IEEE TCSVT 2024) reports that communication costs of 5–12 MB per round per client are fully deployable in real-world scenarios. This strongly supports the practicality and deployability of our approach.\n\nWe sincerely thank the reviewer for raising this insightful question. We will include a detailed discussion of the communication cost in Section K of the appendix (“Communication Cost”) in the revised version.\n\n**Question2 \\& Weakness2:**\n\nThank you for highlighting the need for a clearer workflow illustration. Below we provide an end-to-end summary aligned with Figures 1 and 2:\n\nFigure 1: The Dialogue Graph Network (DGN) and Semantic Conditioning Network (SCN) are first pretrained to capture conversational and semantic dependencies from available modalities. DGN builds speaker and context graphs to model dialogue flow and identity, while SCN uses cross- and self-attention to extract semantic information.\n\nFigure 2: The training proceeds in alternating phases:1.Recovery Module Training: Each client uses its available modalities, with DGN and SCN guidance, to train a local diffusion model for missing modalities. These models are aggregated on the server and broadcast back to all clients; 2.Classifier Optimization: With global recovery models fixed, clients reconstruct missing features, forming full-modality representations for emotion classification. Classifier parameters are then aggregated in the server.\n\n**Question3 \\& Weakness3:**\n\nWe appreciate the reviewer’s careful attention to the ablation design.\n\nOur selection of 120 samples (20 per class) for the t-SNE visualization ablation is inspired by the sampling strategy used in prior work:  “Incomplete Multimodality-Diffused Emotion Recognition” (Wang et al., NeurIPS 2023), where the authors use 70 randomly selected samples (10 per class) for a similar ablation. Compared to this, we chose a slightly larger and class-balanced random subset to enhance representativeness and generalizability. \n\nWe appreciate your suggestion to further clarify the roles of DGN and SCN. In fact, we have conducted detailed ablation experiments, and the individual and combined contributions of DGN and SCN are reported in Table 6 and Figure 7 in the supplementary material (Section F). The results demonstrate that 1) Compared to DGN, SCN provides more effective semantic guidance for modality recovery. 2) Combining DGN and SCN offers more comprehensive information, further reducing semantic confusion in the recovered modalities.\n\n**Question4:**\n\nThank you for this important question regarding information redundancy. \n\nInformation redundancy refers to a scenario where the recovered modality features are highly correlated with, or merely duplicates of, the existing modalities, thus failing to provide additional or complementary information for the downstream task.\n\nIn our framework, the design of the DGN and SCN helps mitigate this issue. Specifically, these modules capture not only local semantic information within each utterance but also incorporate global dependencies, such as speaker relationships and dialogue context, across the entire conversation. As a result, the recovered modalities are not simple \"copies\" of the available modalities, but rather enriched features that embed complementary and context-aware cues.\n\nThis conclusion is supported by the empirical results in Table 1. If severe information redundancy existed, the results with recovered modalities would be similar to those obtained without recovery (i.e., the Baseline). However, in both the IEMOCAP4 and IEMOCAP6 datasets, our models (FedDISC(P) and FedDISC(I)) achieve substantial improvements over the Baseline.\nSpecifically, we performed a one-sided Wilcoxon signed-rank test across the six missing modality cases in IEMOCAP4, and found that the performance gain of FedDISC(P) and FedDISC(I) over Baseline are all statistically significant (W=0, p=0.03125 < 0.05).\nSimilar trends hold for IEMOCAP6. This strongly indicates that our recovered modalities provide complementary and non-redundant information, leading to real gains in performance.\n\nWe will make this clarification more explicit in the revised appendix (Section F DGN and SCN). Thank you again for raising this valuable point."}, {"review_id": "N9WNw22ZOe", "reviewer": "Reviewer_12cs", "summary": "This paper introduces FedDISC, a federated learning method for multimodal emotion recognition in conversations (MERC) under missing modality scenarios. The method integrates generative modality recovery via diffusion models with federated model aggregation. Key components include (1) DISC-Diffusion modules leveraging a Dialogue Graph Network (DGN) and Semantic Conditioning Network (SCN) for context and semantic consistency, and (2) an Alternating Frozen Strategy (AFS) to mitigate optimization conflicts during federated updates. Experiments on IEMOCAP, CMU-MOSI, and CMU-MOSEI demonstrate competitive performance under random and fixed missing modality protocols.", "questions": "1. Can you report the inference time and total communication cost for diffusion models per round? Is real-time deployment feasible?\n2. How does FedDISC scale with the number of clients? Is there any empirical or theoretical evidence for this?\n3.  Were standard data splits used in the evaluation, such as fixed Session 5 for IEMOCAP? If not, comparison fairness is undermined.\n4. Can you provide effect sizes or statistical significance tests for the main results?\n5. Are there plans to extend FedDISC to asynchronous federated learning or real-world settings with heterogeneous clients?", "limitations": "The authors partially address limitations in Section 5. Acknowledging the sampling inefficiency of diffusion models and the lack of asynchronous training support is a good practice. However, practical constraints such as computational cost, deployment feasibility, and generalization to unseen domains are insufficiently discussed.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. Well-motivated, addressing real-world problem of multimodal data incompleteness with privacy constraints.\n2. Federated generative recovery method is a conceptually interesting direction for MERC.\n3. The DISC-Diffusion module design (DGN+SCN) is methodologically reasonable for enforcing multidimensional consistency in recovered modalities.\n\nWeaknesses:\n1. Most technical components (e.g., R-GCNs, conditional diffusion, federated averaging) are direct adaptations of known methods. Originality lies primarily in system-level integration, rather than algorithmic innovation.\n2. Exact encoder architectures, training configurations are not specified in the main paper.\n3. No details on the number of denoizing steps during inference are provided.\n4. There are ambiguities in notation.\n5. Missing explicit clarification as to whether standard splits (e.g., Session 5 as a test for IEMOCAP) were followed or randomized in the cross-validation process.\n6. No ablation for computational cost or real-time feasibility.\n7. Claims about generalization and practical deployability are premature. No cross-corpus transfer experiments or asynchronous client tests, despite acknowledging this as a limitation.\n8. Typo: \"spealer-aware\" to \"speaker-aware\" and \"to recovery the feature\" to \"to recover the feature\".", "was_revised": true, "final_justification": "The paper has several strengths. It tackles an important issue and offers a well-structured solution. Although the technical aspects are based on established methods and do not propose any entirely novel methods, some minor modifications are required. These issues are not sufficient to reject the work. In their rebuttal, the authors address questions regarding speed, scalability, and statistical validation, as well as discuss weaknesses. Their responses have motivated me to change my score to borderline accept. I recommend including the additional details from the rebuttal in the final version of the paper.", "rebuttal": "**Question1 \\& Weakness6：**\n\nThank you for this valuable question. Based on our experimental results and a comparison to recent literature, we believe real-time deployment of our framework is feasible.\n\nInference Time:\n\nIn our experiments, using DDPM (t=1000), the average inference time is approximately 2.13 minutes per epoch. When using DDIM (t=50), this reduces significantly to 5.7 seconds per epoch. \n\nCommunication Cost:\n\nThe communication cost per client is 10.57 MB/round when using the conditional DDPM. Compared to the traditional federated learning framework, FedAvg \"Communication-efficient learning of deep networks from decentralized data.\" ( McMahan, et al. PMLR, 2017.), which incurs a communication cost of 22.64 MB/round, our proposed hierarchical federated framework successfully reduces the communication overhead by more than half. Notably, the diffusion model parameters account for the majority of this cost—approximately 85%, and our framework allows for any lighter-weight conditional generative model to be used as the recovery module. For example, when using DDIM, the communication cost further decreases to 7.18 MB per round.\nMoreover, the recent paper “FedDiff: Diffusion Model Driven Federated Learning for Multi-Modal and Multi-Clients” (Li et al., IEEE TCSVT 2024) reports that communication costs of 5–12 MB per round per client are fully deployable in real-world scenarios. This strongly supports the practicality and deployability of our approach.\n\nWe sincerely thank the reviewer for raising this insightful question. We will include a detailed discussion of the communication cost in Section K of the appendix (“Communication Cost”) in the revised version.\n\n**Question2:**\n\nFor scalability, we first provide a theoretical analysis:\nOur hierarchical federated framework, during the training of the recovery module and the classifier module, can be regarded as two independent FedAvg frameworks. Both strictly follow the synchronous federated learning paradigm, which has been well established in both theory and practice to exhibit linear scalability with respect to the number of clients.\nAs pointed out in  \"Communication-efficient learning of deep networks from decentralized data.\" ( McMahan, et al. PMLR, 2017.), the per-client communication cost and aggregation operations do not increase with the total number of clients; both the communication and local storage burdens per client are O(1). Only the server’s total bandwidth increases linearly (O(N), where N is the number of clients), while each client’s overhead remains constant, making horizontal scalability straightforward.\n\nWe note that, due to the limitations of the dataset, it is not feasible to fairly investigate the effect of increasing the number of clients on model performance in our experiments. For example, when using **four clients** on the MOSI dataset with a missing rate of 0.1, the result is **84.3/84.8**, which is slightly lower than the results with three clients reported in this paper. However, this reduction can be attributed to the fact that distributing the dataset across four clients reduces the amount of local data available to each client. Additionally, variations in client numbers introduce confounding factors that affect the results. Due to the dataset's limitations, it was challenging to isolate and control these variables effectively. We believe this constraint highlights an important direction for future work, where larger datasets or alternative data partitioning strategies could enable a more robust analysis of client scalability.\n\n**Question3 \\& Weakness5:**\n\nWe thank the reviewer for their concern regarding the use of standard data splits, and its impact on comparison fairness. To clarify, our experiments on IEMOCAP4 and IEMOCAP6 (Sec. 4.1) adopted the five-fold cross-validation protocol from \"GCNet: Graph Completion Network for  Incomplete Multimodal Learning in Conversation\" (Zheng et al. IEEE TPAMI, 2023), a widely used benchmark in multimodal emotion recognition. Specifically, we randomly divided the IEMOCAP dataset into five folds, using one fold as the test set and the remaining four for training in each iteration, with the final results averaged over the five runs (Table 1).\n\n**Question4:**\n\nWe thank the reviewer for requesting effect sizes and statistical significance tests for our main results. To validate FedDISC(P)’s superiority, we conducted Wilcoxon signed-rank tests with the alternative hypothesis set to “greater” (i.e., FedDISC(P) outperforms baselines) on ACC across modality-missing conditions in Tables 1 and 2, accounting for small sample sizes and non-normal distributions.\n\nTable 1 (IEMOCAP4 and IEMOCAP6): We compared FedDISC(P) with baseline across 6 modality-missing conditions using ACC. Results show W = 21.00, p = 0.0156 < 0.05, indicating significant superiority of FedDISC(P). The effect size, computed as rank-biserial correlation r (r = Z / √N, N=6), is r ≈ 0.77, reflecting a large effect (r > 0.5).\n\nTable 2 (CMU-MOSI and CMU-MOSEI): For 8 modality-missing conditions, Wilcoxon tests yield W = 45.00, p = 0.0020 < 0.05, confirming significant differences. The effect size is r ≈ 0.81 (N=8), also indicating a large effect.\n\n**Question5:**\n\nWe thank the reviewer for the question regarding plans to extend FedDISC to asynchronous federated learning (FL) and real-world settings with heterogeneous clients. As noted in Sec. 5 (Conclusion and Limitations), our current framework employs synchronous FL. Recognizing the efficient communication potential of asynchronous FL, we plan to extend FedDISC to this paradigm in future work.\n\nAdditionally, to enhance real-world applicability, we intend to extend FedDISC to handle heterogeneous client settings. This includes addressing variations in client data distributions and computational capabilities, using strategies such as personalized FL . Thank you again for your insightful feedback.\n\n**Extension:**\n\nWe provide responses here to the issues raised by the reviewer in the \"Strengths and Weaknesses\" . Weakness 5 and Weakness 6 have already been addressed in our previous responses above.\n\n**Weakness2:**  Exact encoder architectures, training configurations are not specified in the main paper.\n\nThank you for your question. Our modal-specific encoder consists of a pretrained feature extractor followed by a linear layer. Detailed descriptions of the feature extractors used for each modality can be found in Appendix D (Dataset Segmentation and Preprocessing).\n\n**Weakness3:**  No details on the number of denoizing steps during inference are provided.\n\nWe appreciate your careful reading of our manuscript. The number of denoising steps for both diffusion models is specified in Section 4.2, “Comparison with SOTA Methods”: for DDPM, we use 1000 steps, and for DDIM, we use 50 steps.\n\n**Weakness4:**  There are ambiguities in notation.\n\nThank you for pointing this out. We appreciate your attention to detail. We will thoroughly review the manuscript to clarify any ambiguous notations and ensure all symbols and terms are clearly defined in the revised version.\n\n**Weakness8:**  Typo: \"spealer-aware\" to \"speaker-aware\" and \"to recovery the feature\" to \"to recover the feature\".\n\nThank you for catching these typographical errors. We will correct \"spealer-aware\" to \"speaker-aware\" and \"to recovery the feature\" to \"to recover the feature\" in the revised version. We will also carefully check the manuscript for any other spelling or grammatical issues and make all necessary corrections."}, {"review_id": "JhfdzapiwR", "reviewer": "Reviewer_nTQZ", "summary": "The paper proposes FedDISC, a novel framework for Multimodal Emotion Recognition in Conversations (MERC) that addresses the challenge of missing modalities in real-world scenarios. Unlike conventional methods that rely on complete multimodal data, FedDISC integrates federated learning with diffusion models to recover missing modalities across distributed clients. The DISC-Diffusion module ensures semantic consistency between recovered and available modalities through dialogue-aware modeling and semantic conditioning. An Alternating Frozen Aggregation strategy enables collaborative training of recovery and classification modules. Experiments on IEMOCAP, CMUMOSI, and CMUMOSEI datasets show that FedDISC outperforms existing approaches under various missing-modality conditions.", "questions": "1. The authors are encouraged to better clarify the distinctions and advantages of their proposed method compared to previous approaches in the related work section. In particular, a more detailed discussion of existing federated learning methods in multimodal learning is needed to better position the novelty and contributions of FedDISC.\n2. Under the fixed missing protocol, where one modality is completely absent from all samples, why the model still performs modality generation instead of simply relying on the available modalities?\n3. Why does FedDISC perform worse than SDR-GNN on IEMOCAP6 under the fixed missing setting when the textual modality is present?\n4. How do the DGN and SCN modules compare to previous methods that capture context, speaker dependencies, or cross-modal semantics?", "limitations": "The integration of federated learning with diffusion models increases computational complexity and may lead to higher training costs and latency.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\nFedDISC integrates federated learning with diffusion models to enable collaborative missing-modality recovery while preserving data privacy.\nDISC-Diffusion ensures context, speaker, and semantic consistency between recovered and existing modalities using dialogue-aware and semantic conditioning networks.\n\nWeaknesses:\nThe integration of federated learning with diffusion models increases computational complexity and may lead to higher training costs and latency.", "was_revised": true, "final_justification": "The reubttal of authors has addressed Questions 1, 3, and 4. Regarding Question 2, generating missing modalities from existing ones does not introduce new information, and the observed performance gain remains superficial. The authors still lack a deeper, mechanistic explanation for why generation improves results, so I keep the overall scores as it is.", "rebuttal": "**Question1:**\n\nWe sincerely thank the reviewer for their insightful feedback and valuable suggestion to enhance the clarity of our related work section. In response, we will revise Section 2.2 (Federated Learning, Pages 2-3) to provide a more comprehensive discussion of existing federated learning (FL) methods in multimodal learning, focusing on their approaches to handling missing modalities, their advantages, and limitations. The revised section will maintain the same word count as the original to ensure consistency with the manuscript’s structure. Below, we outline the planned content for the revised Section 2.2, which will better position the novelty and contributions of our proposed FedDISC framework.\n\n2.2 Incomplete Multimodal Federated Learning\nMultimodal federated learning(MFL) enables decentralized cross-modal learning while preserving data privacy; however, the inevitable presence of missing modalities in real-world scenarios poses significant challenges to its effectiveness.  Le et al. propose Multimodal Federated Cross Prototype Learning (MFCPL) \"Cross-modal prototype based multimodal federated learning under severely missing modality\" to addresses the issue of missing modalities through prototype learning, they introduce Cross-Modal Alignment (CMA) to align zero-padded features of missing modalities with existing modality features, reducing noise from zero-padding. However, this method can only deeply mine existing data at the prototype level, failing to fundamentally reconstruct missing information. To tackle this, Yin et. al. \"Self-attention fusion and adaptive continual updating for multimodal federated learning with heterogeneous data\" introduce Stable Diffusion into MFL to recover missing modalities. On image-text datasets with missing image modalities, clients upload text embeddings, and the server generates and extracts image modality features. However, this method deploys a global generative model on the server, unable to generate client-specific local feature image modalities, limiting its generalization. To address these issues, our FedDISC framework fine-tunes modality-specific feature recovery modules for each client’s local characteristics.\n\n**Question2:**\n\nWe sincerely thank the reviewer for their insightful question, which allows us to clarify the necessity of modality generation in FedDISC. Under the fixed missing protocol, where one modality is entirely absent, relying solely on available modalities fails to capture sufficient information for robust emotion recognition, leading to degraded performance. This is evident from baseline methods in Table 1 (Page 7), which show significantly lower accuracy and WAF1 scores when using only available modalities. In contrast, FedDISC achieves substantial improvements by generating missing modalities through client-specific diffusion models. In other words, the recovered modality can provide more comprehensive and enriched information.\n\n**Question3:**\n\nWe sincerely thank the reviewer for their meticulous observation. We attribute the performance gap compared to SDR-GNN on IEMOCAP6 under the fixed missing setting with the textual modality present to two key factors.\n\nFirst, the textual modality in IEMOCAP6 contains significantly richer semantic information than audio and visual modalities, diminishing the impact of modality recovery. As shown in Table 1 (Page 7), baselines achieve higher accuracy when the textual modality is available, as text inherently provides more easily extractable semantics. Consequently, the recovery of audio and visual modalities, which is FedDISC’s primary focus (Section 4, Page 4), becomes less critical for final performance when text is present, reducing our model’s relative advantage.\n\nSecond, as analyzed in Section 4.2 (Comparison with SOTA Methods, Page 6), FedDISC’s federated learning paradigm splits data across clients, with each client accessing only one-third of the data available to centralized models like SDR-GNN. This data limitation must be considered when comparing performance, as it inherently constrains FedDISC’s training capacity.\n\n\n**Question4:**\n\nWe sincerely thank the reviewer for their insightful question. Experimental results demonstrate that DGN and SCN outperform previous methods in capturing semantic information for emotion recognition. Detailed visualizations in Figure 3 (Page 8) and Appendix Figure 6 (Page 18) illustrate their effectiveness. For instance, Figure 3 shows the recvery capability of SGR-GNN and GCNet, which also use graph networks to model dialogue context. However, their modality recovery performance is significantly inferior to FedDISC, confirming DGN’s superior ability to capture context and speaker dependencies (Section 4, Page 4). Similarly, IMDer attempts to capture cross-modal semantics but achieves poorer modality recovery compared to FedDISC, which leverages cross-attention mechanisms to ensure semantic consistency across modalities (Page 5). These results validate SCN’s effectiveness in capturing cross-modal semantics."}, {"review_id": "fhaqk94TqB", "reviewer": "Reviewer_rCdQ", "summary": "1.Using a federated learning framework to address the problem of missing modalities also ensures data privacy.\n2. The Federated Dialogue-guided and Semantic-Consistent Diffusion (FedDISC) framework uses a diffusion model to ‌recover missing modality features.", "questions": "1. Add some comparative experiments with recent state-of-the-art methods, such as CIF-MMIN[1], MoMKE[2], CM-ARR[3] and Flex-MoE[4] to demonstrate the superiority of your methed.\n2.The proposed framework is quite complex; it should include more ablation studies to demonstrate the effectiveness of each module within it.\n3. A comparative experiment for the full-modality case should be included in the test set evaluation.", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1.This paper offers a fresh perspective on emotion recognition under incomplete modalities.\n2.The appendix includes extensive supplementary content to help readers better understand the material presented in the main text.\n3.Get a good results across multiple datasets.\n\nWeaknesses:\n1.Lack a comparison with state-of-the-art methods, e.g., CIF-MMIN[1], MoMKE[2], CM-ARR[3] and Flex-MoE[4].\n2.Information regarding the data used for pre-training DGN and SCN is missing.\n3.An ablation study comparing the contributions of DGN and SCN is missing.\n4.Given that federated learning is the main innovation, the paper does not explore the impact of the number of clients on performance.\n5.There is a significant performance disparity between the two diffusion models (DDPM and DDIM), but the paper lacks an in-depth investigation into why this occurs.\n6.How are the clients simulated in the experiments? Are they distinct machines, or is it a simulated distributed environment on the two L40S GPUs?", "was_revised": true, "final_justification": "some of the concerns have been addressed. raised the score to 3.", "rebuttal": "**Question1:**\n\nThank you for your valuable suggestions.\n\n1. On adding recent SOTA baselines:\n\nWe appreciate the reviewer’s suggestion to include comparisons with more recent state-of-the-art methods. In the main text, our comparative experiments already cover the 2021 method (MMIN), as well as recent models from 2023 (GCNet, IMDer, DiCMoR) and 2025 (SDR-GNN). Following your advice—and to maintain a concise table size—we will replaced the 2021 MMIN model with the 2024 CIF-MMIN model in Table 1 and Table 2.\n\nRegarding the four models mentioned by the reviewer:\n\nUnfortunately, as no references were provided, we were unable to identify the corresponding paper for MoMKE.\n\nThe CM-ARR model does not have open-source code, making replication infeasible in the short term.\n\nFlex-MoE is designed for Alzheimer's disease multimodal detection and is not applicable to multimodal emotion recognition benchmarks.\n\nTherefore, we conducted experiments with CIF-MMIN under exactly the same settings as our main experiments, and will included these results in the revised manuscript. The comparative results on the IEMOCAP4, IEMOCAP6, CMUMOSI, and CMUMOSEI datasets are summarized below:\n| dataset   | method      | l         | v         | a         | l,v       | l,a       | v,a       | l,v,a      |\n|-----------|-------------|-----------|-----------|-----------|-----------|-----------|-----------|------------|\n| iemocap4  | FedDISC(P)  | 68.0/68.2 | 63.3/61.7 | 62.6/62.1 | 75.8/75.8 | 78.0/78.1 | 68.7/68.0 | 78.6/78.4  |\n| iemocap4  | CIF-MMIN    | 58.0/59.3 | 53.3/51.3 | 56.3/58.4 | 73.2/74.0 | 74.3/75.6 | 65.7/66.9 | 78.3/78.5  |\n| iemocap6  | FedDISC(P)  | 56.8/56.5 | 56.0/55.5 | 60.6/59.9 | 57.4/57.6 | 59.4/59.3 | 66.4/66.3 | 64.3/64.7  |\n| iemocap6  | CIF-MMIN    | 54.7/54.3 | 39.7/35.2 | 56.3/58.4 | 51.3/52.1 | 53.8/50.1 | 56.5/56.9 | 61.3/60.2  |\n\n| dataset   | method      | 0         | 0.1       | 0.2       | 0.3       | 0.4       | 0.5       | 0.6       | 0.7       | average    |\n|-----------|-------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|------------|\n| CMUMOSI   | FedDISC(P)  | 86.3/86.3 | 85.9/85.8 | 83.3/83.2 | 81.1/81.0 | 79.7/79.6 | 80.2/80.2 | 77.5/77.5 | 75.7/75.8 | 81.2/81.2  |\n| CMUMOSI   | CIF-MMIN    | 84.0/83.6 | 82.5/84.1 | 82.6/82.7 | 80.5/80.9 | 78.4/78.1 | 77.3/77.2 | 74.1/74.2 | 69.8/71.3 | 78.8/78.9  |\n| CMUMOSEI  | FedDISC(P)  | 86.8/86.4 | 86.5/86.1 | 86.4/86.3 | 85.6/84.7 | 84.5/84.3 | 82.5/83.2 | 81.7/82.5 | 81.5/82.2 | 84.4/84.5  |\n| CMUMOSEI  | CIF-MMIN    | 85.8/86.2 | 85.4/85.5 | 85.0/85.3 | 83.1/83.8 | 82.7/82.5 | 80.4/81.1 | 78.5/79.2 | 77.3/77.4 | 82.3/82.6  |\n\nThese results further demonstrate the superiority and robustness of our method compared to strong recent baselines.\n\n2. On ablation studies:\n\nWe sincerely thank the reviewer for this constructive suggestion. However, we beg to differ on the point that additional ablation studies are required. We believe that our paper already provides comprehensive ablation experiments which sufficiently demonstrate the effectiveness of all key modules:\n\n1\\) DGN & SCN:\n\nAs shown in Figure 4 of the main text, we compare conditional modality recovery (with DGN and SCN) versus unconditional recovery. This clearly illustrates that DGN and SCN together enable the model to capture more effective semantic information and guide the recovery process. Furthermore, Table 6 and Figure 7 in the appendix provide detailed metric-based and visualization-based ablations, demonstrating that SCN extracts more effective semantic information than a simple speaker graph network.\n\n2\\) DISC-Diffusion:\n\nFigure 3 in the main text and Figure 6 in the appendix visualize the similarity between recovered and original modalities, showing that our DISC-Diffusion module achieves the closest alignment with the real data distribution.\n\n3\\) AFS Module:\n\nTable 3 in the main text quantitatively demonstrates the significant accuracy improvement contributed by the AFS module under various missing-modality scenarios. Moreover, Figure 8 in the appendix shows that AFS not only improves final performance but also accelerates convergence during training.\n\nIn summary, we respectfully submit that the ablation studies presented in our manuscript are sufficient to validate the necessity and contribution of each module in our framework.\n\n**Question2:**\n\nThank you for your valuable suggestion.\n\nWe would like to clarify that in Table 2, the results under “missing rate = 0” already correspond to the case where the test set is fully available with all modalities. For further clarity, we will also add the full-modality results to Table 1, as shown below:\n\n| dataset  | available | baseline  | FedDISC(P) | FedDISC(I) | GCNet     | SDR-GNN   | IMDer     | DiCMoR    | CIF-MMIN   |\n|----------|-----------|-----------|------------|------------|-----------|-----------|-----------|-----------|------------|\n| iemocap4 | l,v,a     | 78.6/78.4 | 78.6/78.4  | 78.6/78.4  | 78.4/78.3 | 78.5/78.1 | 78.1/78.3 | 78.2/78.3 | 78.3/78.5  |\n| iemocap6 | l,v,a     | 64.3/64.7 | 64.3/64.7  | 64.3/64.7  | 58.6/59.1 | 61.3/61.2 | 58.7/58.4 | 55.9/56.2 | 61.3/60.2  |\n\n\nThe reason we did not originally include full-modality results in Table 1 is that the primary focus of our work is on handling missing modalities, and when all modalities are available, there is no need for modality recovery. What's more, we note that in the recommended baseline in question 1: CIF-MMIN (“Contrastive Learning Based Modality-Invariant Feature Acquisition for Robust Multimodal Emotion Recognition With Missing Modalities”), the authors also do not report full-modality results in their fixed missing-modality experiments. This supports our initial design.\n\nIn summary, for completeness, we will add the full-modality case in Table 1 in our revised manuscript. Thank you again for your valuable suggestion.\n\n**Extension:**\n\nWe provide responses here to the issues raised by the reviewer in the \"Strengths and Weaknesses\" . Weakness 1 and Weakness 3 have already been addressed in our previous responses above.\n \n**Weakness2:** Information regarding the data used for pre-training DGN and SCN is missing. \n\nThank you for pointing out this question. To better simulate real-world scenarios, each client pre-trains its own DGN and SCN modules. The data partitioning for pre-training DGN and SCN is identical to that of the main network.\n\n**Weakness4:** Given that federated learning is the main innovation, the paper does not explore the impact of the number of clients on performance.\n\nThank you for this question. Regarding the impact of the number of clients, please refer to our response to Question 2 from Reviewer 12cs.\n\n**Weakness5:** There is a significant performance disparity between the two diffusion models (DDPM and DDIM), but the paper lacks an in-depth investigation into why this occurs. \n\nThank you for your question. The performance difference between the two diffusion models (DDPM and DDIM) is primarily because DDPM enables more accurate feature recovery compared to DDIM. As analyzed in Section 4.2 “Comparison with SOTA Methods” (Visualization experiments), the visualizations demonstrate that features recovered by DDIM deviate more from the original feature distribution, while features recovered by DDPM are much closer to the original distribution. This results in higher downstream accuracy when using DDPM for modality recovery.\n\n**Weakness6:** How are the clients simulated in the experiments? Are they distinct machines, or is it a simulated distributed environment on the two L40S GPUs?\n\nThank you again for your question. In our federated learning experiments, we follow the classic FedAvg setup: all clients are simulated on the same GPU. Uniquely, we assign the server operations to a separate GPU. For further implementation details, please refer to our upcoming open-sourced code."}]

Author rebuttal:

Meta-review: This paper proposes FedDISC, a novel framework for multimodal emotion recognition in conversations under incomplete modalities. The main contributions are: (1) the integration of federated learning with diffusion-based generative recovery to handle missing modalities across distributed clients; (2) the DISC-Diffusion module, which ensures semantic and contextual consistency in modality recovery using a dialogue graph network and semantic conditioning network; and (3) an alternating frozen strategy that facilitates stable optimization by decoupling the training of recovery and classification modules. The method is evaluated on standard MERC datasets, demonstrating consistent performance gains under various missing-modality settings.

The paper’s strengths lie in its practical motivation, innovative system-level integration, and thorough empirical validation. It addresses a high-impact, real-world problem—modality incompleteness in privacy-sensitive distributed settings—through a federated generative framework, a direction underexplored in prior work. The integration of DGN and SCN to ensure semantic consistency is well-motivated and effective. The experiments are comprehensive, covering both random and fixed missing-modality protocols, and include strong baseline comparisons, t-SNE visualizations, and statistical significance tests. The authors also demonstrate the communication efficiency and scalability of the approach, responding diligently to reviewer concerns.

The main weaknesses noted were the lack of novelty in individual components, some missing implementation details, and the need for clearer distinction from prior work. One reviewer raised concerns about missing ablations, full-modality baselines, and comparison to recent methods. These were convincingly addressed during the rebuttal, with new experiments (e.g., CIF-MMIN comparisons, full-modality results), detailed clarifications, and acknowledgments of limitations.

In conclusion, although some components build on known methods, the cohesive and well-executed integration, coupled with strong empirical performance and real-world relevance, make this a solid contribution to the multimodal and federated learning communities. Having given some consideration, I recommend acceptance, but this decision might need further discussion with SAC.
Decision: Accept (poster)