Learning Reconfigurable Representations for Multimodal Federated Learning with Missing Data
Authors: Manh Duong Nguyen
Trong Nghia Hoang
Thanh Trung Huynh
Quoc Viet Hung Nguyen
Phi Le Nguyen
Venue: NeurIPS 2025 poster ()
Keywords: Federated Learning
Modality Missing
Multimodal Learning
Primary area: general_machine_learning
TL;DR: Learning Reconfigurable Representations for Multimodal Federated Learning with Heterogenous Missing Data

Abstract: Multimodal federated learning in real-world settings often encounters incomplete and heterogeneous data across clients. This results in misaligned local feature representations that limit the effectiveness of model aggregation. Unlike prior work that assumes either differing modality sets without missing input features or a shared modality set with missing features across clients, we consider a more general and realistic setting where each client observes a different subset of modalities and might also have missing input features within each modality. To address the resulting misalignment in learned representations, we propose a new federated learning framework featuring locally adaptive representations based on learnable client-side embedding controls that encode each client’s data-missing patterns.

These embeddings serve as reconfiguration signals that align the globally aggregated representation with each client's local context, enabling more effective use of shared information. Furthermore, the embedding controls can be algorithmically aggregated across clients with similar data-missing patterns to enhance the robustness of reconfiguration signals in adapting the global representation. Empirical results on multiple federated multimodal benchmarks with diverse data-missing patterns across clients demonstrate the efficacy of the proposed method, achieving up to 36.45\% performance improvement under severe data incompleteness. The method is also supported by a theoretical analysis with an explicit performance bound that matches our empirical observations. Our source codes are provided at [https://github.com/nmduonggg/PEPSY](https://github.com/nmduonggg/PEPSY)

Learning Reconfigurable Representations for Multimodal Federated Learning with Missing Data
Duong M. Nguyen'
Trong Nghia Hoang Washington State University, US trongnghia hoangOusu.
University of Illinois Urbana- Urbana-Champatn Champaign, US umduongg@ill edu
Thanh Trung Huynh VinUniversity, Vietnam
Quoe Viet Hung Nguyen Griffin University, Australia
trung. htthvnnniiin.....mmmmm.im.ii.iii edu. vn
Soaayy.nuuen@griffii edu.
Phi Le Nguyen
Hanoi University of Science and Technology, Vietnam lemp@soict. hust.edu. vn
Abstract
Multimodal federated learning in real-world settings often encounters incomplete and heterogeneous data across clients. This results misaligned local feature representations that limit the effectiveness of model aggregation. Unlike prior work that assumes either differing modality sets without missing input features or a shared modality set with missing features across clients, consider more general and realistic setting where each client observes a different subset of modalities and might also have missing input features within each modality. To address the resulting misalignment learned representations, we propose new federated learning framework featuring locally adaptive representations based learnable client-side embedding controls that encode each client's 's data-missing patterns.
These embeddings serve reconfiguration signals that align the globally aggre- gated representation with each client's local context, enabling more effective use shared information. Furthermore, the embedding controls can be algorithmically aggregated across clients with similar data-missing patterns to enhance the robust- ness of reconfiguration signals in adapting the global representation. Empirical results on multiple federated multimodal benchmarks with diverse data-missing patterns across clients demonstrate the efficacy of the proposed method, achieving up to 36.45% performance improvement under severe data incompleteness. The method is also supported by theoretical analysis with an explicit performance bound that matches our empirical observations. Our source codes are provided at
Introduction
Due to the rapid advances loT technologies 4 23 and growing concerns over privacy pro- numerous emerging multimodal federated learning (MMFL) scenar- ios in which clients observe different subsets of input modalities and must collaborate to train common model without sharing data. These scenarios introduce two interrelated data-missing
tection [52]. there
events: (1) clients may have access to only subset feature modalities g 45 (e.g., one device collects audio while another collects physiological signals), and (2) inputs within each modality
Corresponding authors: Duong M. Nguyen, Trong Nghia Hoang, Phi Le Nguyen.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

may be partially missing due to sensor failures or intermittent recording [64| 39]. These chal-
lenges fundamentally disrupt the implicit assumption of traditional federated learning (FL) meth-
ods [38 22 21 66
67
32
51 31
40, 37 16 50
61
[58 41 which presume that all local models
are trained on a common set of feature modalities.
Challenge. When local models are optimized over different feature subsets, they tend to map inputs into incompatible representation spaces. Aggregating such models without proper alignment risks collapsing informative representations into entangled or degraded ones, ultimately reducing global performance. This problem is further exacerbated by heterogeneous data- sata-missing patterns across clients, both in terms of available modalities and partial input observations [42 56 (see Fig. [ These compounded patterss are common in real-world applications such as wearable health monitoring. distributed environmental sensing, and smart infrastructure, where data collection is increasingly decentralized and sensor failures occur more frequent. Effectively addressing both missing modalities and missing features is essential for enabling next-generation distributed computing infrastructures, where learning must operate over heterogeneous, fragmented, and privacy-preserved data sources.
Despite growing interest in multimodal and federated learning, most existing work focuses on idealized settings where all clients observe the same set of modalities. As a result, the general MMFL setting, where both events of data-missing occur, remains largely unaddressed. Existing approaches can be grouped into the following directions:
Limitation of Prior Work.
First, several efforts extend FL to multimodal inputs by designing universal representations [65] 60
43 46 but they assume all clients observe the same modalities, ignoring modality het- erogeneity. Second, centralized data imputation methods, including heuristic imputation [68] (69]
6
neural imputation [9]
57
18
12
19], deterministic reconstruction using available modali-
ties [63 7 47
36
20
44
39 and generative approaches [17] [25] 2 62
30 29], require access
ensure consistent imputation, and thus cannot be applied to fed- erated settings. Third, it is also possible to leverage pre-trained multimodal foundation models 1
to all data-missing patterns
59 33) to provide consistent data imputation, but many scientific domains such as healthcare, there is no FM that spans all feature modalities. Most recently, a few recent FL-specific works 88 45 64 (39 begin to investigate these data- sata-missing challenges in isolation. How- ever, when both modalities and input fcatures are missing, these methods fail to achieve satisfactory performance (see Section
(FMs) [12 19
48
Fundamental Gap. In hindsight, what remains missing in these approaches is a mechanism to capture and communicate how each client's local view of the data is shaped by its specific patterns of missing information. Since the server cannot observe the training data, it lacks the context needed to align or reconfigure representations for any particular client. Conversely, each client is only aware of its own data-missing context and cannot fully interpret or adapt the aggregated representation to its local setting. This reveals the need to learn a shareable data mata-missing profile for each client, which summarizes the characteristics of its local data-missing patterns, providing more specific instructions to reconfigure the shared model towards local data contexts.
Solution Vision. The above reasoning motivates the following key insight and hypothesis. It is possible learn and internalize specific traits in each client's data-missing profile into a set of embedding controls which can be used to reconfigure the shared model towards the local context. In this view, embeddings with similar content can also be aggregated which enables collaboration among clients with similar data-missing profiles. This design enables each client to adapt the shared model to its own incomplete data view, without requiring data sharing or retraining, providing a robust solution to multimodal federated learning with missing data.
Technical Contributions. To substantiate the above vision, have made the following contributions:
We develop a new multimodal federated learning framework (PEPSY) with a client-side design that encodes the characteristic traits of each client's feature modalities, data specifics, and data-missing patterns into set of local embedding controls. These local embeddings are communicated to the server, where they can be aligned and aggregated to capture commonalities across clients with similar data-missing profiles. The aggregated embeddings then serve as instructions to reconfigure the shared representation in manner that is adaptive to each client's local context (Section 2).
2. We develop a rigorous theoretical analysis which establishes direct bound on the expected perfor- mance of PEPSY over random patterns of missing data in terms of the training loss, demonstrating its stable performance and highlight the effectiveness of the proposed method (Section 3)
2

---

Fostig PEPSY
Prier Sokutionn
Missing
Biowd
ITAR
Towards
cleet
6000
Imputation Recerstrucion
view
Dota Missing Profie
Recorfiurc
20-
15 22
B2
complers
Minning Degree
Figure 1: From left to right: (a) Performance comparison showing that FedAvg degrades rapidly with increasing missing data, while framework PEPSY remains robust; (b) Illustration of types of data-missing events MMFL systems: (1) missing modalities and (2) missing input features; (c) Conceptual illustration highlighting the key distinction between our approach and prior work (see Section 2)
3. We evaluate the performance of our proposed framework against existing baselines through extensive experiments on the PTBXL [53] and SleepEDF [24] datasets. The results show that our method consistently outperforms existing baselines across numerous multi-modal data missing scenarios, establishing new SOTA performance in multimodal federated learning (Section
2 Multimodal Federated Learning (MMFL) with Missing Data
2.1 Problem Formulation and Method Overview
Standard Problem Formulation. In a MMFL system, there are K' clients, each with a local dataset DR consisting of [Dx] multimodal observations (Iz, F2). where z denotes the input instance and ya represents the corresponding label. Each instance œ, may miss some modalities, represented by missing set Sa M, where M the full set of modalities. The goal learn global model 0* by minimizing the following loss function:
K Σ la (0), with (o( & c(f(Du:)). k=1
0" argmin K
(I)
where f(Dg; 0) denotes multimodal prediction model with paramterer 0 over dataset Dj. and ([((D;; 0)) is an average loss of 0 over dataset Dj. Following 8 45 64 39 O can be decom- posed into two main modules: feature extractor 0, and post-processing head (including fusion and prediction) 9p. Accordingly, can expressed as f(Dx; 0) & fp (f ( O.); 0p), where fa(-) denotes the feature extractor and f-(-) represents the post-processing head.
Reconfigured Problem Formulation. As each client in MMFL only observes its own data-missing local view, the representations it produces potentially biased. Based this, introduce so-called data-missine profile with embedding controls, i.e., V/ ف { p= to reconfigure these biases into data-complete features. This results in (Du; 0, v as a reconfigured version of original prediction model,
f(Dx; 4) & fo (Dx: 0.) (Du; V):0,)
where denotes set concatenation, and r (Dp; v represents so-called relevance function that returns relevant embeddings for each Va E Dx. Intuitively, this relevance function captures missing pattern information needed to reconfigure instances in DR, which can be learned by rewriting Eq. | as:
K
- (x(0 ;
0-, TT" argmin 0.4
(3)
K
where, (0. v) & C(f(De: O, v)) and (0, v) ())
(4)
where R estimates relevance between each instance Dj and its embeddings selected by r(-). Intuitively, minimizing & (0, V leads to neural components 0 that extract data-missing features, which are reconfigured by for predictions. Conversely, maximizing (0, v enables Vi to adapt to local context, effectively distilling missing patterns.
Method Overview. An overview of our proposal is Fig. 2a Formally, PEPSY operates over multiple communication rounds, each consisting of client-side training and server-side aggregation.
3

---

Prediction
Modality Fusion
Dri-WWrsha Frofiat
KedlogAppepiinn
Probobiissti Synohronization
mis
Lne
OlL
Sec.
Server
2.2.3
selenctdd
Query
MLP
Sec.
mod di
Training
Tiaviing
Toering
Data-Missing Profile
USA @
2.2.1
Average
Concate
Shared Encoder
Vp.P 1..
Ad1
(a) Overall Workflow. PEPSY has two stages: (b) Client Design. Each client extracts modality- and client training and server aggregation. After local data-specific features (w w " then 2 queries the data- training, client parameters perform aggregation, which includes FedAvg 38 and probabilistic synchronization.
the
missing profile V to form =i as the missing-pattem fea- ture. Finally, =h reconfigures (re mod =) into data-
complete features for downstream tasks.
Figure 2: Overview of the overall server-client workflow of PEPSY and its client design.
On the client side, each client extracts information from its local dataset, potentially with some modalities missing, and leverages the extracted information to select relevant embeddings from I for each instance Ed thereby 3 constructing data-complete representations. Further details provided in Sections 2.2.1 and 2.2.2 respectively. To ensure final representations are faithfully data-complete, we enforce these features to be comparable with full-modality features before fusion and prediction (see Section 2.2.3p. On the server side, due to variable size of data-missing profile non non-parametric clustering problem, as This process repeats for T rounds until convergence.
per client, treat the data-missing profile aggregation
presented in Section 2.3
2.2 Client Design
This section explains how clients learn the data-missing profile and use to reconfigure biases caused by limited local data views. An overview of the client design is shown in Fig. 2b
2.2.1 Data-Missing Representations
Intuitions. In the presence of missing modalities, the information within multimodal instance can be decomposed into three components: modality-speciffc (distinguishing different modalities); data-specific (capturing the integrity of the individual instance); and missing-pattern information (distinguishing different missing patterns). Based on this decomposition, we extract these components construct a comprehensive data-missing profile for each client.
Formally, given an instance Id = {x Vi E M Sd} we first construct (1) modality-specificc features ds } and (2) data-specifie features { The former represented by learnable i.e.,
embeddings W/mod_ = {woddd) |M to ensure data invariance and are shared across all instances,
unme mnmood (Vd). The latter are constructed by mapping and normalizing each observed modality aa (Vi M Sa) to corresponding representations, denoted h For missing modalities, a common averaging approach [54] 28 to reconstruct their features, resulting in the formulation:
w
ds
A I(i & Sa)ha + 1 Sa)
M
SA
Σ h jdS,
(5)
where I depicts an indicator function. To ensure the feature reconstruction Eq. 5 are truly data- specific, introduce data-spe loss that regularizes the features from the same instance's 's
Other neural components be aggregated effectively using FedAvg 38

---

available modalities to be closer than those from different instances:
exp(hash.)
(dd Sa)
-
log
(6)
d2 En, &Se, AzES,, exp(ham hark2)
where his represents the & fz-normalized feature of hds. Intuitively, minimizing Las ensures has preserves instance identity across modalities while reducing the impact of missing patterns Sa. thereby improving prediction consistency and stability. This justified by the theorem in Section
Remark. While wat encodes modality-speciffe information and with captures data-specific details influenced by the missing pattern Sa (Eq. [5), together they comprehensively represent the data- missing information æ This combined information can distilled into the data-missing profile, allowing future clients leverage similar data views to handle their local context.
2.2.2 Embedding Controls Selection
Intuition. Since data-missing features reflect the client's ' local missing patterns, learning data-missin profiles requires interaction between these features and embedding controls. We model this interaction as query-key matching process that selects the most relevant embeddings for each instance to distill and reconfigure, formulating the final data-complete features. Details are below.
Given data- Lata-missing representations (w"T unins) E M, from a' we allow select the relevant embeddings from V for reconfiguration. The relevance between each modality æai and particular embedding control v (p = .....),, T), denoted as "(x,, up,). is defined as follows:
(x p))) - e ((q((((( (q(x))).. k( k))),,))
where e(:,-) depicts the cosine similarity, q( A MLF [wo w fully captures data-missing information from z Here k(-) an identity function to distill the original information directly from dd to 2b g. allowing accurate reconfiguration from 2b without distortion. To prevent the model from distributing fata-missing information in æy too broadly and diluting learned data-missing profile, we only allow k relevant embedding controls selected for each instance, with << | To enforce this, we introduce regularization term:
M
A
Σ
Σ (x v),
DEVa
where Vas is the set of the most relevant embeddings for each modality was within the client's local suta-missing profile. This regularizer encourages each instance to focus on a small, relevant subset of embedding controls, promoting more precise relevance assessment and better distillation. We use the averaged embedding to represent the whole selected set Las resulting in missing-pattern representation mim The final representation is then formed was [w wnl mis)
2.2.3 Reconfiguration Regularization and Modality Fusion
Reconfiguration Regularization. By leveraging the missing profile, we form the final representation Wai by concatenating three types of information wird w and w To ensure the final represen- tation faithfully reflects the full-modality information of instance Ey. we introduce a contrastive loss L. reconfiguration signal. This loss encourages the projected representations w of wai from the same instance Ta to be close (similar to Eq. 6 Intuitively, this regularization guides the data-missing embeddings to reshape representations into data-complete forms, hence ensuring effective reconfiguration signals. Note that w.i Vi E M, are used solely for regularization.
Modality Fusion. Since w.s provides high-level representation of the original feature, we leverage the similarity among { (ura:: as attention weights to fuse { (urae) } together and form a so-called cross- modal representation {Ča: Finally, combine the cross-modal representation č.s and the original representation wai to obtain the final representation Cas of instance Ez. Ca = Qu + (1 caai) Wa. where Oui is computed by learnable function s( ((wa. Cail), with denoting element- wise concatenation. The resulting representation Cai is then passed to the prediction head, ensuring that it captures both the completeness of the data and enriched cross- modal contextual information.
MLP denotes linear projector
5

---

Training Objective. After producing final prediction using prediction head, the client model is evaluated by task-specific loss function Lrask- Overall, the training objective for the local model C & Ltask + X(Lds + Fre) ryR, where À and 1j are weighting coefficients that control the contributions of Lds frre and the relevance R, respectively.
2.3 Server Aggregation
While traditional server aggregation algorithms [8 aggregate common neural components among clients, it struggles with our data-missing profiles due to alignment issues. Local data- missing profiles are learned in arbitrary orders across clients, leading to misalignment where identical emhedding positions may represent different data-missing patterns. Consequently, directly merging these representations can produce suboptimal results. To overcome this, we frame data-missing profile alignment as a clustering task that groups embeddings from diverse client views into a global profile, Since each client may select different number of embeddings within its data-missing profile bb, this becomes non non-parametric clustering problem [58] 34 27]. This study adopts PFPT [58] as the profile aggregation method, enabling the number of clusters adapt dynamically to data complexity, or missingness level in our context. Each client refines the global profile using its private data, producing locally augmented controllers whose size and complexity reflect the client's missingness level. Using PFPT"s aan-aaametric nature, the server clusters similar controllers and updates the global profile to reflect the missingness complexity of the whole system, which is then shared with clients for the next training round. This process allows PEPSY to align missingness profies across clients and effectively handle heterogeneous data-missing patterns (see Section for details).
3 Theoretical Analysis
Ideally, we expect the model's predictions to remain robust even in the absence of certain modalities. In this section, we present a theoretical analysis of the convergence behavior of our model's output for given instance x under two conditions: when all modalities are available and when some are missing. Specifically, we demonstrate that our training objectives are designed to minimize the discrepancy between these two prediction outcomes.
Theorem 3.1 Let x. E D be an arbitrary instance with a missing modality pattern C. MM, where M denotes the full set of modalities. Suppose o and y represent the model's outputs test time when x is missing modalities in S, and when all modalities are present, respectively. Let Ez,s denote the expectation over all instances x and all possible missing patterns S. Then, if the client model is -- ipschitz continuous, the distance between yo and s
can be bounded by the empirical training
loss as follows:
Ex,s (Ca,s(Ca(( S)] E Ey,sllv2 -v-ll S S0 (ISII
M
log (IM)
(IM) - 181)
|
Observation 1. Theoretical analysis shows that the expected deviation caused by missing modality patterns S controlled by our proposed loss Las, which is directly minimized during training. Reducing Las lowers the model's 's dependency on missing data, tightening the theoretical error bound and ensuring stable, reliable predictions despite incomplete inputs. Thus, our loss design both mitigates the impact of missing modalities and improves generalization across diverse test conditions.
Observation 2. In the ideal case where the solution is optimal, i.e., s (Ir, 5)] = O, the right-hand side of Eq. 9 simplifies to (HIS) Vlog M - log(M - {s)))). When - 0, i.e., all modalities are available, both sides of the bound converge zero expected. In the worst-case scenario, where |S| = MM| the right-hand side becomes (({[IM)-)) 11 1) log [MI). depending only on the Lipschitz constant μ.. This aligns with the intuition that Las(.)) -) minimizes modality diserepancies within a shared embedding space but does not constrain the model's global behavior, leaving the remaining deviation governed by the smoothness of the learned function, as reflected in μ.
Overall, the stability of PEPSY to varying missing patterns depends on three factors: the alignment quality of data-specific features (Lax(,, -)), the number of missing modalities ((S), ), and the smoothness of the learned model, characterized by the Lipschitz constant μ.. Theorem 3. supports PEPSY's effectiveness in federated learning, which matches our empirical observation.

---

Table 1: Performance of baselines on the PTBXI. and EDF datasets under various missing patterns in train and sets, for both ITD and Non-IID scenarios. The hest and second-hest results are highlighted in bold red and blue, respectively.
Mathod
HDD
Now-II
0,4
PedPrec
71 61
11:3-11
71 3:90
1843
51.1 1/6
51:1++5.0
9000 13:15
1.51
14:77-11
0.2
Fedta79
22.36
60.55:3..00
99.27
99 37-35
51.99421555
257552270
4
40.5 11.20
10-15-1000 1006
10.70
PTBXL
FSPOY
7 BU8
4.25
75.4
69:20 2
M6.5 -
6
66.55.535
KIPL [51
00:00-111 14
65 16 2.13
55 61
1041 + 16.99
15:9-650
6136 4176--35 3.97
13
63:10-277
10366-1 16
46:11 17.45
06 18.99
0.95 13.3
16.00
Pedtao 57
14.20 00.4
13.06 043
41.35 6664
05 1041
51.03
20.5
45390
1..29
15.06 00.7
9.5
45:07 29.36
230
PedPrad 13
66.08 0.
43.5 5 D43
15:59-05
15.5 12:33
54.0 11.05
11.58 13.00
11:1 0.43
66.0--030 30
10.35 11.00
1437-604
121% [4
44.27 006
43.15 2033
41.2
41.3421121
2.35
22.34 13.75
1133
P
41.91
3:3
= 44
63.9-
4377 140
43 1
18.85
42
1555-11 15
1666-1155
(3-0-111
Padeee 12
19:0
17:45
00.43 10
7
13.07
4500
1:39
16.00-12:11
12:31
35
13.96-14 1800
1100
PSPaY
08.76
5.44
49.37
-8.27
46.77 2,46
54.84 3.22
50.25
4.11
54.00
SUJ
55.15 6.15
PedPesx
51.33
14.10
43.7 20:4
24.34 1411
13.13
36.15
41.77
(53
12:00
+
45.01
0.99
1.0
0.3
Peditp1it
PedE [4]
41:44:3.11
48 03
32.99 22
40.30 4012
42:11:142
in 1 14
36 8.221
43 0005
1027-001
450241145
40 0:77
80:1 0.77
42:78
à 1.44
49 1050-035 57
4137 134
40:31 +014
(n 00
1:300
FSP37
4832 8.38
0.33
52.78
4.1
41.52
58.96 1.99
66887
4 Empirical Evaluation
4.1 Experimental Settings
Dataset and Missing Modality Simulation. Our approach is evaluated on two datasets: PTBXL (53
(12 modalities) and Sleep-EDF (24) (5 modalities). Each dataset is split into 80% for training and 20% for testing, with the former distributed across K clients in both IID and Non-IID settings. Following 33]. we define ps the ratio of samples with missing modalities, and pm as the ratio of missing modalities within those s The missing degree is then defined as pm X Ps, representing the simulate modality
overall proportion of instances with missing modalities. Using these definitions,
missing patterns by constructing binary matrix (DD)), where ((Dx)) [6.ml] E {0, 1) indicates whether modality m is missing (0) or available (1) for sample i. The incomplete dataset Dj D2 @(D)), where @ denotes element-wist wise multiplication, is then used for the experiments. Details for modality missing patterns simulation is presented in Appendix B
Baselines and Evaluation Metries. We compare PEPSY with five baselines: FedProx [31]. FedMSplit [8]. MIFL [45], FedInMM [64], and FedMAC (39. FedProx disregards missing modali- ties, FedMSplit and MIFL focus on modality-missing event, while FedInMM and FedMAC address feature-missing events. These baselines provide comprehensive benchmark for evaluating our method. We use accuracy on the server's dataset as a performance metric for the whole system. Implementation details are provided in Appendix C
4.2 Performance under Similar Missing Statistics between Training and Testing
Results under the IID setting. Table shows that PEPSY consistently outperforms other methods in most experimental scenarios with varying missing statisties in IID settings. For the PTBXL dataset, when the missing degree is low (e.g.. Pm. 0.2). the differences achieving similar accuracy. However, as the missing degree increases (c.g., Pm = 0.8), PEPSY maintains significant advantage. outperforming other methods. This trend is even more pronounced in the EDF dataset, where PEPSY surpasses the basclines by up to 11.67% in all missing scenarios. While most methods experience substantial performance drops, PEPSY remains robust, achieving the highest accuracy in 40/40 cases. This is because the data-missing profile provides an informative reconfiguration signal that reprograms feature construction for more robust predictions.
minimal, with all methods
Results under the NIn-IID setting. In the complex Non-IID setting, PEPSY again outperforms all other methods, shown in Table I On the PTBXL dataset, PEPSY surpasses FedMAC and other approaches by nearly 15.83% in slightly missing scenarios (pm- = 0.2), maintaining its advantage even as missing patterns become more extreme, with 64.69% accuracy Pm/p. 0.8/0.8. On
A tuple (pm, ps) is called missing statistic.
7

---

Table 2: Performance of baselines under various missing statistics, where the missing statistics of the clients and server are different.
16
Ove
Method
Toding missing stalislies (pwr/p.)
0.200.2
0,4/3.4
0.60/.
0,808
1,00.4
0,611
0.8/1
FedPProx 11
20.24%
57.75%
38 18.84%
$4.58%
66 41%-
79,89%
25.85%
00000
N171. E8S
75.70%
73.37%
72: 185
335
73.66%
63.05%
46.15%
FecToNe 2-
77.185%
73.00%
68.98%
55. 86%
72.63%
51.70%
58.97%
Training missing staiiilie Utw)
FwcHSp1ax
70,249
57.76%
38.84%
54.68%
66.65%
20,89%
25,85%
Alinmment Weight
FedMAc [0)
79075
79.45%
77.307
73.39%
77.30%
74.022
63,685
PEPSY
79075%
79.19
79.57%
77.55%
78.31%
77.93%
76.76%
Figure 3: Impact of alignment loss on
Feditroz
11
77,05%
75.669
74 0255
66 84%
74 40%
60.61%
54.15%
performance deviation.
1,505
M17L 835
FecInNei 2-
73.77%
44.77%
44.39
74.02%
72 72.3853
42. 1258
42. 25%
58%
73 73.90%
44.01%
70,625
35,055%
31,52%
62.55%
Table 3: Ablation studies on different aggre- gation methods.
FeckSp1ix
77.05%
75.66%
74.02%
66.84%
74. 14.40%
69.01%
59,14%
FedMAo [0
73.91%
76.55
76.04%
72.51%
75.79%
73.01%
00.951
PEPBY
77.68%
75.66'
77.18%
75.91%
75.91%
74.40%
74.15%
0,50,
M17L [:5
FedMAD 19
FedInees 2-
FedHSplit
FeePProx
81
1
53.47%
38.71%
36.57%
36.57%
59.27%
13.42%
33.42%
35.81%
19.02%
50.06%
31.15%
45. 45.025%
60.40%%
31.15%
31.90%
34,13%
39.34%
34.93%
54,43%
29.77%
59 19.02%
$4.36/5
36.19%
36.19%
41.11%
27.49%
53.85%
27.49%
27.49%
44.52%
26.61%
26.61%
36.70%
44,5415
28.75%
pate
Pecken
Syudeditos
FEPST
Fedtrox
Motaal
53.02%
09.59%
71.34%
Y1.22%
72.64%
69 29.459
61.27%
64.19%
69.11%
65.44%
68.15%
71.62%
71,38%
59.77%
65.10%
51.0%%
0.5
71.15
12.29%
62.55%
56.75%
PEPBY
61.41%
62.17%
60.91%
61.29%
61.67%
59.52%
58.76%
Fvcken
Federox
08.50%
64522
53,448
58.15%
59.20%
41.44G
50.57%
58.516
All experimental results reported in Tab and Tab Hare conducted under the IID setting. The best and second best results are highlighted in bold red and blue, respectively.
10.5166
60.23%
65.47%
68.95%
55.76%
1.0
FadDrox
SysPesPosx
16.00%
03.22%
015225
57.35%
66.31%
54.27%
50574
the EDF dataset, PEPSY similarly outperforms FedMAC by significant gap and retains its lead in challenging scenarios. Across both datasets, PEPSY consistently maintains superior performance as the degree of missingness increases, highlighting its robustness to data heterogeneity and diverse missing patterns in federated contexts.
4.3 Performance under Different Missing Statistics between Training and Testing
We evaluated the effectiveness of our proposal by conducting more experiments with varying missing statisties between clients and servers in the IID setting. Table shows that PEPSY outperforms other methods across different missing statistics. When clients have missing data (pm. P.- = 0.0/0.0), PEPSY achieves the highest accuracy in most testing missing scenarios, surpassing other baselines by an average of 3.45%. This trend continues with high client missing rates (pm/p, 1.0/0.5), demonstrating robustness to extreme missing patterns. In the challenging inter-client missing scenario (Pm/P= 0.5/1.0) PEPSY outperforms competitors by up to 14,, highlighting PEPSY's ability to maintain consistent performance across diverse and complex client-server missing patterns.
4.4 Ablation Studies
Impact of Server Aggregation Algorithms. We conduct ablation study on server aggregation methods to assess the effectiveness of probabilistic alignment (denoted as Syn) in our proposed framework (see Tab.B), Denoting probabilistic synchronization as Syn, we compare FedAvg (38), FedProx 31,, and their probabilistic alignment variants SynFedAvg (which is used PEPSY) and SynFedProx. The results show that combining FedAvg and Syn significantly improves both performance and robustmess in PEPSY, surpassing others and persists at higher missing rates. This is because the probabilistic synchronization mitigates inconsistent modality patterns, while skewed data distributions have less impact in this setting, then can be handled by FedAvg. These results highlight the effectiveness of server aggregation of PEPSY across diverse missing patterns.
Impact of Alignment Loss. Fig. 3 jillustrates the effect of alignment loss on PEPSY's performance by varying the alignment weight. The model is trained in full-modality scenario (pm/PP = 0.0/0.0) and tested on both full-modality (pm/pa - = 0.0/0.0) and extreme-missing scenarios (pm/ Pp = =0.8/1.01 1.0). We assess the performance gap between these scenarios to evaluate the impact of alignment loss. As expected, increasing the alignment weight reduces the performance gap in both IID and Non-IID settings, demonstrating the contrastive regularizer's effectiveness in instance-ave aware alignment and improving model robustness. Importantly, these results support the theoretical bound outlined in 3
Impact of Data-Missing Profile. To demonstrate the effectiveness of our proposed data-missing profile in handling fata-missing events, we compare PEPSY with variant, PEPSY-NP (No Profile), where the data-missing profile is excluded, across various missing statistics. As shown in Fig. 5a
8

---

08/0.2
08/0.4 4
08/0.6
08/0.8
FedProx
FedMAo
PEPSY
Modality
Modallity 2
Modality 3
Modality 4
Modality 5
Figure 4: Modality representations of different methods under multiple missing scenarios. We train and provide I-SNE 2D visualizations of modality representations constructed by three methods, including our proposal, in different pm. /Pp settings. All experiments are conducted on EDF dataset, nonlID setting.
Pe/P,
Pa/P,
Pw/P,
2.80.
3628
CUNA
PaiPs
(a) Impact of control pool (b) Visualization of global control embeddings after 500 training iterations under on proposal's performance different missing scenarios. The reduced distance between consecutive iterations under different missing indicates convergence, while the variation shows that embeddings capture differ- ent aspects from each client.
scenarios.
Figure 5: Ablation studies our proposed data-missing profile.
incorporating missing profile consistently enhances PEPSY's performance in all test cases, with significant gains as the number of modalities missed increase. This is because more missing modalities causes greater variation lata-missing patterns across clients, making the shared data-missing profile essential reconfigure those variability.
Data-Missing Profile Diversity and Convergence. To analyze the behavior of the learned data- missing profile, we visualize the 2D t-SNE embeddings of global profile for the PTBXL dataset over 500 communication rounds under different missing settings (see Fig. (bb). The centroids of the embeddings, computed every 25 iterations, are marked by stars, with their update trajectory shown by dashed spline curve. As training progresses, the distance between successive centroids decreases, indicating convergence. Additionally, the spread of the embeddings gradually expands relative to their centroid, reflecting their adaptation to the diverse missing patterns across clients, suggesting that these embeddings are effectively optimized to handle varying client's 's local context.

---

Modality Alignment Analysis. Fig. compares modality alignment among our proposed PEPSY and two baselines, FedProx and FedMAC, representing standard FL method and the next-best performer in most experiments. Both FedProx and FedMAC fail to align modalities, reflecting their dependence on specific data-missing patterns FedProx lacks an alignment mechanism, while FedMAC discards modality-specin profile, reduces sensitivity to missing patterns and achieves clear modality alignment after training. More experimental results can be found in Appendix F
cues. In contrast, PEPSY, guided by shareable data-missing
5 Related Works
Multimodal Learning and Missing Modalities. Multimodal learning has gained attention for its potential to improve knowledge in centralized settings, particularly in the medical domain, where combining modalities is crucial for diagnostic accuracy [4] 23
13 (55 However, most methods
assume full modality availability, which is often not the case in real-world scenarios with missing
modalities. To address this, several approaches have been proposed: Zhang et al. [68] and Zhou et
al. [69 use heuristic and statistical imputation, while neural imputation methods (9
57
18
14
12
learn imputation models before inference. Pretrained foundation models [12 19
48
59 33
10
can be leveraged transfer knowledge imputation embeddings. and generative techniques such as VAEs, GANs, or diffusion models [17] [25 9 2 [8] 62 14 30 29 can build new imputation
models. However, both approaches have clear limitations: the first requires large public datasets, often unavailable in sensitive domains like healthcare, while the second requires full-modality data at
44 (44 rely on available modalities to extract or reconstruct missing representations by decomposing each modality into modality- and data-specific features.
the outset. Other works 663 47 36 20,
Multimodal Federated Learning. Driven by growing concerns over data privacy, security and transfer ineffectiveness, federated learning (FL) [38] collaborative learning paradigm is introduced to allow multiple devices to train a shared model while keeping their local data private. This approach preserves privacy and reduces data transfer overhead [32 51 31 15 I1 44) 26, 35 61] have. however, mostly focused on uni-modal data (e.g., immge or text) while the rapid advancement of mobile phones has increasingly led to the collection of multimodal
and Internet of Things (IoT) devices 4 23
datasets. Therefore, prior works 23
56 have extensively explored multimodal federated learning (MMFL), ranging from modality fusion to feature construction to enable richer and more comprehensive representations, which in turn enhances model performance and robustness. This new multimodal data paradígm has motivated growing body of research on MMFL.
Tackling Missing in Multimodal Federated Learning. A key challenge in MMFL is inconsistent learning progress across clients due to heterogeneous modality combinations, arising from modality missing (inter-elient missing) and input feature missing (intra-client missing) 43 39]. Modality missing occurs when clients have different modality combinations, each dataset remaining com-
plete 42 45 56), while input feature missing reflects the absence of specific modalities within an 39 Initial efforts 88
individual client's dataset, mimicking real-world scenarios 43
451 focused on
modality missing, and recent approaches such as FedInMM 64 and FedMAC 39 have addressed input feature missing but failed when both data-missing events occur, limiting their applications. This highlights the need for solutions that effectively manage these data-missing events in multimodal federated learning, ensuring stable and robust solution under different levels of heterogeneity.
6 Conclusion
This paper presents a novel solution to the challenge of missing modalities in multimodal federated learning. We propose PEPSY, a method that captures each client's local data-missing view in data-missing profile. This profile is then used to reconfigure data-missing biased representations to be faithfully data-complete. On the server side, these profiles are aggregated probabilistically into a global data-missing profile for the entire system, allowing collaboration among clients with similar data views. Theoretical analysis confirms PEPSY's stability across diverse missing modality scenarios, while empirical results demonstrate that it outperforms existing methods by up to 36.45% % in addressing missing modalities in heterogeneous settings. PEPSY thus offers a flexible and stable solution for complex federated systems, with strong potential for real-world applications.
10

---

Acknowledgement
This work financially supported by VinUniversity under Grant No VUNI2112.SGG04 This work utilized GPU compute resource at SDSC and ACES through allocation CIS230391 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services and Support (ACCESS) program [3] which is supported by U.S. National Science Foundation grants #2138259, #2138286, #2138307, #2137603, and #2138296. We would like to thank the Thomas and Margaret Huang Endowed Professor in Signal Processing and Data Science at the University of Illinois Urbana- Champaign, US and fellowship granted by VinUni- -ilinislooois Smart Health Center, VinUniversity, Vietnam for supporting the authors' conference travel.
References
[1] Abdul Fatir Ansari, Lorenzo Stella, Ali Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Scbastian Pineda Arango, Shubham Kapoor, Jasper Zschiegner, Danielle C. Maddix, Hao Wang, Michael W. Mahoney, Kari Torkkola, Andrew Gordon Wilson, Michael Bohlke-Schheiieee Schneider, and Bernic Wang. Chronos: Learning the language of time series. Transactions on Machine Learning Research, 2024.
[2] Bruno Aristimunha, Raphael Yokoingawa de Camargo, Sylvain Chevallier, Oeslle Lucena, Adam Thomas, M. Jorge Cardoso, Walter Lopez Pinaya, and Jessica Daftlon. Symthetic sleep EEG signal generation using latent diffusion models. In Deep Generative Models for Health Workshop NeurIPS 2023, 2023.
[3] Timothy J. Boerner, Stephen Deems, Thomas R. Furlani, Shelley L. Knuth, and John Towns. Access: Advancing innovation: Nsf's advanced cyberinfrastructure coordination ecosystem: Services support. In Practice and Experience in Advanced Research Computing 2023: Computing for the Common Good, PEARC 23, page 176.176, New York, NY, USA, 2023, Association for Computing Machinery.
[4] Alberto Brunete, Ernesto Gamhao, Miguel Hernando, and Raquel Cedazo. Smart assistive architecture for the integration of iot devices, robotic systems, and multimodal interfaces in healthcare environments. Sensors, (1(6)221 2021.
[5] Defu Cao, Furong Jia, Sercan Ö. Arik, Tomas Pfister, Yixiang Zheng, Wen Ye, and Yan Liu. Tempo: Prompt-based generative pre-trained transformer for time series forecasting. In ICLR, 2024.
[6] Haokun Chen, Yao Zhang, Denis Krompass, Jindong Gu, and Volker Tresp. Feddat: An approach for foundation model finetuning multi-modal heterogeneous Federated Learning. In Proceedings of the 2024 AAAI Conference on Artificial Intelligence, volume 38, pages 1 1285--1293, 2024.
[7] Jiayi Chen and Aidong Zhang. Hgmf:: heterogeneous graph-based fusion multimodal data with incompleteness. In Proceedings of the 26th ACM International Conference on Knowledge Discovery & Data Mining, pages 1295-1305, 2020.
[8] Jiayi Chen and Aidong Zhang. FedMSplit: Correlation-adaptive federated multi-task learning across multimodal split networks. In Proceedings of the 28th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pages 37-96,22 2022.
[9] Mengxi Chen, Fei Zhang, Zihua Zhao, Jiangehao Yao, Ya Zhang, and Yanfeng Wang. Probabilistic conformal distillation for enhancing missing modality robustness. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet. J. Tomezak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 36218-36242. Curran Associates, Inc., 2024.
[10] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patter Recognition (CVPR), pages 2818 182889,,, June 2023.
[11] Yac Jee Cho, Andre Manoel, Gauri Joshi, Robert Sim, and Dimitrios Dimitriadis, Heterogeneous ensemble knowledge transfer for training large models in Federated Learning. Computing Research Repository arXiv Preprints, 2204. 12703, 2022.
[12] Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. A decoder-only foundation model for time-series forecasting. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 10148 10167. PMLR, 21-27 Jul 2024.
11

---

[13] Qi Dou, Quande Liu, Pheng Ann Heng, and Ben Glocker. Unpaired multi- segmentation via knowledge distillation, 2020.
[14] MohammadReza EskandariNasab, Shah Muhammad Hamdi, and Soukaina Filali Boubrahimi. Chronogan:
Supervised and embedded generative adversarial networks for time series generation. In 2024 International Conference on Machine Learning and Applications (ICMLA), pages 567-574, 2024.
[15] Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning: A meta-learning approach. Computing Research Repository arXiv Preprints, 2002.07948, 2020.
[16] Ziwei Fan, Hao Ding, Anoop Deoras, and Trong Nghia Hoang. Personalized federated domain adaptation for item-to-item recommendation. Robin J. Evans and Πlya Shpitser, editors, Proceedings of the Thirty- Ninth Conference on Uncertainty in Artificial Intelligence, volume 216 of Proceedings of Machine Learning Research, pages 560-570. PMLR, 31 Jul-04 Aug 2023.
[17] Vincent Fortuin, Dmitry Baranchuk, Gunnar Raetsch, and Stephan Mandt. Gp- Deep probabilistic time series imputation. In Silvia Chiappa and Roberto Calandra. editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings Machine Learning Research, pages 1651-1666 PMLR, 26-28 Aug 2020.
[18] Asadullah Hill Galib, Pang-Ning Tan, and Lifeng Luo. FIDE: Frequency-intten intlated conditional diffusion model for extreme-awar time series generation. In The eiirtt Annual Conference on Neural Information Processing Systems, 2024.
[19] Mononito Goswami, Konrad Szafer, Arjun Choudhry, Yifu Cai. Shuo Li, and Artur Dubrawski. MOMENT: A family of open time-serics foundation models. In Proceedings of the 41st Interaational Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 16152. PMLR, 21-27 27 Jul 2024.
[20] Lisa Anne Hendricks, John Mellor, Rosalia Schneider. Jean- Baptiste Alayrac, and Aida Nematzadeh. Decoupling the role of data, attention, and losses in multimodal transformers. Transactions of the Association for Computational Linguistics, 9:570-585, 2021.
[21] Minh Hoang, Nghia Hoang. Bryan Kian Hsiang Low, and Carleton Kingsford. Collective model fusion for multiple black-box experts. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2742-2750. PMLR, 09-15 Jun 2019.
[22] Trong Nghia Hoang, Quang Minh Hoang, Kian Hsiang Low, and Jonathan P. How, Collective online learning of Gaussian processes in massive multi-agent systems. In Proceedings of theAAAl Conference on Artificial Intelligence, pages 7850-7857, 2019.
[23] Parminder Kaur, Husanbir Singh Pannu, and Avleen Kaur Malhi, Comparative analysis on cross-modal information retrieval: A review. Computer Science Review, 30:300 2021.
[24] B. Kemp, A.H. Zwinderman, B. Tuk, H.A.C. Kamphuisen, and J.J.L.. Oberye. Analysis of a sleep-dependent neuronal feedback loop: the slow-wave microcontinuity of the eeg. IEFF Transactions Biomedical Fingineering, 47(9): 185-1 1194,199,,00 2000.
[25] Gavin Kerrigan, Justin Ley, and Padhraic Smyth. Diffusion generative models in infinite dimensions. In Francisco Ruiz, Jennifer Dy, and Jan- aul-eilem van de Meent, editors, Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206 Proceedings of Machine Learning Research, pages 9538-9563. PMLR, 25-27 Apr 2023.
[26] Heasung Kim, Hyeji Kim, and Gustavo De Veciana. Clustered federated Icarning via gradient-basce partitioning. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller. Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp. editors, Proceedings of the 41 International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 24137-24193. PMLR, 21-27 Jul 2024.
[27] Kwangho Kim, Jisu Kim, Larry A. Wasserman, and Edward H. Kennedy. Hierarchical and density- -bsed causal clustering. In A. Globerson, L. Mackey, Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang. editors, Advances in Neural Information Processing Systems, volume 37, pages 30363-30393, Curran Associates, Inc., 2024.
[28] Min Gu Kwak, Lingchao Mao, Zhiyang Zheng. Yi Su, Fleming Lure, Jing Li, and Alzheimer's Disease Neuroimaging Initiative. A Pross-Modal mutual knowledge distillation framework for alzheimer's disease diagnosis: Addressing incomplete modalities. October 2024.
12

---

[29] Daesoo Lee. Sara Malacame, and Erlend Aune. Vector quantized time series generation with a bidirectional prior model. In Francisco Ruiz, Jennifer Dy, and Man-Willem van de Meent, editors. Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206 Proceedings of Machine Learning Research, pages (665-7693. PMLR, 25-27 27 Apr 2023.
[30] Hongming Li, Shujian Yu, and Jose Principe. Causal recurrent variational autoencoder for medical time series generation. Proceedings of the AAAI Conference on Artificial Intelligence, 37(7)::562-88770 Jun. 2023.
[31] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems, 2:429-450, 2020.
[32] Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of FodAvg on non-Ill IID data. Computing Research Repository arXiv Preprints, 1907.02189 2019.
[33] Yong Liu, Haoran Zhang. Chenyu Li, Xiangdong Huang, Jianmin Wang, and Mingsheng Long. Timer: Generative pre-trained transformers are large time series models. Ruslan Salakhutdinov, 7ico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceed- ings of the 4sst International Conference on Machine Learning. volume 235 of Proceedings of Machine Learning Rexearch. pages 32399, PMLR, 21-27 Jul 2024.
[34] Zhiwen Luo, Wentao Fan, Manar Amayri, and Nizar Bouguila, Dynamic deep clustering of high- dimensional directional data via hyperspherical embeddings with bayesian nonparametric mixtures. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.I, KDD '25, page 938 949, New York, NY, USA, 2025. Association for Computing Machinery.
[35] Mengmeng Ma, Tang Li, and Xi Peng. Beyond the federation: Topology-aware federated learning for generalization to unseen clients. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41 International Conference Machine Learning, volume 235 of Proceedings of Muchine Learning Research, pages 33794-33810. PMLR, 21-27 Jul 2024.
[36] Mengmeng Ma, Jian Ren, Long Zhao, Davide Testuggine, and Peng. Are multimodal transformers robust to missing modality? In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 18177- 18186, June 2022.
[37] Tengfei Ma, Trong Nghia Hoang, and Jie Chen. Federated learning of models pre-trained on different features with consensus graphs. In Uncertainty Artifical Intelligence, pages 1336-1346, 2023.
[38] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication--ffiiceenn learning of deep networks from decentralized data. In Artificial Intelligence and Stutistics, pages 1273-1282, 2017.
[39] Manh Duong Nguyen, Trung Thanh Nguyen, Huy Hieu Pham, Trong Nghia Hoang, Phi Le Nguyen, and Thanh Trung Huynh. Fedmac: Tackling partial-modality missing federated learning with cross- modal aggregation and contrastive regularization. International Symposium on Network Computing and Applications, 2024.
[40] Nang Hung Nguyen, Due Long Nguyen, Trong Bang Nguyen, Thanh Hung Nguyen, Huy Hicu Pham, Truong Thao Nguyen, and Phi Le Nguyen. CADIS: Handling cluster-skewed Non-IID data in federated learning with clustered aggregation and knowledge distilled regularization. In Proceedings of the 23rd IEEE/ACM International Symposium on Cluster, Cloud and Internet Computing, pages 249-261, 2023.
[41] Nang Hung Nguyen, Truong Thao Nguyen, Trong Nghia Hoang, Hieu H. Pham, Thanh Hung Nguyen, and Phi Le Nguyen. Safa: Handling sparse and scarce data in federated learning with accumulative learning. IEEE Transactions on Computers, 74(6): 844 1856, 2025.
[42] Xiaomin Ouyang, Zhiyuan Xie, Heming Fu, Sitong Cheng, Li Pan, Neiwen Ling, Guoliang Xing, Jiayu Zhou, and Jianwei Huang. Harmony: Heterogencous multi-modal federated learning through disentangled model training. In Proceedings of the 2 Annual International Conference on Mobile Systems, Applica- tions and Services, MobiSys 23, page 530-543, New York, NY, USA, 2023, Association for Computing Machinery.
[43] Yuanzhe Peng. Jieming Bian, and Jie Xu. FedMM: Federated multi-modal learning with modality heterogeneity in computational pathology. In Proceedings of the 2024 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 1696-17000 2024.
13

---

[44] Hai Pham, Paul Pu Liang. Thomas Manzini, Louis-Philippes Morency. and Barnabás Póczos. Found in translation: Learning robust joint representations by eyclic translations between modalities. Proceedings the AAAI Conference on Artificial Intelligence, 33:6892-6899, Jul. 2019.
[45] Thu Hang Phung. Binh P Nguyen, Thanh Hung Nguyen, Quoe Viet Hung Nguyen, Phi Le Nguyen, and Thanh Trung Huynh. A contrastive learning and graph-based based approach for missing modalities in multimodal federated learning. In Proceedings of the 2024 International Joint Conference on Neural Networks, pages 1-8, 2024.
[46] Thu Hang Phung, Manh Duong Nguyen, Thanh Trung Huynh, Quoe Viet Hung Nguyen, Trong Nghia Hoang, and Phi Le Nguyen. Federated prompt-tuning with heterogeneous and incomplete multimodal client data, In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), October 2025.
[47] Petra Poklukar, Miguel Vasco, Hang Yin, Francisco Melo, Ana Paiva, and Danica Kragic. Geometric multimodal contrastive representation learning. In Proceedings of the 2022 International Conference on Machine Learning, pages 17782-17800, 2022.
[48] Kashif Rasul, Arjun Ashok. Andrew Robert Williams. Hena Ghonia, Rishika Bhagwatkar, Arian Khorasani, Mohammad Javad Darvishi Bayazi, George Adamopoulos, Roland Riachi, Nadhir Hassen, Marin Biloš, Sahil Garg, Anderson Schneider, Nicolas Chapados, Alexandre Drouin, Valentina Zantedeschi, Yuriy Nevmyvaka, and Irina Rish. Lag-Ilama: Towards foundation models for probabilistic time series forccasting, 2024.
[49] Sebastian Ruder. An overview of gradient descent descent optimization algorithms. Computing Research Repository arXiv Preprints, 1609.04747, 2016.
[50] Rachael Hwee Ling Sim, Yehong Zhang, Trong Nghia Hoang, Xinyi Xu, Bryan Kian Hsiang Low, and Patrick Jaillet. Incentives in private collaborative machine learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[51] Canh T Dinh, Nguyen Tram, and Josh Nguyen. Personalized federated learning with Morcau envelopes. Advances in Neural Information Processing Systems, 33:21394-2 -944-1455, 2020.
[52] U.S. Department of Health and Human Services. Standards for Privacy of Individually Identifiable Health Information. httpss: / //wwww hhs http:///aaaaooccpppppeeeeo////// CFR Parts 160 and 164.
htmll 2002, 45
[53] Patrick Wagner, Nils Strodthoff, Ralf-Dieter Bousseljot, Dieter Kreiseler, Fatima I Lunze, Wojeiech Samek, and Tobias Schaeffter. PTB-XL, large publicly available clectrocardiograpyy datasct. Scientific Data, 7(1):154, 2020.
[54] Hu Wang, Yuanhong Chen, Congbo Ma, Jodie Avery, Louise Hull, and Gustavo Carneiro. Multi-modal learning with missing modality via shared-specific feature modelling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15878- 15887, 2023.
[55] Hu Wang, Jianpeng Zhang, Yuanhong Chen, Congbo Ma, Jodie Avery. Louise Hull, and Gustavo Careeir.. Uncertainty-awac multi- mootl learning via cross-modal random network prediction, 2022.
[56] Shu Wang, Zhe Qu, Yuan Liu, Shichao Kan, Yixiong Liang, and Jianxin Wang. Fedmmr: Multi-modal federated learning via missing modality reconstruction. 2024 IEEE International Conference on Multimedia and Expo (ICME), pages 1 1-6, 2024.
[57] Yuanzhi Wang. Zhen Cui, and Yong Li. Distribution-consisteet modal recovering for incomplete multimodal learning. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), pages 22025-22034, October 2023.
[58] Pei-Yau Weng. Minh Hoang, Lam M. Nguyen, My T. Thai. Tsui- Wei Weng, and Trong Nghia Hoang. Probabilistic federated prompt-tuning with non-IID and imbalanced data. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[59] Gerald Woo, Chenghao Liu, Akshat Kumar, Caiming Xiong. Silvio Savarese, and Doyen Sahoo. Unified training of universal time series forecasting transformers. arXiv preprint arXiv:2402.02555 2024.
[60] Baochen Xiong, Xiaoshan Yang, Fan Qi, and Changsheng Xu. A unified framework for Federated Learning. Neurocomputing, 480:110-111 118.011-8,,110 2022. 2022.
14

---

[61] Kunda Yan, Sen Cui, Abudukelimu Wuerkaixi, Jingfeng Zhang, Bo Han, Gang Niu, Masashi Sugiyama. and Changshui Zhang. Balancing similarity and complementarty for federated learning. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, Felix Berkenkamp, editors, Proceedings the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 55739-55555 55758. PMLR, 21-27 Jul 2024.
[62] Xinyu Yang, Yu Sun, Xiaojie Yuan, and Xinyang Chen. Frequency-awe aware generative models for multivariate time series imputation. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 52595-52633 Curran Associates, Inc., 2024.
[63] Guan Yu, Quefeng Li, Dinggang Shen, and Yufeng Liu. Optimal sparse linear prediction for block moock-i multi-modality data without imputation. Journal of the American Statistical Association, 115(531) ): 1406- 1419, 2020.
[64] Songean Yu, Junbo Wang. Walid Hussein, and Patrick C.K. Hung. Robust multimodal federated learning incomplete modalities. Computer Communications, 214:234-243, 2024.
[65] Qiying Yu al. Multimodal federated learning via contrastive representation ensemble. Computing Research Repository arXiv Preprints, 2302.08888, 2023.
[66] Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Trong Nghia Hoang, and Yasaman Khazaeni. Bayesian nonparametric federated learning neural networks. International Conference on Machine Learning, 2019.
[67] Mikhail Yurochkin, Mayank Argawal, Soumya Ghosh, Kristjan Greenewald, and Trong Nghia Hoang. Statistical model aggregation via parameter matching, In Advances in Neural Information Processing Systems, pages 10954-10964, 2019.
[68] Changqing Zhang, Yajie Cui, Zongho Han, Joey Tianyi Zhou, Huazhu Fu, and Qinghua Hu. Deep partial multi-view learning. IEEE Transactions Pattern Analysis and Machine Intelligence, 44(5):2022-2155,
2020.
[69] Tongxue Zhou, Pierre Vera, Stéphane Canu, and Su Ruan. Missing data imputation via conditional generator and correlation learning for multimodal brain tumor segmentation. Pattern Recognition Letters, 158:125-133 2022.
15

---

A Broader Statement of Impact
This research addresses the challenge of heterogeneous missing data in multimodal federated learning. Our novel design and theoretical analysis help bridge gaps between incomplete multimodal clients fragmented systems by effectively handling diverse missing data patterns. This enables practical applications in privacy-sensitive multimodal settings with highly incomplete data. While the potential real-world use of our methods could raise ethical concerns, these are indirect and unpredictable consequences beyond the scope of this work. Our experiments rely solely on publicly available datasets, and no ethical issues arise from our evaluation process.
B Missing Modality Simulation
This section details how we simulate missing modality in a comprehensive and controllable way. Following [Р), we define two types of ratio in missing modality, denoted ps and pm- First, Ps- namely sample ratio, the ratio of samples with missing modalities over a given dataset. Second, pm is modality ratio, and used as the ratio of missing modalities within those samples. For simplicity, pair (pm. P.) can be called missing statistics, since it reflects statisites of modality missing both detailed and overall views (see Fig. 6 The missing degree is then defined pm Pa. representing the overall proportion of instances with missing modalities. These missing statistic can remodel the an arbitrary dataset D via a missing matrix:
b
b
M
((D)
(10)
b M
where bam E (0, indicates whether modality m is missing (0) or available (1) for the d th sample. Here, [M] is the cardinality of M, and [D] is the number of samples. The incomplete dataset D can be obtained by multiplying D by the missing matrix:
a = [ad @ "bai...,,ba..
(11)
where represents element-wise multiplication. Examples of incomplete datasets are shown in Fig. 6 In this work, we apply the same (Pm/pa) pairs for all clients in our experiments.
C Implementation Details
Dataset Preparation. All baselines use data from the PTBXL and EDF datasets. The PTBXL dataset contains 3,963 clinical samples across five classes. Each sample includes 12 modalities. corresponding to clectrocardiogram (ECG) recordings, and is labeled with single class. Details can be found in [45). The EDF dataset consists of 197 full-night polysomnographic (PSG) recordings with five key modalities (excluding rectal temperature and biomarkers). Each recording is segmented into multiple sleep stages, including Wake and stages S1-S4. For this work, we relabel S1 and $2 as N1 and N2, and merge S3 and S4 into N3, resulting in a 5-class classification problem [2]. We segment all sleep recordings into individual signals, each representing a sleep pattern, creating unified dataset of 8,755 signals. This unified dataset is used for all experiments. Both datasets are divided into training and testing sets with ratio 80/20. The testing are used for evaluation on the server side, while the training sets are split to all clients following IID or NonIID settings. For NonIID setting, we use Dirichlet distribution with a = 0.5 to distribute training data points. All modalities in this work are signal-based modality.
Hyperparmeter Settings. All methods in this work use an Inception Network the modality encoder, following 45
Experiments are run on an A6000 GPU with 48GB of memory. For classification, we use Cross Entropy Loss for Lrask- The embedding dimension set C 128. There are K = 32 clients in total, with 10 clients randomly selected to participate in each training round. Each selected client trains the model for = 3 epochs per round. Optinization done using Stochastic Gradient Descent (SGD) 49 Communication with the server occurs over T 1000 rounds. Both the alignment contrastive weight ()) and the relevance regularization weight (1) are set to 0.1 for all experiments. However, À is increased to 0.2 when pm {0.8, 1.0), corresponding
16

---

0.2/0.2
0.2/0.8
0.2/1.0
0.8/0.2
1.0/0.8
Modelity
Instance
Instance
Instance
Proportion
0.6
5.0
Modality
Modality
Modality
Modality
Modality
Micsing Modolities
Available Modalities
Figure 6: Examples of incomplete datasets D with varying missing statistics (pm/ (p.)..). By controlling these missing statistics, we create diverse evaluation scenarios that reflect real-world conditions.
Table Hyperparameter setting all baselines and PEPSY
Batch
Communication
Eps, in
Contrastive Weight (A)
Optimizer Learning Rate
Sampled Clients
Dataset
Method
p_m
Size
Round (7)
Local Training (E)
Clients (K)
SGD
FedProx
Ir: 0.01
HIFL
SGD
PTBXL
32
FedNSplit
SGD
FedNAC
32
1000
0.1
PEPSY
SGD
0.8
32
1000
0.2
10
SGD
1.0
32
1000
3
0.2
Ir: 0.01
32
10
SGD
FedProx
0.2
128
500
3
0.1
0.1
32
10
SGD
EDF
FedInNM
0.4
128
500
3
0.1
Ir: 0.1
32
10
FedMSplit
FedMAC
128
500
0.1
0.1
10
PEPSY
SGD
SGD
0.2
to extreme missing modality scenarios that require stronger alignment. Detailed hyperparameter settings are listed in Tab. 4 Unless otherwise specified, the original configurations from the referenced papers.
D Theorem Proof
D.1
Theorem Setup
This section provides the initial setup for our proof for Theorem 3.1
From now on, we remove the notation for simplicity. Following notations in Section T
subseript indicating instance index our proposed method described in Section 2 can be expressed as composition of two internal
functions:
([w)){M) fo ((f MM)) Here, fp(-) and f-(-) are post-process head and
feature extractor, respectively. In specific, fa(+) takes each modality as input and generates modality representation w; (as shown in Section 2 by concatenating three types information including modality-ppecific (winal)) data-specific (with) and missing-patter (wilis) features, i.e., [w with o w.""]. In addition, to make the proof easy to follow, we denote h; and u as extraction for present modalities and imputation missing modalities, respectively, as described in Section 2.2.1 leading to follow-up notation of modality representations w; and u. (u). To clarify, if the notation h; used for missing modality, i.e., E S, means that h, here the "true" feature that modality presents. We use this notation in our proof from
17

---

Assumption D.1 The post-processing head f is Lipschitz continuous with respect to the input vector 2, i.e., there exists constant L such that for all æ, y E R". the the following condition holds:
f - -
where fo. - is the post-processing head. || denotes the chosen norm (here the by-norm), and L is a Lipschitz constant.
Assumption D.2 During test time, all parameters of the proposed framevork are bounded. Specifi- cally, for any weight matrix A, we have:
7 S
where || denotes the Ez-norm and ε and e are positive constants that bound the spectral norm of A. This assumption similarly applies to the output representations that are transformed by the learned weight matrices.
we assume that the neural network used as the post-processing head in our proposed design is Lipschitz continuous. This assumption is widely accepted in the machine learning community due its relevance in ensuring stable and smooth behavior of the model.
In Assumption D.1
Assumption D.2 states that the learned parameters the network are bounded during test time. This assumption is reasonable and holds true in most real-worke scenarios, where the model parameters are deterministic and constrained within known ranges during inference. Such bounds are typically enforced either through explicit regularization during training or through implicit constraints imposed by the training process itself (e.g., gradient clipping or weight normalization). Therefore, this assumption is only theoretically sound but also consistent with common practices in machine learning.
Remark D.3 (Bounded Extracted Representations/ In our data-specific representation extraction, each output feature h; of modality normalized to zero mean and unit variance (via Batch Normal- ization layer), followed by a learned scaling (7) and shift (3) parameters. When Assumption D.2
holds, we have E, S |r> E, and Eg S ||||| S Eg and derive:
|| = 11yh + A1l S ||>l || + ||811.
(12)
where h; is batch- h, Since the normalized term has unit variance, its norm is bounded by VC, where C is the feature dimension. Hence,
max(c, - 0) - hall S VC
(13)
Let = 2 max((,, VC Ez. 0) and ets & a deterministic range. Consequently, the imputation feature derived by taking average of available modalities is bounded for the same reason.
Eq. 13 shows that [/a || is bounded within
D.2 Theoretical Analysis in Simple Case
In this section, we first investigate the behavior of PEPSY a simple case of missing modality before further generalization. Let consider the deviations of our proposal when feeding full-modality input and one missing the first [S] out of M modalities, i.c., S =
as follows:
s -y111
(14)
fp ( S (ur (wr(w))][1)) fp (fww:)
(15)
| [[)](u) wi w ..... ... || (w)) lwii |
(w)
s
(16)
[s]
S Σ
(w) - fpm
(17)
18

---

Here, we use first-order Taylor approximation |S] times to transform Eg. 15 to Eq. 16 Since fp(-) is L- L-Lipschitz (see Assumption D.I. Eq T7 can be transformed as:
y
I
(18)
IM
$ L
(unx(u)) - w0;|
(19)
1=1
IM
S I.
Σ
[[[ [1o mood u (
h, o "
(20)
IM
L Σ
00 (u h2) argmax (eq((((()))) (wffoo u), Up)) argmax (e(q(wff o h4), (h))))))))
(21)
Vp
where w; (.(u) is the imputed representation for modality i, obtained using the imputation data-specific feature =n and w, is the original modality feature. Here, we represent the query-key matching function argmax e(q(w)ped w. ), 25p)) as an approximate attention selecting the u with the 4
highest weight, by using softmax function (t..)) e softmax(e(q()).)) For simplicity, we use ă as a Lipschitz constant of this approximated similarity function. Considering individual modality component (w) wyll, these lead to the following derivations:
(w) wil
(22)
00 (u h.)
[o (unnfood u, xb,)) a awwffod his wp))] o 1bp
(23)
S 5Iu hill + Σ
oo [ mod u, ubp) o ow hes bb)) o 1 I|-
(24)
S <|u hill + +ö
1| hill x |1
(25)
p=1
< (1 ōTmax 4p
||ue - hill
(26)
S PV //|/2 + || 2 2uh! T
(27)
where μ 1 + Tmma and S, denotes upperbound of emhedding controls, which is fixed in
Vp
test time. Taking the summation over all i, we obtain:
S
(w:(u)= wwill
(28)
[s|
|| + |1,,112 2uh;
(29)
[5|
IMI Σ
IM|
S 11
hy
+
11..11
2
Σ -=|
h
(30)
M
S
M
S
+=18++1
Here, u represents the imputed data-specific representation, computed as the mean of corresponding features from the available modalities (see Section 2.2.1. This justifies the transformation from
Eq. 29 to Eq. 30 Based on Remark D.3]
we have:
[s]
lww (w) ww;l
(31)
19

---

8
|M
2
2
|8| (I - 181) +2
+ +G
hihh
(32)
M
7/
|
|
IMI
11
36 7/
2
2
Σ
h
(33)
M
-
[|]
j=
S
[M
7
3c 7/8
2
Σ
h
(34)
M
[
i=1
s|+1
IMI
11
1
Σ
3e 78
2
2hgh
(35)
MM|
- |S|
|M [|]
j=1/++1
j=
S1+1
112
8
M
Σ ==1
3c +2
2h3h
(36)
V M - |S|
S
+1
Here, the bound on (w) - will highlights how the interaction terms between h2 and h, contribute to the overall norm. Furthermore, the hight-handed side of 36
is non-negative showing the validity of
this transformation. If we further substitute Eq. 36 in Eq 19
we obtain an intermediate inequality:
122
LM
1 1
Σ
3
nh,
(37)
M
S
j=|S|+1
where we restate 1²² + L without loss of generalization since both μ and L are constant.
D.3 Theoretical Analysis Generalization
In this section, extend the bound in Eq. 37 originally derived assuming the first [S] modalities of . are missing. The current bound assumes the missing modalities are the first [S] in order. We generalize this to the case where any subset S C M of size [S] is missing. To do this, generalize bound Eq. 37 over missing modality set S, and over all instances of an arbitrary dataset D.
D.3.1 Generalize over Missing Modality Set.
Given M the set of all modalities, with cardinality [MM, we define S S M as a subset representing the missing modalities, with cardinality of (S]. For each missing modality E S, we define a random variable Z as follows:
2
Σ(3e 2hgh,), jeS
(38)
The expected value of v Z's averaged over all possible missing subsets S, is then computed the following equation:
E
Z's
Σ SCM
V
z's
(39)
(
(3c2
22
(40)
(((()))
SC.
jØS
where (()))) denotes the number of ways to choose |S] elements from M. To derive Eq 40 from we apply the Jensen's 's inequality due to the concavity of square root function.
Eq. 39
Observation. The term Es Eies (3Es(3ez
2h,h, ) means that we are summing over all
subsets S A of fixed size [S]. For each subset, we sum over all ordered pairs (i,j) where i E S and j & S. For a fixed pair (1,5) with + j. the number of subsets S that include i and exclude depends only on and In other words, once and are fixed, the remaining [ 1 elements of S
20

---

must be chosen from the remaining |M| - 2 elements (excluding i and j), giving exactly (si- subsets. Therefore, each term (3 2h, h, ) appears precisely (is(-1) times in the sum. This lets
us rewrite the original triple sum as double sum over all ordered pairs (i,j) with + j. multiplied by the constant (N-1)- simplifying into:
IMIIMI ΣΣ jfi
Σ ΣΣΣ SCM iES jØS
2 70
2h,h
(IM) 2) (3e² -2h,h)) [S]
-2
(41)
Substituting Eq. 41
into Eq. 40
we obtain:
s
V
Z's
(42)
I IMI
(IM 2)
S] - (3c 2h2h2 )
(43)
(
==1 j41
IM IM
| IS)!
(IM) - 2)
(3
2h1h)
(44)
MMili||
(IS) T) -18| 18| 1)!
[M M
I - |S|
(3e3. 221h,
(45)
MI(M 1)
i=1
MM IM
M |S
Σ
(3e (3e 2hgh,)
(46)
MI(M 1)
We now bound the expectation of Eq.
37
over all possible missing modality patterns (S) as follows:
Es
3 201]
1
y
(47)
(5)
SCM
1
Σ
Σ j&s
(3c,)
h,11)
(48)
M - -|81 (8((((9)
SCM
iES
1
Es. V Z'S
(49)
M -181
2
|M| - -S
M M
(3ct +2 2h:h,)
(50)
M 18|
[M][IMM -)
IM IM
ΣΣ 1
(3
2h,h,)) }).
(51)
M 1)
In summary, in this section, we derive an upper bound for the expected outcome deviation in missing- and full- full-modality scenarios over the missing scenarios (S) as:
Es |100 2016 201]= s MMI(MI - 1)
12/
[M IM ΣΣ
(3ct]
2h,h
(52)
i= s41
1=1
21

---

D.3.2 Generalize over Instances
This section describes how we generalize the bound in Eq. 52 to batch- or dataset-level. Furthermore, we reveal the connection between our theoretical bound and the training loss function that we propose, indicating the effectiveness of training loss in our proposal. To address this, we start by considering the mean difference over a dataset with cardinality [D]:
D Rs
I
voall
121S²2
1
ID
IM
IM
(3r 1
2hartees
(53)
MMI(IM) 1)
D
34
VPPl D [IVIMI -1)
D
M
I
(3e2,
2h)
(54)
in which Eq. 53 transformed to Eq. 54 by using triangle inequality. To avoid confusion, we analyze the right-hand term separately, as it plays a central role in the transformation process. Let has denote the fz-normalized feature, i.e., has = has/ /
DI
M
M
33 2
(55)
p
M
MI
22
70
2 T
33
(56)
i+i
LM
2
+ 20,0
ΣΣΣ
h + log (ID((D) 1)IM
S3
9
(57)
2
7
+2
< WiDliiiii 1) + 2E.9 diijfi
Σ (- h-ahh log (DI((D) 1)IM ?)
5
(58)
S MiMiiiiie 1)ct3
+ 22 2 Σ
hachh + log (ID[((D) 1)IM exp(p
50
(2)))))
(59)
d
S < -1)
D
[ IM
+ 2
Σ
-logesp exp (haittt + log
Σ
Σ datt1
Σ
Σ 1z
exp( (-(( (1-2)))))
(60)
Exg
di
expl Soohhaiee
$ WMMiiliiii - 1)c
+ 26
Σ log
(61)
EPPl
[D]
D
[IMI Σ E/Ml exp(p ([ر)) EgB
diij
dofdd
exp(hactis))
S SMDDIMII[M 1 + 2E
Σ log
(62)
ΣD
Σ
ΣM
M
EM exp(Fes Rdgg
d
Ed
ks
2 SSDDIMM(M - 1) + 26,
2
DD) Las (aat 0)
(63)
where Las (.-)) defined in Section
2.2.
Substitute Eq. 63 into Eq. 54
have:
[D]
1
Σ Es
22 5ct
+
Σ
Las 0)
(64)
D
Mallii(ia)
1)
22

---

We now investigate how the presence of missing modalities impacts the bound, and consequently, the effectiveness of our approach. Assume each instance F E 'D has missing set Sa with the same cardinality (S), i.e., S M, |Sal = [S] vd. Hence,
exp(haihag)
Las (xd, 0) =
Σ
log
(65)
Σd d2
E dtt Edy
Eds MM
exp(ha hagke)
D
D
M
LM]
log exp( - ( T
+ log
Σ
exp(bark midats))
(66)
di
da
Rn
Let Ai _ Edi.zi - ha and A2 - Edi
Σ
D
Σn
M
Σ MI
exp(ha, dockeds we now
d2
k1
further expand each term as follows:
Consider
A1:
D
M
A =
haih d
(67)
[D]
M
[MI[MI] - 1) (I.) |
hach
(68)
(IMI 51)]5] [M[[M 1)
-IM((MM- 1) (IMI - |8
(IM) 2)
|21
hach
(69)
(IMI- - SS)SS MI!
(1S) D) - |S| 1
[ IM
MMI(MM 1)
[M 2
h
(70)
(IMI SI (15))
[S] 1
[D]
= M - 1) (I - S|)
Esa
ΣΣ> Σ
- h
(71)
d iESaJ$8S
Under missing modality scenarios, i.e., Sa + 0, hai V E Sa is approximated as M Lies. hag In other words, express Eq. 7 as:
|MK[M 1)
A
(IMI - (S))SS R.s.
haithe
(72)
iESa J&Sa
D
(IM) - ([M)-18))SS) L.S.
IMI[M - 1)
1
Σ Kg
Fran te
(73)
M - S
iES j&Sa
[
IMIMM 1) 1) E.S.
Σ
Σ
hr dt
(74)
(IMI - 181)2151
iESa JES K.
s M 1) E.s.
DD]
Σ
hax Fe
(75)
(IM) - 8112
S&SA S
(76)
Consider A2:
DI M IM
A
exp (aaik k2
(77)
12
M
Σ
Σ
exp(hd, Wddded
Σ
(78)
di
12ESa2
23

---

P
Σ
exp(haiii hasta
mdate) +
Σ
exp(hdid Adaia)
di didd
S2
E2
exp(hdiss hdste) + f2
Σ
exp(hassi Mowill
(79)
is ESE,
x2
P
Σ
exp(tarn haii hddjs)
exp(harii
di d
Ji JaRSu2
Σ
exp(hani Redrsa)
E
Σ
exp(hdrn Addete
(80)
i ES Sa
i1ESu; 12E8d
Σ JiESa J2
exp(hdin h Rdaso)
Σ
exp (Add
Σ
haziz)
[M |||
do
Si
E
E
exp(
1
Σ
harji hdnjz)
|S|
jI
S2
exp
1
Σ
Σ
ha.ji Maass
(81)
(I 81)3 EiES, jadSaaz
i E f2ESu2
Σ
exp(harái h + |8|
Σ exp (hdrii [ -11 181 jo
Σ hdnia)
d1
Ji gSa,
jiKSa
J2
+ |S|
Σ exp(
Σ
ha hdddi)
M |S|
So
jigSa
+ 1 exp(
Σ
Σ
hagi Mdsste)
(82)
(IM) - S1))
Su8Sa,
Observation. Exponential is convex function, hence we apply Jensen's inequality to the followings:
1 Entso, harii Iddge)) S (IMI-151) 1. |S exp( (1122 128822
1
En ES,, exp(ha.j, hddsz) S2
-T
2. |S] ESIEns exp(hann MM|-is| Engesoo hddie))
Σ in ESa, exp(ha h in 5
M-
3. [8] Engss, exp( [MI Enes,, hai Rdair)
T [S] Σ ji ESA, exp h
Ja J2
which derive Eq. into:
D
SS|
SS]
Ae
1
+
(83)
M
-|8|
[M
[S]
dr.dadd, 4S S
[D]
[S)
+ +1) 2
Σ
exp(ha.nn hdni))
(84)
M - S
di di ii
24

---

1
M
exp(ha.ji hdds2)
(85)
M
[8|
d1.da #d,
Su
Sa
Substitute Eq. 71 and 85
into Eq.
66
we have:
Las 0)
(86)
D
- MMMM - 1) Esa
Σ
Σ
(IMI 11112
J&Sa Kg
IM
2
log
exp(hd Reb22)
(87)
M
S
du dod ji f2
D
< MMIMM 11 Esa
haa hidj
(IM) [81)
JES. KeS
+ 1 log (MM) - S
IM
exp(hdis haztza
(88)
di di.da d ji SSa
J2d
- MMI[MM 1) DD
Σ
Es. ΣV Σ
log exp harh
(IM) 1|))2
j&Sa kg..
IM
[D]
exp(haisi |
8og
Σ
(89)
M [8]
de daddtpph in SSa
Sa
- MMI[M 1)
DDI Σ
exp Fas
Esa
log
(IM) 1812
Σnn. de En RSS,
exp(hs, R2os))
d
jeSa KgS
12
IM
2
Σ log
(90)
M
(S|
J&SA S.
< MM 1)
IM
2
Σ
Ess
Las (ard, Sa)
(IM) 1181)
(91)
(IMI (S))
[. - |S]
Substitute Eq. 91 in Eq. 97] have:
Es
| æs
vo.1ll ||
(92)
D
d=1
2
S 111s)
+2 7/8
Las (xx, 0)
(93)
DI 1)
2
|
$ 11Ss
5e
Esa Las Sa)
2G, log
(94)
1/8
(IMI - 1S1)2
D
([M)
(S)
which is equivalent to:
Ex.s -1 20ll
(95)
25

---

2r
S 1 B + (IM) - (81) Ex, Ldslx 8)
+ 2 log
(96)
(IM) - [8])
$ 0 1 (IM) (S])
Ex,s Cdo (20 (x,
|M2
+ log (IM -
(97)
E
Complexity Analysis
E.1
Analysis
We start by introducing the time complexity of traditional FL algorithms, such as FedAvg, FedProx as a baseline to analyze the time complexity of PEPSY. Let:
d: feature extractor size
T: number of embedding controls in local data-missing profile.
m:
dp: embedding control dimensionality.
dy: key vector dimensionality
number of embedding controls selected per query (small constant)
E: local epochs
B: batch size
ng: local data size
M: number of optimization iterations of PFPT-based clustering
Table 5: Comparison of Time and Communication Complexity of PEPSY and traditional FL
Component
Traditional FL
PEPSY
Local Computation
( nt .d)
O ( [k
E [[(d + dp) T- -dx))
Client Communication
O(d)
Old + pdp)
Server Aggregation
O(Kd)
O(Kd+ MK3p²d2)
Traditional FL. Each client updates its local parameters over E iterations, with batch size of B using a model of size d. This cost:
Trocal = O(E ni d) Subsequently. the modal parameters B of all clients sent to the server costing: Teom (dd On the server side, all parameters of K clients are combined, typically using variants of weighted average leading to aggregation time cost: Tserver O(K .d)
PEPSY Modification. In PEPSY, the added cost comes from each client's data-missing profile and the pFpT-based clustering [58]. The time complexities are as follows:
Embedding Controls Selection. Each client computes key vector q Rdd per batch, compares with controls, and selects top-k controls. If done once per batch, this adds O ( B - dx) per round.
The selected controls are injected into the model and used during both forward and backward passes. This adds gradient updates with cost O(d + к dp). Over E. 2g steps, the total local computation
cost is: Tical O RL E' [(d
dp) + dxl)
Communication Cost. Clients also send their p with p S selected control embeddings (a subset of data data-missing profile), adding to the model upload cost: Teom = O(d +p-ppp.0 pdd
Server Aggregation and Clustering. Model aggregation stays OKKd), but PFPT adds overhead from bi-le optimization (over iterations) and Hungarian matching. Clustering over Kp points add more time complexity the cost: Terver O(Kd M 2
Discussion. While reducing the cost associated with the data-missing profile nontrivial, the computational cost of the PFPT-based clustering algorithm can be optimized. If we fix the model
26

---

Table 6: Empirical computational overhead of baselines and proposal comparison.
Method
Computation Metric
0.2/0.2
0.2/0.4
0.2/0.6
0.2/0.8
0.2/1.0
Training time per round (s)
50.21
50.43
50.41
49.8
49.77
FedProx
Inference time (s)
3.56
3.57
3.6
3.74
3.59
GPU for training (GB)
2.72
2.72
2.72
2.72
2.72
Training time per round (s)
94.11
94.04
93.92
92.98
93.34
MIFL
Inference time (s)
4.11
4.12
4.13
4.1
4.16
GPU for training (GB)
3.26
3.26
3.26
3.26
3.26
Training time per round (s)
100.23
97.71
97.95
99.28
96.44
FedInMM
Inference time (s)
4.89
4.86
4.83
4.95
4.89
GPU for training (GB)
2.55
2.55
2.55
2.55
2.55
Training time per round (s)
86.34
86.63
86.56
86.67
86.18
FedMSplit
Inference time (s)
3.59
3,58
3.6
3.6
3.6
GPU for training (GB)
3.21
3.21
3.21
3.21
3.21
Training time per round (s)
51.77
51.11
51.07
51.19
51.21
FedMAC
Inference time (s)
4.56
4.98
4.69
4.69
4.88
GPU for training (GB)
1.99
1.99
1.99
1.99
1.99
Training time per round (s)
141.12
153.95
137.66
140.12
146.48
PEPSY
Inference time (s)
4.69
4.99
4.9
4.73
4.87
GPU for training (GB)
2.61
2.63
2.15
2.86
2.8
architecture and instead adopt standard federated learning (FL. approach on the server side, the clustering step is removed, and the total server cost becomes O(Kd + Krd,), where each client sends its full data-missing profile of size т.
E.2 Empirical Overhead
As shown in Table we compared the computational overhead of PEPSY with existing baselines in different Pm. PPs scenarios and found that the additional cost in PEPSY primarily incurred during training, regardless of the missingness scenario. This aligns with the time complexity analysis, as the PFPT-based clustering algorithm requires more time for clustering. inference time and GPU usage remain comparable to other methods, while still delivering superior performance. This is because the data-missing profile relatively small compared the model size, adding minimal overhead to each forward pass.
contrast, PEPSY's
E.3 Recommended Solution
To improve PEPSY's computational efficiency match that of traditional FI., tune the PFPT clustering cost to stay within this bound. Specifically, by setting O(M 3 = O OKTdd)) We solve for p to determine the number of selected controls each client needs to send. Assuming
In practice, this can be implemented by having each client transmit only the top-p most frequently selected controls from its profile.
M. K. and d, are fixed system parameters, this yields: = -
MKd,
27

---

Table 7: Impact of top-p frequently selected controls from each client's profile overall performance.
Experiments are conducted on EDF datasets.
Method
Overall Accuracy (%)
FedProx
43,24
MIFL
43.18
FedInMM
40,56
FedMSplit
45.18
FedMAC
49.8
PEPSY (p 5)
50.77
PEPSY (p 10)
50,45
PEPSY (p 20)
51.51
PEPSY (no limit)
56.36
As can be seen from 7 our method still outperforms the baselines substantially even when is reduced match the cost of FedAvg. This is rum on the EDF dataset, with 0.2/0.2 missingness, and for each client, we only take most selected embedding controls to sent to the server.
F
Additional Experimental Results
F.1 Additional Comparison with Baselines
Extensive Missing Scenarios Analysis. In addition to the results in the main text, conducted further experiments comparing the performance of PEPSY (our method) with baselines under more varied missing modality scenarios. Specifically, expanded the values of Pm and ps include 0.4, 0.6, and 1.0, covering a range from 0.2 to 1.0. The results are shown Tab. and Tab. TO]
As can be seen in these tables, PEPSY consistently outperforms all baselines across all testing scenarios. For the PTBXL dataset (see Tab. S). the performance gap is small (3% - 4%) when the missing degree is low, e.g., Pm 0.2. However, as the missing degree increases (e.g.. Pm 0.8 and Pm = 1.0), PEPSY maintains clear advantage over other methods both IID and NonIID settings, with significant gap of approximate 11% in accuracy. Similarly, for the EDF dataset, PEPSY outperforms baselines by a significant margin up to nearly 10% across additional missing modality scenarios. This demonstrates the effectiveness and robustness of our approach to missing modalities in federated learning systems, regardless of data heterogeneity.
Table 8: Performance of baselines on the PTBXL dataset under various missing patterns in train and test sets, for both IID and Non-IID scenarios. The best and second-best results are highlighted in bold red and blue, respectively. We use a hyphen (-) to denote pm/ pp. = 1.0/1.0, indicating that all modalities are missing and these cases excluded from evaluation.
pmips
Methud
ITD
NonfID
FedProx
71.63%
63.81%
65.57%
64.69%
45.76%
47.79%
45.27%
39.97%
33.67%
37.55 17.58%
71.37%
65.95%
46%
45.02%
53.85%
39.22%
37.33%
38.08%
37.20%
0.4
PadTNMM
PedNSplit
70.62%
69.61%
68.35%
62.93%
64.69%
60.28%
63.43%
60.66%
64 19%
38.97%
63.43%
53.97%
48.17%
13%
43.17%
62.29%
61.66%
46.27%
34.30%
59.500 5255
FedMAC
75.79%
74.02%
73.52%
73.64%
67.
69.48%
52.21%
45.65%
47.41%
PEPSY
78.44%
77.55%
76.04%
76.29%
71.37%
71.12%
71.12%
68.10%
70.87 %
70.62%
PedProx
72.38%
69. 74%
65.07%
63.18%
47. 47.41%
44.01%
38.08%
37.45%
28.75%
29. 09.000
67.59%
25.47%
25.60%
PadInMM
67.21%
61.79%
54%
58.26%
25.60%
62.42%
59.14%
49.56%
56.36%
49 43%
0.6
PedMSplit
69.10%
63.81%:
51.45%
40.48%
37.07%
40.73%
47.29%
38.71%
35.43%
26.48%
FadMAG
75.28%
73.52%
56.75%
50.06%
PEPSY 76.58%
74.53%
74.15%
74.15%
$7.63%
70.87 %
69.23%
68.47%
68.98 %
$8. 68.76%
PedProx
75.03%
72.63%%
68.73%
58.51%
61.03%
51.57%
42. 62%
33.29%
PedIaMM
62.80%
62.42%
53.97%
49.68%
59.02%
54.85%
50.06%
41 41.866%
1.0
PedxSplit
72.13%
68.10%
66.46%
54,48%
57.25%
52.08%
45 02.02
33.92%
PedxaC
75.16%
74.40%
72.38%
69.74%
59.52%
44.51%
51.32%
41.74%
PEPSY
76.04%
77.05%
75.03%
72.76%
71.25%
67.21%
68.60%
59.14%
28

---

08/0.2
08/0.4
08/0.6
08/0.8
FedProx
FedMAC
PEPSY
Modality
Modality
Modality
Modality
Modality
Figure 7: Modality representations of different methods under multiple missing scenarios. We train and provide I-SNE 2D visualizations of modality representations constructed by three methods, including our proposal, in different Pm.P, settings. All experiments are conducted on EDF dataset, nonlID setting.
Table 9: Ablation studies crucial components of PEPSY under different missing statistics (pm/pa)) We report top-1 accuracy across multiple experiments EDF dataset, NonIID setting.
Method 0.8/0.2
0.8/0.4
0.8/0.6
0.8/0.8
0.8/1.0
PEPSY-WV 46.49%
47.92%
52.42%
52.08%
43.98%
PEPSY-NR 43.30%
43.47%
43.47%
43.58%
19.97%
PEPSY 51.80%
51.06%
55.05% %
52.25%
46.09
Modality Alignment Analysis. Fig. compares modality alignment of proposed PEPSY and two other baselines, namely FedProx and FedMAC, which correspond to traditional FL. method and second-best approach in most evaluation experiments. Intuitively, achieve high performance regardless of available modalities, an optimal solution should align modalities well a representation space, which hence discards reliance on present modalities. As be seen from Fig. 7 FedProx and FedMAC fail to align different modalities, indicating their strong dependence on different available modality sets. This is because FedProx does not have mechanism for modality alignment, while FedMAC discards modality-speciff information. In contrast, our proposed PEPSY integrates both modality- and data- specific information, which are futher reconfigured by shareable Bata-missing profile leading to less reliance on modalities. The figures show how all modalities are aligned after PEPSY's training, highlighting effectiveness of the proposal under missing modality scenarios.
Additional Ablation Studies
In this section, we conduct additional ablation studies two crucial components in our design: data- missing profile, along with the relevance loss term, and modality fusion, along with the reconfiguration regularization. Correspondingly, we introduce two variants of PEPSY, namely PEPSY-NP (No Profile)
29

---

PEPSY
PEPSY
PEPSY
Miss 1
Miss 1,4
Miss 1,4,5
Miss 1,3,4,5
Full Modality
Figure 8: Stability of modality representations under different missing modality scenarios. Ideally, modality's representation should remain stable regardless of which other modalities missing. This stability achieved when either data-missing profile removed (-NP version) or the reconfiguration signal is omitted (-NR version) from our proposed PEPSY.
Table 10: Performance of baselines on the EDF dataset under various missing patterns in train and test sets, for both IID and Non-IID scenarios. The best and second-best results highlighted in bold red and blue. respectively. We use a hyphen (-) to denote pm/pp. 1.0/1.0, indicating that modalities are missing and these cases are excluded from evaluation.
pmlys
Method
IID
NonllD
0.4
0.6
0.4
0.6
0.8
FedProx
44.38%
44.25%
43,705
44.95%
43.07%
45.00%
44.55%
44.61%
44.55%
44.72%
MIFL
43.72%
44.72%
44.49%
FedIaMM
40.50%
40.50%
40.67%
40.56%
40.90%
40.62%
42.38%
40.50%
40.45%
41.19%
FedMSplit
44.95%
45.10%
44.61%
44.61%
44.67%
44.43%
44.38%
44.61%
44.10%
44.21%
FedMAC
50.49%
48.26%
48.09%
50.03%
41.93%
49. 89%
46.49%
46.66%
44.72%
46.83%
PEPSY
55.68%
55.33%
54.54%
55.45% %
49,91 %
58.02 %
52.54%
49.80%
48.32%
51.97 %
FedProx
34.91%
34.23%
33.14%
29 89%
42.61% 1%
41.24%
42.50%
42,56%
43. 18.1%
40.45%
MIFL
44.32%
42.84%
43.98
44.78%
44.61%
18%
44.38%
44.38
44. 10%
FedIaMM
40.67%
40.44%
40.56%
40.62%
40.45%
41.47%
41.7%
40.73%
60.62%
40.67%
FedMSplit
44.38%
44.55%
44.61%
44.44%
43.47%
44.15%
44.55%
15%
42.27%
43 53%
FedMAC
50.99%
49.40%
48.66%
48.20%
16.71%
47.80%
47.46%
45.58%
43.64%
38.62%
PEPSY
51.28%
50.54%
50.26%
50.60 %
44.66 %
48.66
51.12%
49.67%
51.85% %
45.07 %
FedProx
36.22%
35. 14%
33.89%
31.72%
44.38%
44.67%
44.44%
43.75%
MIFL
42.56%
42.90%
41.19%
41.47%
44.15%
43.75%
44.27%
44.21%
FedInMM
40.45%
40.56%
40.50%
40.22%
40.56%
40.39%
40.38%
40.27%
FedMSplit
43.47%
43.47%
42.56%
.24%
42.44%
43.98%
43,70%
44. 44.89%
FedMAC
40.22%
40.45%
40.96%
38.11%
47.22%
46.83%
46.44%
46. 15%
PEPSY
54.93%
52.48
48.49 %
45.41 %
50.09
48.26%
49.67%
49.96%
and PEPSY-NR (No Reconfiguration) To evaluate their contributions in our proposal, we analyse both quantiative and qualitative results.
Quantitative Results. Tab. shows impacts of different components on the final components. First, when we remove data-missing profile (see PEPSY-NP variant), the performance drops from 0.2% to 4%, indicating the importance data-missing profile to stablize output performance. variant, the reconfiguration supervision signal, contrastive alignment based loss, is preserved, hence ensuring modalities are aligned, which are eventually similar to modality fusion in previous works [54] 39 On the other hand, omitting reconfiguration signal and modality fusion, which results in PEPSY-NE variant, worsen final performance by larger margin, up to more than 26%. This is because without the reconfiguration signal, the data-missing profile lacks guidance to reconfigure the biased information generated from raw data into complete ones, hence failing to handle missing modalities efficiently. In summary, both components crucial in our design ensure robust and stable performance in multimodal federated learning.
this
Qualitative Results. We further visualize representations that each PEPSY variant constructs for an individual modality under different missing scenarios, given the same trained backbone. In particular, each variant is trained on specific missing statistic pm/ Pm/pp 0.8/0.8 8 in NonIID setting and tested on
30

---

Table 11: Performance of baselines under image-sensor modality settings, conduced EDF dataset two representative missing scenarios. The and second-best results are highlighted bold red and blue, respectively.
pm/ps
FedProx
MIFL
FedInMM
FedMSplit
FedMAC
PEPSY
0.2/0.2
44.32
44.89
40.22
43.93
39.70 38. 16
44.95 44.78
0.8/0.8
41.76
43.07
40.21
42.56
handcrafted missing tests, including: Miss 1 (modality is missed); Miss 1, 4; Missing 1, 4, 5; Miss 1, 3, 5; Full modality. Intuitively, representation constructed for modality should remain closely aligned across all tests. As can be seen in Fig. while two ablated variants PEPSY-NP and PEPSY-MF fail to ensure this stability, all settings, highlighting its stable feature construction. This is because our data data-missing profile effectively distills data- mata-missing information from raw data, which are used later for reconfiguration. These visualizations further emphasize completeness of our design.
proposed PEPSY construct closely aligned representations in
F.3 Ablation on different forms of modality
To evaluate our proposed PEPSY framework more comprchensively, we conduct an additional experi- ment in an image-sensor multi-modal setting to show the broad generality of the proposal, instead of sensor-based based modality settings the original benchmark datasets. In specific, signal-based modality into an image showing fluctuation of the signal, leading to an image- based modality. Our algorithm has access to this image-basee based modality but not the original signal- -ased modality. It will learn to combine this image-based modality with other signal-hased modality to make accurate predictions. We further replaced the corresponding feature extractor simple convolutional neural network to handle image-based modality, and run several experiments to show the efficiency of our algorithm different modality domains. Each image modality is of size 128 x 64, and normalized to scale from 0 to 1, as presented in Table
converted one
Table IT shows that even under missing settings with different modality forms, PEPSY still outper- forms all other baselines. This additional experiment futher emphasizes the superiority PEPSY in the ability to handle severe missingness.
G Limitations
Although PEPSY outperforms prior methods in handling heterogeneous data-missing patterns in mul- timodal federated learning, may face challenges when downstream task domains vary significantly. Large domain shifts can create distinct, domain-specific missing data profiles that require more train- able embeddings for effective adaptation. key open question is whether we can quantify these shifts and bound the number of embeddings needed for reconfiguration but important for future research, especially in federated settings with clients operating in diverse domains and missing data patterns. Moreover, this study relies on training models from scratch and does not leverage pretrained foundation models. Future efforts could explore incorporating pretrained encoders to build shareable missing data profiles, improving representation learning efficiency and effectiveness.
issue beyond this work's ' scope
31

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our contributions are detailed in Section 2 3 and
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer to Appendix G
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
32

---

Justification:
Please refer to Section 3 and Appendix D
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Our experiment protocol is described in Section The implementation details are described in Appendix and
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
33

---

Answer: [Yes]
Justification: All the used datasets are publicly available. Upon acceptance, we will release our code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cmm ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
//nips. //nips.cc/ppl cc/public/guu wet.ggddddee/eeebbiniinn
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Such details can be found in Section 4 Appendix and B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We have provided error bars for main results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
34

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Compute resources are in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We have read the NeurIPS Code of Ethics and do not find our work violate any aspects the code
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We provide a statement of impact Appendix A
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
35

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: Our work do not create new dataset or create new pre-trained NLP or vision models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite the source of all datasets and baselines used in experiments.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the batasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
36

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our work does release any new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
37

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: Our contributions do not involve LLMs important components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy fattps://neur// 2 for what should or should not be described.
38


Reviews:
[{"review_id": "xFXTjTN9A3", "reviewer": "Reviewer_nfrN", "summary": "This paper addresses the issue of incomplete and heterogeneous data in multimodal federated learning (MMFL), where clients may possess different modality subsets and suffer from missing features within modalities. The authors propose a novel framework called PEPSY, which leverages learnable client-side embedding controls to encode data-missing patterns and generate locally adaptive representations. These embeddings serve as reconfiguration signals to align globally aggregated representations with each client’s local context, and embeddings from clients with similar missing patterns are algorithmically aggregated to enhance the robustness of reconfiguration signals for global representation adaptation. The key contributions include: developing the PEPSY framework to handle both modality and feature missing scenarios by integrating missing-pattern information into representation reconfiguration without raw data sharing; establishing a theoretical performance bound via theorem proof to demonstrate PEPSY’s stability under random missing patterns by minimizing prediction discrepancies between complete and missing modality cases; and empirically validating that PEPSY achieves up to 36.45% performance improvement over baselines on federated multimodal benchmarks (PTBXL and SleepEDF) under severe data incompleteness, showcasing robust performance in both IID and Non-IID settings. This work provides an effective solution for missing data in MMFL, addressing the gap in existing methods that fail to handle combined modality and feature missing challenges.", "questions": "1. The theoretical analysis in the paper is based on the assumption of μ-Lipschitz continuity of the model, which may not fully cover real-world scenarios with complex and irregular data distributions. Additionally, the performance boundaries in extreme missing scenarios (such as when only a single modality is available) are not clearly demonstrated. It is recommended that the authors relax the theoretical assumptions by introducing more universal continuity conditions or validate the model's performance in extreme missing scenarios through experiments simulating complex data distributions, so as to enhance the practical applicability of the theory.  \n\n2. The dynamic selection of client-side embedding controls and the server-side PFPT probabilistic clustering algorithm are likely to cause a significant increase in communication overhead in large-scale federated learning scenarios with numerous clients, but the paper does not provide a detailed analysis of the computational efficiency in this regard. It is suggested that the authors supplement the communication complexity assessment in large-scale scenarios or propose targeted lightweight optimization strategies, such as sparse embedding selection or more efficient clustering algorithms, to improve the feasibility of model deployment in practical large-scale environments.  \n\n3. The current experiment uses a binary missing matrix generated with fixed probabilities, which fails to fully simulate real-world continuous missing patterns (such as prolonged sensor failures) or inter-modality dependencies (where the absence of one modality may affect the validity of another). It is recommended that the authors introduce more realistic complex missing pattern generation mechanisms in future experiments, such as simulating continuous missing based on temporal dependencies or considering potential correlations between modalities, to verify the model's adaptability in more authentic scenarios.  \n\n4. The need for manual tuning of contrastive loss weights (λ, η) may severely limit the model's automated deployment and application across different domains. It is suggested that the authors explore integrating automated tuning mechanisms, such as Bayesian optimization or model-based meta-learning, and optimize the tuning strategy through testing on cross-domain datasets to enhance the model's convenience and performance stability in various application domains.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths**  \n1. The PEPSY framework is proposed to address the co-occurrence of modality and feature missing in multimodal federated learning. It encodes data-missing patterns through client-side embedding controls, enabling global aggregated representations to adapt to the local context of each client, which fills the gap of traditional methods in handling mixed missing scenarios.  \n2. Experiments validate the effectiveness of PEPSY on PTBXL and SleepEDF datasets, achieving up to 36.45% improvement over baselines under severe data incompleteness. It demonstrates robustness to heterogeneous missing patterns in both IID and Non-IID settings, with theoretical analysis consistent with empirical results.  \n\n**Weaknesses**  \n1. The theoretical analysis relies on the assumption of μ-Lipschitz continuity of the model, failing to cover non-smooth data distributions. It also does not clarify the performance bounds in extreme missing scenarios (e.g., only single modality available), and the correlation between the dimension of embedding controls and the complexity of missing patterns lacks theoretical interpretation.  \n2. The dynamic selection of client-side embedding controls and server-side PFPT probabilistic clustering may lead to a surge in communication overhead in large-scale federated scenarios (e.g., thousands of clients). The weight coefficients of contrastive losses (λ, η) require manual tuning, and the cross-domain application lacks an automated adaptation mechanism.  \n3. The experiments generate binary missing matrices with fixed probabilities, failing to simulate continuous missing in time-series data or inter-modality dependencies (such as synchronous audio-visual missing). Additionally, it does not verify the adaptability to newly added modalities after training, limiting the expandability in dynamic IoT scenarios.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing our contribution and for the constructive feedback. We address all questions from the reviewer below.\n\n&nbsp;\n\n> **Q1. The theoretical analysis in the paper is based on the assumption of μ-Lipschitz continuity of the model, which may not fully cover real-world scenarios with complex and irregular data distributions. Additionally, the performance boundaries in extreme missing scenarios (such as when only a single modality is available) are not clearly demonstrated.**\n\n&nbsp;\n\n## Lipschitz assumption\n\nAlthough neural networks are highly nonlinear, the Lipschitz smoothness assumption - i.e., that the gradient of the loss is Lipschitz-continuous - is widely regarded as reasonable for theoretical analysis, especially in federated learning. This is because optimization typically occurs within a bounded parameter region, where the loss landscape is empirically smooth. As such, while global Lipschitz continuity may not hold, local smoothness is often sufficient and observed in practice. Consequently, this assumption is standard in many foundational FL analyses [1, 2, 3].\n\nHaving said that, we agree with the Reviewer that it’s beneficial to explore alternative, more relaxed continuity conditions in future work. One potential direction that we will pursue is to explore the semi-smoothness assumption [4], which is a result based on overparameterized neural network theory [5]. This assumption is proved to be held for deep learning in the centralized setting [5] and extended for FL, which could help our bound become more practical.\n\n[1] Tian Li, Anit Kumar Sahu et al., \"Federated Optimization in Heterogeneous Networks,\" MLSys 2020.\n\n[2] Hung T. Nguyen, Vikash Sehwag et al. “Fast-Convergent Federated Learning”, IEEE Transactions on Cognitive Communications and Networking, 2021\n\n[3] Farzin Haddadpour, Mohammad Mahdi Kamani et al. “Federated Learning with Compression: Unified Analysis and Sharp Guarantees”, AISTATS, 2021.\n\n[4] X. Li, Z. Song, R. Tao and G. Zhang, \"A Convergence Theory for Federated Average: Beyond Smoothness,\" 2022 IEEE International Conference on Big Data (Big Data), 2022\n\n[5] Zeyuan Allen-Zhu et al., “A Convergence Theory for Deep Learning via Over-Parameterization”, ICML 2019\n\n&nbsp;\n\n## Performance bound under extreme case\n\nWe would like to demonstrate our performance bound under extremely missing cases. Suppose there is only one modality given an input, i.e., $|S| = |M| - 1$, the RHS of the provided theorem becomes  $ \\mathcal{O} \\left( \\mu (|M| - 1) \\sqrt{ \\mathbb{E}\\_{x, \\mathcal{S}} \\left[ \\mathcal{L}\\_{ds}(x, \\mathcal{S}) \\right] + 2 \\log |M| } \\right) $, which depends on Lipschitz constant $\\mu$ and empirical loss $\\mathcal{L}\\_{ds}$, aligned with our observations and empirical results in the paper. If we can optimize empirical $\\mathbb{E}\\_{x, \\mathcal{S}} [ \\mathcal{L}\\_{ds} (x, \\mathcal{S}) ] = 0$, i.e., the performance bound now depends only on the smoothness constant $\\mu$ of the model (stated in line 214).\n\n&nbsp;\n\n## Empirical validation on extreme missing scenarios\n\nFurthermore, we presented empirical experiments of multiple extreme modality-missing scenarios, including the case when there is only one modality available for each client (80% of 5 modality are missed), in **Table 7 and Table 8, Appendix E**.\n\n&nbsp;\n\n> **Q2. The dynamic selection of client-side embedding controls and the server-side PFPT probabilistic clustering algorithm are likely to cause a significant increase in communication overhead in large-scale federated learning scenarios with numerous clients.**\n\nThe communication overhead over conventional FL methods (e.g., FedAvg)  is negligible in our framework. It also has a minimal effect on the computational cost overall. To see this, let the model size be $O(d)$ in FedAvg, **the conventional communication cost per iteration, per client is then $O(d)$**. Now, in our context, we need to additionally communicate the set of $p$ controllers encoding the data-missing profiles with size $O(d_p)$ each. As $p \\times d_p$ is relatively much smaller in comparison to $O(d)$ (e.g., $p \\times d_p / d$ ~ 0.1-0.2%), **the overall communication cost remains $O(d)$**. For a thorough comparison, we present numbers of bits in communication and show that the overhead is not significant as below:\n\n| Method   |  0.2/0.2  |  0.4/0.4  |  0.6/0.6  |  0.8/0.8  |\n|----------|:---------:|:---------:|:---------:|:---------:|\n| FedProx  | 125360192 | 125360192 | 125360192 | 125360192 |\n| PEPSY    | 125558930 | 125593828 | 125557128 | 125583219 |\n| Overhead |     0.16% |     0.19% |     0.16% |     0.18% |\n\n&nbsp;\n\n**Time complexity analysis**.\n\nWe also appreciate the Reviewer's suggestion on providing a more thorough complexity analysis. It is provided below in comparison to seminal FL work (FedAvg, FedProx) as their communication and compute costs are relatively least expensive. Due to limited space, we would like to refer the Reviewer to our response to Reviewer n9eK for details.\n\nAs shown in the analysis, **the added overhead cost grows linearly in the number of sent embedding controls from clients to the server, and is not substantial**. Furthermore, we note that PFPT's hyperparameters can also be tuned to keep server-side computation comparable to traditional FL without harming the performance significantly. As a result, PEPSY still outperform the baseline substantially even when the number of sent embeddings is reduced to match computation cost of FedAvg. Please refer to our response to the Reviewer n9eK for detailed analysis.\n\n&nbsp;\n\n> **Q3. The current experiment uses a binary missing matrix generated with fixed probabilities, which fails to fully simulate real-world continuous missing patterns (such as prolonged sensor failures) or inter-modality dependencies (where the absence of one modality may affect the validity of another).**\n\n\nWe agree with the Reviewer that validating the proposed method on more complex scenarios with continuous missing patterns (**inter-modality dependencies**) will broaden its applicability. We also want to emphasize that scenarios **with independent missing patterns (though simpler) also occur in numerous real-world scenarios** such as safety consideration [1], patient dropout [2] or simply high cost [3]. \n\nGiven this, we believe that the practical impact of our proposed work remains substantial. In our future work, we will evaluate our methods more extensively in the reviewer’s recommended settings. \nFor now, we have conducted additional experiments to evaluate our method on simulated scenarios with continuous missing patterns with inter-modality dependencies, following the reviewer’s suggestions. The results are positive as reported below. \n\nFor now, we have conducted additional experiments to evaluate our method on simulated scenarios with continuous missing patterns with **inter-modality dependencies**, following the reviewer’s suggestions. The results are positive as reported below.\n\nThese are data-missing scenarios where the absence of one modality can influence the validity of another. In particular, we first generate the binary missing matrix which initializes independent missing events. We then simulate inter-modality missing dependencies using the following rule: if modality $x_i$​ is missing and $i+2\\leq N$, then modality $x_{i+2}$​ is also set to missing. This enforces that certain modality combinations - such as (1, 3), (2, 4), ..., (N−2, N) - cannot co-occur in the same input. Using this rule and our original matrix generator, we generate a variety of test scenarios. The results are conducted on the EDF dataset (5 modalities) and the performances are presented below. \n\n|  Methods  | 0.4/0.2 | 0.4/0.6 | 0.4/1.0 |\n|:---------:|:-------:|:-------:|:-------:|\n|  FedProx  |  45.56  |  44.61  |  40.84  |\n|    MIFL   |  44.84  |   44.1  |  44.32  |\n|  FedInMM  |  40.96  |   40.5  |  40.56  |\n| FedMSplit |  44.78  |  43.93  |  43.81  |\n|   FedMAC  |  41.76  |   42.9  |  47.18  |\n|   PEPSY   |  53.45  |  54.91  |  53.23  |\n\n**This demonstrates that under inter-modality missing scenarios, PEPSY also consistently outperforms other baselines.**\nIntuitively, this means the more complex and sophisticated patterns of inter-modality missingness can also be distilled effectively into structured data-missing profiles of PEPSY. \nThis highlights the robustness and effectiveness of our developed method in practical settings. \n\n[1] Pedro Ramos-Cabrer, JPM Van Duynhoven, A Van der Toorn, and K Nicolay. 2004. MRI of hip prostheses using single-point methods: in vitro studies towards the artifact-free imaging of individuals with metal implants. Magnetic resonance imaging 22, 8 (2004), 1097–1103.\n\n[2] Yongsheng Pan, Mingxia Liu, Yong Xia, and Dinggang Shen. 2021. Disease-image specific Learning for Diagnosis-oriented Neuroimage Synthesis with Incomplete. Multi-Modality Data. IEEE Transactions on Pattern Analysis and Machine Intelligence (2021).\n\n[3] FM Ford and J Ford. 2000. Non-attendance for Social Security medical examination: patients who cannot afford to get better? Occupational medicine 50, 7 (2000), 504–507.\n\n&nbsp;\n\n> **Q4. The need for manual tuning of contrastive loss weights (λ, η) may severely limit the model's automated deployment and application across different domains.**\n\nWe thank the Reviewer for the suggestions and constructive feedback. We will explore that in future work where such elaborated tuning can help in scenarios with more sophisticated, inter-modality dependent missing patterns. In the current focus of this paper on independent missing patterns, these parameters are less sensitive according to our earlier investigation. For example, setting $\\lambda=0.1, \\eta=0.1$ works well for all experiments. \n\nGiven the above, we hope the Reviewer can take another look and consider upgrading the rating if our responses have addressed all concerns sufficiently. Otherwise, please let us know if you still have questions for us"}, {"review_id": "0GVWLwVCNU", "reviewer": "Reviewer_iy3T", "summary": "This paper proposes to dynamically reconstruct the representation of each client through embedding controls, enabling the global model to maintain stable performance when faced with multiple complex data missing patterns (missing modality、missing feature).", "questions": "Please refer to the weakness.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength：\n\n（1）This paper addresses the issues of modality missing and feature missing under multimodal federated learning.\n\n（2）Model missing data as structured vectors and enable their sharing within a federated framework.\n\n（3）This paper theoretically analyzed the impact of data missing on prediction output and proposed a clear upper bound of error, enhancing the credibility of the method.\n\nWeakness：\n\n（1）\tAlthough the paper uses PFPT to aggregate the embedding controls of each client, the order and semantics of the controllers learned by each client are not consistent. Does this paper perform semantic interpretability or consistency checks on the embeddings after clustering? If different clients represent different missing semantics, would blind aggregation compromise representation quality?\n\n（2）\tThe controller selection mechanism is based on cosine similarity matching. Does this similarity matching seek to find a specific controller that matches the current client's missing pattern?\n\n（3）\tWithout providing the code, it is incredible that performance can be maintained at 16% even with 80% missing data. Which module do you think can maintain model performance in a severely missing data scenario and provide the corresponding basis?\n\n（4）\tIn the study of multimodal federated learning scenarios, there has been some work on cross-modal and missing modality, but this paper does not provide a thorough review of related work. To my knowledge, there is also some FL+ missing modality work that considers the use of parameters from other clients.", "was_revised": true, "final_justification": "I appreciate the authors' efforts in addressing my concerns. I have carefully read the rebuttal. I will keep my score.", "rebuttal": "> **Q1. The order and semantics of the controllers learned by each client are not consistent?**\n\nWe would like to assure that using PFPT as a procedure in our method PEPSY, the order and semantic of the learned and aggregated controllers are consistent, as explained below.\n\nRegarding **order consistency**, by design, our client-side representation reconfiguration mechanism is invariant against controllers' order permutation. Our server-side clustering scheme (via PFPT) is also permutation invariant by design. PFPT identifies and aggregates controllers encoding similar data-missing profiles, rather than aggregating them blindly on a default order. \nThus, the aggregation outcome won’t be affected by the different orders in which local clients encode data-missing information into their controllers.\n\nRegarding **semantic consistency**, PEPSY, by design, ensures that clients with similar missing profiles will create similar queries defining a particular “missing direction” and hence, the selected controllers of these clients would be aligned with those “missing directions” via the cosine metric. \nAs PFPT uses a clustering-based aggregation mechanism which only aggregates controllers in the same cluster, the semantic of those controllers are aligned and can be aggregated without collapsing important information. As such, the aggregated controller in each cluster preserves semantic consistency.\n\nTo verify this, **one empirical indication of semantic consistency is the diversity across clusters of controllers**. \nIn particular, if controllers with similar semantics are merged appropriately into a centroid, there would be different separated centroids encoding different semantic representatives, leading to increasing diversity in the profile (over time). \nIn contrast, violated semantic consistency results in new collapsed embeddings, and fails to build well-separated clusters. This violation, however, is not the case according to Fig. 4b in our paper, which shows the t-SNE plots of the (learned) data-missing profile over 500 communication rounds on PTBXL. \nThe plots show that each embedding control follows a different trajectory with increasing diversity (across clusters), suggesting that the data-missing profile aligns semantically similar controllers over multiple rounds.\n\n&nbsp;\n\n> **Q2. The controller selection mechanism is based on cosine similarity matching. Does this similarity matching seek to find a specific controller that matches the current client's missing pattern?**\n\nSemantically, PEPSY ensures that clients with similar missing profiles will create similar queries defining a particular missing direction, leading to different directions for different missing patterns. Hence, the selected controllers of these clients will be aligned with those missing directions based on the directional similarity. This suggests that the cosine metric which measures directional similarities in feature space [1,2] are most suitable in this case.\n\nTo ablate the impact of using metric measuring directional similarity, we compare performances of our original design (cosine similarity) and other common distances such as (negative) MSE, MAE as presented below.\n\n|     Relevance Metric     | 0.2/0.2 | 0.8/0.8 |\n|:------------------------:|:-------:|:-------:|\n| Cosine similarity (ours) |  48.76  |  52.78  |\n|          (-) MAE         |  45.18  |  48.83  |\n|          (-) MSE         |  45.01  |  45.75  |\n\nThe results show that when we use metrics that are non-directional, the performance decreases. This is because in these cases, clients with similar missing profiles fail to construct a missing direction, hence making the aggregation ineffective.\n\n[1] Zifeng Wang et al., “Learning to Prompt in Continual Learning”. (CVPR 2022)\n\n[2] Xuefei Zhe et al., “Directional statistics-based deep metric learning for image classification and retrieval”, Pattern Recognition, Volume 93, 2019\n\n&nbsp;\n\n\n> **Q3. Without providing the code, it is incredible that performance can be maintained at 16% even with 80% missing data. Which module do you think can maintain model performance in a severely missing data scenario and provide the corresponding basis?**\n\nIn our PTBXL experiments with 80/80 missingness, the model achieves reasonable 75%, and 69% accuracy in IID and Non-IID settings, despite substantial data loss. \nTo elaborate, we note that [1] has shown that reasonable prediction can be made using data from 1/12 modalities. \nThis suggests that for this particular prediction task, essential predictive information can be found within a small fraction (says, 20%) of the dataset. \nFurthermore, the missing patterns are also independent across clients which means even at 80% data loss, the chance of an unrecoverable piece of information being lost across all clients is low. \nThus, with accurate and comprehensive data-missing profiling providing effective representation reconfiguration, our method is able to integrate the available data across clients effectively, recovering most of the key information and thus preserving reasonable performance despite substantial data loss.\n\nThe most important component that helps PEPSY to maintain high performance is the **data-missing profile and the reconfiguration signal**. The core insight is that our approach treats **missingness as structured, learnable information**, rather than as random noise. In other words, each incomplete instance is decomposed as missingness and raw data, both are scattered across clients, and need aggregation to improve overall performance of a given FL system.\n\nThe embedding controls form a data-missing profile that encodes each missing pattern as a distinct direction in latent space. These controllers act as functional priors, guiding the model in how to interpret incomplete inputs. Instead of relying solely on observed data, the model leverages this profile to reconfigure biased features to approximate full semantic representations. By explicitly modelling missingness, rather than imputing or ignoring it, the framework remains stable and effective, even under severe data loss.\n\nPEPSY consistently generates data-complete representations across varying missing patterns (Fig. 7). Without the missing-data profile (PEPSY-NP) or reconfiguration signal (PEPSY-NR), the model fails to recover missing information, leading to notably worse performance - especially under high missingness (Tab. 5).\n\n[1] Dongdong Zhang et al., “Interpretable deep learning for automatic diagnosis of 12-lead electrocardiogram”, iScience, 2021.\n\n&nbsp;\n\n> **Q4. This paper does not provide a thorough review of related work. To my knowledge, there is also some FL+ missing modality work that considers the use of parameters from other clients.**\n\nAs suggested, we extend our literature review which an be grouped into three main sub-categories.\n\n**Prototype Sharing for Missing Modalities** [1,4]. These methods share class- or modality-level prototypes across clients, which are used for regularization and, in some cases, to impute missing modalities. However, the signals they construct are not tailored to each input and fail to capture missing patterns at the instance level. In contrast, PEPSY enables input-specific reconfiguration through a controller selection mechanism and guided supervision. Moreover, these methods construct prototypes heuristically, resulting in less informative modality representations. Instead, we learn optimized missingness representations that produce more accurate, input-aware signals for reconstructing features tailored to each input. Our method is also the first that establishes a theoretical guarantee for heterogenous missing patterns across clients. Furthermore, these methods [1,4] focus only on modality-missing across clients and do not account for data-missingness. \n\n**Prompt-based Personalization Under Missing Modalities** [3]. This method uses collaborative prompts to personalize pretrained foundation models for clients with different available modalities. While prompts offer some flexibility, the method does not model missingness or adapt to absent data, lacking input-level awareness and useful prompt selection for reconstruction. In contrast, our approach learns a detailed missingness profile from clients using a set of embedding controls. These controls are selected to augment the input representation during inference, allowing fine-grained adaptation at both the instance and modality levels. Additionally, we focus on standard FL rather than prompt-based methods, as no existing foundation model in our health monitoring setting covers all modalities.\n\n**Prototype Sharing for General Heterogeneity** [2,5]. There also exist FL methods that share a similar goal of improving cross-client generalization through global prototype alignment. These approaches focus on unimodal settings and do not account for missing modalities. As a result, they are not well-suited for multimodal scenarios with missing inputs.. In contrast, our method explicitly models each client’s missing modalities using a learned controller set, which is integrated into the training loss. This enables learning and aggregation to be directly informed by the structure of missing data.\n\n[1] Huy Q. Le et al. “Cross-Modal Prototype based Multimodal Federated Learning under Severely Missing Modality”, CoRR 2024\n\n[2] Yue Tan et al. “FedProto: Federated Prototype Learning across Heterogeneous Clients”. AAAI 2022.\n\n[3] Wenli Li et al., “MFLCP: Personalized Multimodal Federated Learning via Collaborative Prompting with Missing Modalities”, ICMR 2025.\n\n[4] Guangyin Bao et al. “Multimodal Federated Learning with Missing Modality via Prototype Mask and Contrast”, arxiv 2024\n\n[5] Jianqing Zhang et al. “FedTGP: Trainable Global Prototypes with Adaptive-Margin-Enhanced Contrastive Learning for Data and Model Heterogeneity in Federated Learning”, AAAI 2024\n\nWe will include the above literature review in our revision."}, {"review_id": "NlGUg2d8gC", "reviewer": "Reviewer_n9eK", "summary": "The paper studies multimodal federated learning under both missing modalities and missing features. The authors proposed a reconfiguration based framework with a data-missing embedding profile. The missing data will be carefully reconstructed based on the profile, which will be then fused with the available data. There is also a theoretical analysis on the convergence behavior of the model under missing data. Comprehensive experiments agains SOTA baselines show accuracy improvements under severe data incompleteness.", "questions": "* What is the overhead of the method compared to existing baselines?\n* How will k, which controls the number of relevant controls selected for each instance, affect the performance?", "limitations": "Would be good to discuss the limitations in the main paper instead in the appendix", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strength:\n* The paper is organized clearly and easy to follow\n* The proposed idea of using learnable client-side embedding controls makes sense and is interesting\n* Comprehensive experiments are conducted under different missing statistics.\n\nWeakness:\n* The overhead of the method compared to existing baselines are not analyzed\n* The sensitivity of parameter k, which controls the size of the neighborhood in embedding control, is not analyzed", "was_revised": true, "final_justification": "The authors have provided the results to address my concerns. I would recommend accepting the paper, as it is a solid paper that addresses a real-world problem and has conducted comprehensive experiments.", "rebuttal": "We thank the Reviewer for recognizing our contribution and for the constructive feedback. We address all questions from the reviewer below.\n> **Q1. The overhead of the method compared to existing baselines are not analyzed**\n\nOur method does incur client-side communication overhead and server-side processing overhead. However, such **overhead is not substantial**. We analyze these overheads via a detailed time complexity analysis below.\n\nWe start by introducing the time complexity of traditional FL algorithms (FedAvg, FedProx) as a baseline to analyze the time complexity of PEPSY. \n\n## Notations and Definitions\n\nLet:\n* $d$: feature extractor size\n* $\\tau$: number of embedding controls in local data-missing profile.\n* $d_p$: embedding control dimensionality.\n* $d_k$: key vector dimensionality\n* $\\kappa$: number of embedding controls selected per query (small constant)\n* $E$: local epochs\n* $B$: batch size\n* $n_k$: local data size\n* $p$: number of embedding controls each client sends per round\n* $M$: number of optimizing steps in PFPT\n\n| Component            | Traditional FL                                          | PEPSY                                                                                               |\n|----------------------|---------------------------------------------------------|-----------------------------------------------------------------------------------------------------|\n| Local Computation    | $\\mathcal{O}\\left(E \\cdot \\frac{n\\_k}{B} \\cdot d\\right)$ | $\\mathcal{O}\\left( \\frac{n\\_k}{B} \\cdot E \\left[ (d + m \\cdot d\\_p) + \\tau \\cdot d\\_k \\right] \\right)$ |\n| Client Communication | $\\mathcal{O}(d)$                                        | $\\mathcal{O}(d + p d_p)$                                                                            |\n| Server Aggregation   | $\\mathcal{O}(K d)$                                      | $\\mathcal{O}( Kd + MK^2p^2d_p^2 )$                                                                  |\n\n## Detailed Analysis\n\n**Traditional FL**. Each client updates its local parameters over $E$ iterations, with batch size of $B$ using a model of size $d$. This cost:\n\n$ \\mathcal{T}_{local} = \\mathcal{O}(E \\cdot \\frac{n_k}{B} \\cdot d)$\n\nSubsequently, the modal parameters of all clients are sent to the server costing:\n\n$\\mathcal{T}_{com} = \\mathcal{O}(d)$\n\nOn the server side, all parameters of $K$ clients are combined, typically using variants of weighted average leading to aggregation time cost:\n\n$\\mathcal{T}_{server} = \\mathcal{O}(K \\cdot d)$\n\n**PEPSY Modification**. \nIn PEPSY, the added cost comes from each client's data-missing profile and the PFPT-based clustering. The time complexities are as follows:\n\n*Embedding Controls Selection*.  \n\nEach client computes a key vector $q \\in \\mathbb{R}^{d_k}$ per batch, compares it with $\\tau$ controls, and selects top-$\\kappa$ controls. If done once per batch, this adds $\\mathcal{O}\\left( \\frac{n_k}{B} \\cdot p \\cdot d_k \\right)$ per round. The selected controls are injected into the model and used during both forward and backward passes. This adds gradient updates with cost $\\mathcal{O}( d + \\kappa \\cdot d_p )$. Over $E \\cdot \\frac{n_k}{B}$ steps, the total local computation cost is:\n$\\mathcal{T}_{\\text{local}} = \\mathcal{O}\\left( \\frac{n_k}{B} \\cdot E \\cdot \\left[ (d + m \\cdot d_p) + \\tau \\cdot d_k \\right] \\right)$\n\n*Communication Cost*.\n\nClients also send their $p$ with $\\ p \\leq \\tau$ selected control embeddings (a subset of data-missing profile), adding to the model upload cost:\n$\\mathcal{T}_{\\text{com}} = \\mathcal{O}(d + p \\cdot d_p)$\n\n*Server Aggregation and Clustering*.\n\nModel aggregation stays at $\\mathcal{O}(K d)$, but PFPT adds overhead from bi-level optimization (over $M$ iterations) and Hungarian matching. Clustering over $Kp$ points add more time complexity to the cost:\n$\\mathcal{T}_{\\text{server}} = \\mathcal{O}( K d + M K^2 p^2 d_p^2 )$\n\nAssuming a fixed model architecture, standard FL methods (e.g., FedAvg, FedProx) incur a cost of $\\mathcal{O}(K d + K \\tau d_p)$. \nTo match this, PFPT can achieve similar efficiency by setting $p = \\mathcal{O}\\left( \\sqrt{ \\frac{\\tau}{M K d_p} } \\right)$, given fixed $M$, $K$, and $d_p$. We have verified this via the following ablation studies on $p$. \n\n**As reported below, our method still outperforms the baseline substantially even when $p$ is reduced to match the computation cost of FedAvg.**\nThis is run on the EDF dataset, with 0.2/0.2 missingness. For each client, we only take $p$ most selected embedding controls to send to the server per round.\n\n\n|      Method      | Accuracy (%) |\n|:----------------:|:------------:|\n|      FedProx     |     43.24    |\n|       MIFL       |     43.18    |\n|      FedInMM     |     40.56    |\n|     FedMSplit    |     45.18    |\n|      FedMAC      |     49.8     |\n|   PEPSY (p = 5)  |     50.77    |\n|  PEPSY (p = 10)  |     50.45    |\n|  PEPSY (p = 20)  |     51.51    |\n| PEPSY (no limit) |     56.36    |\n\n## Empirical estimation\n\nWe compared the computational overhead of PEPSY with existing baselines in different $p_m/p_s$ scenarios and found that the additional cost in PEPSY is primarily incurred during training, regardless of the missingness scenario. This aligns with the time complexity analysis, as the PFPT-based clustering algorithm requires more time for clustering. \nIn contrast, **PEPSY's inference time and GPU usage remain comparable to other methods, while still delivering superior performance**. This is because the data-missing profile is relatively small compared to the model size, adding minimal overhead to each forward pass.\n\n| Method    | Metric                      | 0.2/0.2 | 0.2/0.4 | 0.2/0.6 | 0.2/0.8 | 0.2/1.0 |\n|-----------|-----------------------------|:-------:|:-------:|:-------:|:-------:|:-------:|\n| FedProx   | Training time per round (s) |  50.21  |  50.43  |  50.41  |   49.8  |  49.77  |\n|           | Inference time (s)          |   3.56  |   3.57  |   3.6   |   3.74  |   3.59  |\n|           | GPU for training (GB)       |   2.72  |   2.72  |   2.72  |   2.72  |   2.72  |\n| MIFL      | Training time per round (s) |  94.11  |  94.04  |  93.92  |  92.98  |  93.34  |\n|           | Inference time (s)          |   4.11  |   4.12  |   4.13  |   4.1   |   4.16  |\n|           | GPU for training (GB)       |   3.26  |   3.26  |   3.26  |   3.26  |   3.26  |\n| FedInMM   | Training time per round (s) |  100.23 |  97.71  |  97.95  |  99.28  |  96.44  |\n|           | Inference time (s)          |   4.89  |   4.86  |   4.83  |   4.95  |   4.89  |\n|           | GPU for training (GB)       |   2.55  |   2.55  |   2.55  |   2.55  |   2.55  |\n| FedMSplit | Training time per round (s) |  86.34  |  86.63  |  86.56  |  86.67  |  86.18  |\n|           | Inference time (s)          |   3.59  |   3.58  |   3.6   |   3.6   |   3.6   |\n|           | GPU for training (GB)       |   3.21  |   3.21  |   3.21  |   3.21  |   3.21  |\n| FedMAC    | Training time per round (s) |  51.77  |  51.11  |  51.07  |  51.19  |  51.21  |\n|           | Inference time (s)          |   4.56  |   4.98  |   4.69  |   4.69  |   4.88  |\n|           | GPU for training (GB)       |   1.99  |   1.99  |   1.99  |   1.99  |   1.99  |\n| PEPSY     | Training time per round (s) |  141.12 |  153.95 |  137.66 |  140.12 |  146.48 |\n|           | Inference time (s)          |   4.69  |   4.99  |   4.9   |   4.73  |   4.87  |\n|           | GPU for training (GB)       |   2.61  |   2.63  |   2.15  |   2.86  |   2.8   |\n\n\n&nbsp;\n\n> **Q2. How will $\\kappa$ , which controls the number of relevant controls selected for each instance, affect the performance?**\n\n\nWe provide below additional ablation studies on evaluating the performance of our method with different values of $\\kappa$ on the EDF dataset.\n\n|   $\\kappa$   |   1   |   3  |   5   |   10  |   15  |   20  |\n|:------------:|:-----:|:----:|:-----:|:-----:|:-----:|:-----:|\n| Accuracy (%) | 47.06 | 48.7 | 51.17 | 52.71 | 55.11 | 52.03 |\n\nThe reported results generically show that increasing $\\kappa$ tends to improve overall performance as it allows more capacity and flexibility for the data-missing profile to learn local missing patterns. However, beyond some points (e.g., $\\kappa = 15$), the information distilled from missing patterns to the missing profile becomes too fragmented (due to selecting too many controllers), which decreases the performance."}, {"review_id": "JmLiH24i8V", "reviewer": "Reviewer_iEDv", "summary": "This paper addresses the challenge of multimodal federated learning (MMFL) in the presence of incomplete and heterogeneous data, where clients may have access to different subsets of modalities and suffer from partial feature missingness within each modality. The authors propose PEPSY, a novel federated learning framework that introduces learnable client-specific embedding controls to capture and communicate each client’s unique data-missing patterns. These embeddings serve as reconfiguration signals that align the global model with each client's local context, enabling more effective and robust representation learning.\n\nThe framework operates by:\n1.\tExtracting modality-specific and data-specific features on the client side.\n2.\tUsing a query-key mechanism to select relevant embeddings from a shared pool that reflect the missing patterns.\n3.\tAggregating embedding controls across clients via a probabilistic clustering-based alignment on the server.\n\nTheoretical analysis provides performance bounds showing that PEPSY’s design minimizes prediction error due to missing modalities. Empirical evaluations on PTBXL and Sleep-EDF datasets under IID and Non-IID settings demonstrate significant improvements over baselines, achieving up to 36.45% accuracy gains in high missingness scenarios.", "questions": "Before addressing specific questions, I strongly encourage the authors to review the **Clarity** section under **Weaknesses**, particularly issues raised in Sections 2.1–2.3. Several definitions, notational choices, and design rationales lack sufficient explanation. Addressing these concerns would significantly improve the comprehension of the paper and may positively influence the overall assessment.\n\nBelow are the key questions that, if clarified, could strengthen the paper further:\n1. **Clarify the Construction and Scope of the Data-Missing Profile ($\\Psi$)**  \n   How is the data-missing profile $\\Psi$ initialized and updated during training? Is there a single global profile, or does each client maintain a local version? Why is there no client index in the notation if the profiles encode client-specific missingness patterns?\n\n2. **Explain and Justify the Optimization Formulation (Eq. 3)**  \n   Why is the original federated loss in Eq. (1) replaced with a difference involving the utility term $u_k(\\theta, \\Psi)$? What does this subtraction represent, and how should it be interpreted? \n\n3. **Clarify the Use of Contrastive Learning for Reconfiguration (Section 2.2.3)**  \n   Why is contrastive loss suitable for aligning partial and full-modality representations? What exactly are the positive and negative sample pairs used during training?\n\n4. **Describe PFPT and Its Role in Server Aggregation (Section 2.3)**  \n   What are the main characteristics of PFPT, and why is it an appropriate choice for aggregating the data-missing profile across clients?\n\n5. **Discuss the Lipschitz Continuity Assumption in Theorem 3.1**  \n   Given that neural networks rarely satisfy strict Lipschitz conditions, how practical is this assumption? Can the bound be extended or relaxed under more realistic constraints?\n\n6. **Clarify the Definition of \"Modality\" Used in the Dataset Description**  \n   The paper states that PTBXL has 12 modalities and Sleep-EDF has 5, but these datasets are generally not considered to contain that many distinct modalities under standard definitions (e.g., text, audio, image). Could the authors clarify what they count as a modality in this context? \n\n7. **Provide an Analysis of Computational Complexity**  \n   The proposed method introduces multiple components—embedding control pools, query-key relevance selection, contrastive regularization, and PFPT-based aggregation. Can the authors provide an estimate or breakdown of computational and communication overhead introduced by each component?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## **Strengths**\n\n### **Quality**\n- The method addresses an important and realistic challenge in multimodal federated learning (MMFL): handling both *missing modalities across clients* and *missing features within modalities*. The proposed PEPSY framework is designed to tackle this dual challenge and is evaluated through solid empirical experiments.\n\n### **Clarity**\n- The paper clearly motivates the real-world relevance of MMFL under partial observability and articulates the limitations of prior work. The distinction between different types of \"missing-ness\" (modality-level and feature-level) is helpful for framing the contribution.\n\n### **Significance**\n- The addressed problem is important for real-world applications involving distributed sensing and privacy-preserving learning, particularly in domains like healthcare and IoT. The strong empirical gains and general design framework make this work a promising direction for advancing robust and flexible MMFL.\n\n### **Originality**\n- While the paper proposes a novel solution, there is some conceptual proximity to existing work:\n  - **Personalized Federated Learning (PFL)**: Many PFL approaches use local embeddings or client-specific signals to personalize global models. PEPSY similarly leverages client-side latent information via embedding control pools.\n  - **Domain Adaptation**: The idea of learning latent codes to adapt shared models to local contexts mirrors approaches in domain adaptation, especially under distribution shifts.\n---\n\n## **Weaknesses**\n\n### **Clarity**\n\nWhile the paper presents a compelling high-level idea, the technical exposition—especially in **Section 2**—is difficult to follow. Several core concepts and mathematical formulations lack clarity or motivation, making it hard to understand the method in depth. Specific issues include:\n\n- **Data-Missing Profile Ambiguity (Section 2.1–2.2.2)**  \n  It is unclear how the data-missing profile $\\Psi$ is computed or updated, whether it is global or client-specific, and how it's initialized. The absence of a client index in the notation adds confusion, especially given the claim that the profile reflects local missingness patterns.\n\n- **Equation Transitions (Section 2.1)**  \n  The transition from Eq. (1) to Eq. (3) introduces a new utility term $u_k$ without sufficient justification. Its role in the optimization problem, and why the objective is now a difference of two terms, is not explained intuitively or mathematically.\n\n- **Inconsistent Notation (Section 2.1)**  \n  In the expression for $u_k(\\theta, \\Psi)$, the right-hand side omits $\\theta$, though it appears to depend on model parameters via the relevance function. This creates notational confusion in a key part of the formulation.\n\n- **Unclear Objectives (Section 2.1)**  \n  The claim that minimizing $\\ell_k(\\theta, \\Psi)$ helps extract data-missing features, and that maximizing $u_k(\\theta, \\Psi)$ helps $\\Psi$ adapt to local context, is not justified or supported by theory or experiments.\n\n- **Vague Terminology (Section 2.1–2.2.2)**  \n  Phrases like “reconfigured by $\\Psi$” or “diluting learned data-missing profile” are conceptually vague and lack clear operational definitions, making the method harder to interpret.\n\n- **Embedding Controls Design (Section 2.2.2)**  \n  The identity function is used as the key function $k(\\cdot)$ in a query-key matching setup, with no justification for its effectiveness. Also, the decision to limit relevance to only $\\kappa$ embeddings is explained in abstract terms without empirical motivation or ablation.\n\n- **Contrastive Learning Justification (Section 2.2.3)**  \n  The use of contrastive loss for reconfiguration regularization is introduced abruptly. It is unclear why this choice is suitable for ensuring “data completeness,” and the sampling of positives and negatives is not specified.\n\n- **Definition of $h_{di}$ (Section 2.2.1)**  \n  The modality-specific feature $h_{di}$ is mentioned as a key component in representation construction, but its computation is unspecified. There’s no mention of what neural layers produce it or how it fits into the overall architecture.\n\n- **Server Aggregation with PFPT (Section 2.3)**  \n  The server uses PFPT (probabilistic federated prompt tuning) for aggregating data-missing profiles, but this method is not described. Readers unfamiliar with PFPT are left without context on its mechanism, benefits, or why it was chosen over alternatives.\n\n### **Quality**\n\n- **Computational Complexity**  \n  The method comprises several sophisticated components—embedding control pools, query-key matching, contrastive and alignment losses, and probabilistic clustering (PFPT). These likely incur nontrivial computational and communication overhead, but the paper does not discuss the system cost or efficiency.\n\n- **Limited Dataset Variety**  \n  Although performance on PTBXL and Sleep-EDF is promising, both datasets are from biosignal/health domains. Broader validation on more diverse multimodal settings (e.g., video+audio, language+vision) would better support the generality of the approach.\n\n### **Significance**\n\n- **Strong Theoretical Assumptions (Section 3)**  \n  Theorem 3.1 assumes that the model is $\\mu$-Lipschitz continuous—a strong condition not typically satisfied by deep networks. The paper does not discuss whether this assumption is realistic or how sensitive the bound is to its violation, limiting the practical applicability of the theoretical result.\n\n### **Originality**\n\n- **Conceptual Overlap with Prior Work (Section 2.1–2.2.2)**  \n  While PEPSY presents a technically novel mechanism, its conceptual structure shares similarities with:\n  - **Personalized Federated Learning (PFL):** where client-specific embeddings or local statistics are used to personalize global models.\n  - **Domain Adaptation:** where shared models are adapted using domain-specific latent codes.\n\n  To more clearly distinguish the contribution, the authors could:\n  - Emphasize the novelty of *cross-client collaboration via clustering* of embedding profiles.\n  - Provide explicit empirical and theoretical comparisons to relevant PFL and domain adaptation baselines.", "was_revised": true, "final_justification": "After a few back-and-forth rounds, I was satisfied with the author's response during the rebuttal.", "rebuttal": "We thank the reviewer for the detailed questions. We summarize the overall conceptual workflow of PEPSY and address the reviewer’s questions below. \n\n**Client-side**: The server sends the global model parameters and data-missing profiles encoded within a set of embedding controls to each client. Given a multimodal input, each client computes features of its data-missing patterns (Sec. 2.2.1) and uses a key-query mechanism to select relevant controls that help generate data-complete representations (Sec. 2.2.2). This selection is guided by the reconfiguration signal (Sec. 2.2.3), and the resulting features are fused via an attention module to produce the final prediction.\n\n**Server-side**: Each client maintains its own data-missing profile as a set of embedding controls encoding various missing patterns. To align embedding controls encoding similar patterns, we use PFPT - a non-parametric clustering method - to cluster and aggregate them across clients. The non-parametric nature of PFPT helps the global data-missing profile (i.e., global embedding clusters) grow with the diversity and misalignment across clients, making it suitable for the varying missingness settings.\n\n# Q1. Data-missing profile clarity\nIn PEPSY, the server maintains a global data-missing profile, initialized as a small set of random embedding controls, with each client holding a local copy. This profile is refined through iterative client-side training and server-side clustering. After training, each client updates its local controls and sends them to the server. To resolve cross-client misalignment, the server applies a non-parametric clustering algorithm to group controls encoding similar patterns and compute global controls as the corresponding cluster centroids, which are then broadcast to clients for the next round.\n\nIn Eq. (2), the data-missing profile does not include a client index because it denotes the global data-missing profile that we aim to learn. Eq. (2) denotes the reconfigured version of the original model given the learned data-missing profile (via FL). This data-missing profile will be learned via solving Eq. (3) which characterizes the distributed computation structure of FL.\n\n# Q2. Explain new FL formulation in Eq (3)\nEq. (1) characterizes the standard FL formulation without accounting for missing-data patterns. Eq. (3) augments it to **explicitly model and learn the missing-data profile** encoded with the set of embedding controls $\\Psi$. The subtraction of $u_k(\\theta, \\Psi) $ from the prediction loss $ \\ell_k( \\theta,\\Psi) $ represents a regularization of local learning  $ \\ell_k( \\theta, \\Psi) $ with a missing-aware signal $u_k(\\theta, \\Psi)$. This signal guides the client in reconfiguring its representations based on missing patterns. Minimizing the regularization loss $-u_k(\\theta, \\Psi) $ updates the embedding controls to reflect the client’s data context, which are then shared and aggregated by the server for the next round.\n\n# Q3. Contrastive loss for aligning partial and full modality features\nWe do not use contrastive loss to align partial and full modality features, but to ensure that **reconstructed modalities of the same input map to similar coordinates in a semantic space**. Specifically, we apply the same missing-aware reconfiguration mechanism to generate a reconstructed representation $w\\_{di}$ for each modality $i$ of input $x\\_{di}$, regardless of its missing pattern. For the same input, we encourage all reconstructed modalities to align in the semantic space, while modalities from different inputs remain distant, ensuring semantic consistency within each instance. Let $ \\hat{w}\\_{di}$ the semantic coordinates of $w_{di}$ extracted via a neural projector. The loss uses positive pairs $(\\hat{w}\\_{di}, \\hat{w}\\_{dj}) $ from the same input and negative pairs $ (\\hat{w}\\_{d'i}, \\hat{w}\\_{dj}) $ from different inputs.\n\n# Q4. Main characteristics of PFPT\nA key feature of PFPT is its non-parametric clustering nature, which allows the number of clusters to grow with the data complexity. Since PEPSY represents each local data-missing profile as a set of embedding controls, scaling with missingness, PFPT is well-suited for constructing these profiles under varying missing patterns. \n\nPFPT is a prompt aggregation method that uses non-parametric clustering. It treats each local model as a set of prompts, which are shared with the server. The server merges these into global prompt sets by clustering similar prompts. The number of clusters adjusts automatically based on the complexity of the data, allowing the model to scale with data diversity.\n\nOur data-missing profile, a set of embedding controls, shares similarities with prompt sets. In PEPSY, each client updates the global profile using its private data, producing a locally augmented set of controllers. The size and complexity of these sets grow with the client's missingness level. The server's goal is to (1) match similar augmented controllers and (2) let the global profile grow with the complexity of data-missing patterns, making PFPT a natural fit.\n\nHowever, we emphasize that in the original work, PFPT performs clustering on the prompt set in foundation FL to handle different data distribution and label skewness patterns, while PEPSY aligns different missing profiles across clients.  **For the effectiveness of PFPT-based clustering, we refer the reviewer to Fig. 4b in our paper, showing its trajectory and convergence.**\n\n# Q5. Assumption in theoretical analysis\nDue to limited space, we refer the reviewer to our response to Reviewer nfrN (Q1). \n\n# Q6. Modality clarity in experiments\nIn our setting, a **modality refers to a sensor signal**, as used in both the EDF and PTB-XL datasets. Each input comprises signals from multiple sensors, where each sensor captures a distinct aspect of the patient's condition. Since these signals vary in distribution and scale, they are treated as separate modalities. This setup is consistent with prior works [1].\n\n[1] Nguyen et al., FedMAC: Tackling Partial-Modality Missing in Federated Learning with Cross-Modal Aggregation and Contrastive Regularization, NCA 2024\n\n# Q7, Computation cost\nOur method does incur client-side communication overhead and server-side processing overhead. However, such **overhead is not substantial**. We also note that the parameter configuration of PFPT-based clustering can be tuned to match the conventional cost of FedAvg while preserving performance advantage. Due to limited space, please refer to our response to Reviewer n9eK for detailed explanation of the above.\n\n# Q8. Experiments on other multimodal domains\nFollowing the reviewer’s suggestion, we added an experiment in an image-sensor multimodal setting to demonstrate the method’s generality. Here, a signal-based modality was converted into an image representing its fluctuation, and the original signal was withheld. Results are shown below.\n| pm/ps   | FedProx | MIFL  | FedInMM | FedMSplit | FedMAC | PEPSY |\n|---------|---------|-------|---------|-----------|--------|-------|\n| 0.2/0.2 | 44.32   | 44.89 | 40.22   | 43.93     | 39.7   | 44.95 |\n| 0.8/0.8 | 41.76   | 43.07 | 40.21   | 42.56     | 38.16  | 44.78 |\n\n**The above reported results confirm that PEPSY still outperforms the existing baselines in both slightly and extremely missing scenarios (0.2/0.2 and 0.8/0.8)**. Similarly, we observe that PEPSY is more consistent compared to other baselines with minimal performance gap under increasing missingness.\n\n# Q9. Discussing related works on personalized FL and domain adaptation.\nWhile personalized FL and domain adaptation methods tailor models to individual clients or domains, PEPSY takes a different approach, focusing on the global FL setting by building a shared profile of data-missing patterns across all clients.\n\nPersonalized FL tailors models to local data distributions, producing one model per client and evaluating performance across them [1,2]. Domain adaptation transfers knowledge from a source to a target domain using alignment techniques [3,4], but typically assumes centralized settings and complete data. Both approaches address distribution shifts, not missing data, and it remains unclear how they could be adapted to effectively handle modality-level missingness.\n\nIn contrast, PEPSY is designed to recognize and align diverse missing data patterns across clients into a holistic data-missing profile. This is achieved via aggregating embedding controls from clients - these controls are designed to capture missing-data patterns, not local model behavior. Unlike prior work, these embeddings are not stored on-device nor used for local fine-tuning. Instead, they are regularized (as in Sec. 2) to help reconfigure missing features into complete representations, addressing the system-level bias issue outlined in Sec. 1.\n\nAdditionally, PEPSY introduces a non-parametric PFPT-based clustering algorithm designed for the highly heterogeneous nature of our FL setup. Clients have diverse and misaligned missing patterns, often with varying embedding orders, making a fixed number of clusters too restrictive. The non-parametric design allows flexible grouping and better captures the structural diversity of missing-data profiles. In contrast, existing PFL methods [1,2] do not address this misalignment, which can lead to feature collapse. **To our knowledge, PEPSY is the first to explicitly model missing patterns across clients and aggregate them into a system-level data-missing profile.**\n\n[1] Marfoq, O. et al. Personalized Federated Learning through Local Memorization. ICML 2022\n\n[2] Tan et al. Federated learning from pre-trained models: a contrastive learning approach. Neurips 2022\n\n[3] Sun et al. PartialFed: Cross-Domain Personalized Federated Learning via Partial Initialization. Neurips 2021\n\n[4] Hu et al. Learn to Preserve and Diversify: Parameter-Efficient Group with Orthogonal Regularization for Domain Generalization. ECCV 2024"}]

Author rebuttal:

Meta-review: After the discussion phase all reviewers agree that the submission is of high enough quality for publication at NeurIPS. As Area Chair, I agree and recommend acceptance.
Decision: Accept (poster)