Federated Learning for Time-Series Healthcare Sensing with Incomplete Modalities

## Abstract

Abstract. Many mobile sensing applications utilize data from various modalities, including motion and physiological sensors in mobile and wearable devices. Federated Learning (FL) is particularly suitable for these applications thanks to its privacy-preserving feature. However, challenges such as limited battery life, poor network conditions, and sensor malfunctions can restrict the use of all available modalities for local model training. Additionally, existing multimodal FL systems also struggle with scalability and efficiency as the number of modality sources increases. To address these issues, we introduce FLISM , a framework designed to enable multimodal FL with incomplete modalities. FLISM leverages simulation technique to learn robust representations that can handle missing modalities and transfers model knowledge across clients with varying set of modalities. The evaluation results using three real-world datasets and simulations demonstrate FLISM ’s effective balance between model performance and system efficiency. It shows an average improvement of .067 in F1-score, while also reducing communication (2.69 × \times × faster) and computational (2.28 × \times × more efficient) overheads compared to existing methods addressing incomplete modalities. Moreover, in simulated scenarios involving tasks with a larger number of modalities, FLISM achieves a significant speedup of 3.23 × \times × ∼ similar-to \sim ∼ 85.10 × \times × in communication and 3.73 × \times × ∼ similar-to \sim ∼ 32.29 × \times × in computational efficiency.

## 1. Introduction

Real-world sensing applications often employ multimodal systems that integrate various sensors to enhance the application performance and functionality (Ramachandram and Taylor, ). For instance, autonomous vehicles combine LiDAR, radar, and cameras to detect objects accurately (Liu et al., ; Li et al., ), while mobile health devices use physiological and motion sensors to monitor activities, emotions (Park et al., ) and stress levels (Schmidt et al., ). Similarly, dietary monitoring tools analyze data from wearables to identify eating episodes (Bahador et al., ; Merck et al., ).
However, the transmission of raw data for these applications raises privacy concerns. Federated Learning (FL), a privacy-centric machine learning approach, addresses these issues by allowing local model training without sending raw data to a central server.

Multimodal sensing applications face another challenge: they often assume that all modalities are consistently available and functional (Bijelic et al., ; Deldari et al., ; Jain et al., ). Yet, real-world obstacles such as device malfunctions (Vaizman et al., ), poor network conditions (Feng and Narayanan, ), or privacy concerns (Li et al., ) frequently result in missing modalities. Traditional centralized machine learning approaches tackle these challenges with statistical methods (Yu et al., ) and imputation using deep learning (Zhao et al., ; Zhang et al., ). However, the privacy-preserving nature of FL restricts the direct exchange of raw data between clients and a server, thus limiting the application of these methods in an FL context.

The FL community is adapting to handle challenges in multimodal FL, including those involving incomplete modalities.
Recent strategies primarily employ *late fusion*, where the server aggregates model weights for each modality separately (Ouyang et al., ; Feng et al., ; Zheng et al., ). These works often focus on tasks with limited input modalities, typically two like LiDAR and radar (Zheng et al., ) or three such as audio, depth, and radar (Ouyang et al., ). However, many multimodal sensing applications require integrating more than three modalities. The late fusion method, which trains separate unimodal models for each modality, leads to scalability challenges in communication and computation, increasing overhead as the number of modality sources grows.

To overcome scalability and efficiency challenges in multimodal sensing with incomplete modalities, we introduce *FLISM*, an FL framework that adopts an *early fusion*-based approach. In contrast to late fusion, early fusion allows clients to train a single unified model, significantly enhancing training efficiency, especially for multimodal data with 1D modality sources. The key idea of *FLISM* is that within a multimodal FL system, certain clients possess modalities that are more crucial and beneficial to application performance. These clients can simulate a wider range of modality combinations with missing modalities, facilitating the transfer of robust representations to clients with limited capabilities. *FLISM* consists of three components: missing-modality-robust learning, client role assignment, and cross-client-knowledge transfer. Initially, all multimodal clients simulate a range of missing modality scenarios, training the model to be robust under various missing-modality conditions. Clients are then assigned either as facilitators, who guide and support others through simulation generation, or learners, who primarily gain from this support. Finally, cross-client knowledge transfer aids the transfer of simulated knowledge to learners, enhancing robustness against missing modalities.

We implemented *FLISM* and evaluated its performance using three real-world publicly available multimodal sensing datasets. We also conducted experiments with simulated data to assess the scalability of *FLISM* compared with existing late fusion baselines tackling incomplete modalities in multimodal FL system. Our results indicate that:

- •
*FLISM* outperforms late fusion methods, achieving an average increase of .067 in F1-score, along with the reductions in both communication and computational overhead: on average, 2.69$\times$ faster and 2.28$\times$ more efficient, respectively.
- •
In simulated scenarios involving tasks with a larger number of modalities, *FLISM* significantly reduces communication costs by a factor of 3.23$\times$$\sim$85.10$\times$ and computation costs by a factor of 3.73$\times$$\sim$32.29$\times$.
- •
We performed a detailed component-wise analysis to validate the effectiveness of each element within *FLISM*, demonstrating their essential contributions to enhancing model performance.

## 2. Motivation

We present preliminary results that illustrate the impact of incomplete modalities in sensing-based applications in FL settings.
Our analysis reveals that current approaches become inefficient and scale poorly as the number of modalities increases in multimodal FL sensing applications.

### 2.1. Performance Degradation

Figure: Figure 1. The model performance decreases as the ratio of clients with incomplete modalities increases in RealWorld-HAR (left) and WESAD (right) datasets.
Refer to caption: extracted/5606462/figures/motivation/acc_drop_rw.png

Most multimodal sensing applications presume the consistent availability and functionality of all modality sources (Bijelic et al., ; Deldari et al., ; Jain et al., ). Yet, in practical scenarios, various factors such as device malfunctions (Vaizman et al., ), suboptimal network conditions (Feng and Narayanan, ), or user privacy concerns (Li et al., ) often result in the unavailability of certain modalities.

The standard method of imputing missing modalities using raw data statistics (Zhang, ; Van Buuren, ) is not feasible in FL environments as the server is unable to access clients’ raw data. Consequently, zero-imputation becomes the only available option (Van Buuren, ) but it causes performance degradation. To evaluate the performance with zero-imputation in the absence of modalities, we conducted experiments on two representative mobile sensing datasets, RealWorld HAR (Sztyler and Stuckenschmidt, ) and WESAD (Schmidt et al., ), each featuring ten modalities, such as accelerometer, gyrosocope, temperature, electrocardiogram, electrodermal activity. (See §4.1 for detailed dataset description).
We simulated conditions where $p$% of clients possess incomplete modality data.
Specifically, we allowed $p$% of clients (with $40\leq p\leq 80$) to randomly omit up to $M-1$ modalities from their training data, where $M$ represents the total number of available modalities
(details in §3.1). Figure 1 shows the average F1-score of a classification task in a respective dataset (activity recognition in RealWorld-HAR and stress detection in WESAD) as the number of clients with incomplete modalities increases.
The results reveal a consistent degradation in model performance across both datasets, highlighting the detrimental impact of missing modalities on model performance.

Motivation #1: As the number of clients with incomplete modalities increases, the performance of multimodal sensing FL systems progressively deteriorates.

### 2.2. Scalability and Efficiency Challenges

#### 2.2.1. Demand for Scalable FL Systems for Mobile Sensing Applications

In mobile sensing applications, leveraging various input modalities, from physiological sensors for emotional assessment (Kreibig, ) to wearable devices for dietary monitoring (Bahador et al., ; Merck et al., ), enhances app performance (Ramachandram and Taylor, ). Additionally, there has been a recent expansion in both the variety of personal devices (such as smartwatches, bands, rings, and glasses) (applevisionpro, ; Ltd., ) individuals own and in the spectrum of sensors (encompassing motion sensors, and physiological sensors such as photoplethysmography (PPG) and electrodermal activity (EDA) sensors) (Schmidt et al., ) that are incorporated into these devices. This highlights the need for a scalable, multimodal FL system that can support multimodal sensing applications with tens of modalities. Yet, existing systems struggle to scale efficiently with an increase in the number of modality sources.

We demonstrate that existing methods addressing missing modalities in FL such as late fusion-based schemes (Xiong et al., ; Salehi et al., ; Ouyang et al., ) and imputation with generative models (Zheng et al., ; Zhao et al., ), show limited scalability and inefficiency in resources and computation.

Figure: Figure 2. An example of early fusion (left) and late fusion (right) methods in multimodal learning, demonstrated with three modalities.
Refer to caption: extracted/5606462/figures/early_and_late_fusion_difference.png

Figure: Figure 3. An increase in the number of MACs (left) and number of model parameters (right) with the growing number of modality sources for late fusion-based multi-modal training scheme.
Refer to caption: extracted/5606462/figures/motivation/macs_and_num_params.png

#### 2.2.2. Late Fusion-based Schemes

Most existing work (Xiong et al., ; Salehi et al., ; Ouyang et al., ) tackle the missing modality problem in FL using late fusion. The distinction between early fusion and late fusion in multimodal deep learning is depicted in Figure 2. In contrast to early fusion, where all sensing modalities are combined at the input stage to train a unified feature extractor and classifier, late fusion separately trains a unimodal feature extractor for each modality. This approach leads to significant scalability and efficiency challenges as the number of modalities in a multimodal task increases.
To better assess the differences in efficiency and scalability, we compared the MACs (Multiply-Accumulate operations) and the number of trainable parameters between early and late fusion mechanisms. We specifically focused on MACs and number of model parameters because these metrics are pivotal in evaluating a model’s resource consumption — including CPU, GPU, and memory usage — which directly influences its practical deployment.
Figure 3 shows the results. We used CNN + RNN-based network, a widely adopted architecture in multimodal sensing applications (Ordóñez and Roggen, ; Li et al., ; El-Sappagh et al., ).(^1^11We also conducted experiments with CNN-based (LeCun et al., ; Zeng et al., ; Yang et al., ) networks and observed similar trend.)
We computed the MACs and trainable parameters for a two-second window with a sampling rate of 500Hz, utilizing the thop library (tho, ) compatible with PyTorch (Paszke et al., ).
This analysis demonstrates that early fusion methods exhibit only a marginal increase in MACs and number of trainable model parameters, which is nearly imperceptible compared to the linear escalation seen in late fusion approaches. This trend suggests that late fusion is less scalable, particularly for multimodal sensing tasks that involve numerous modality sources.

#### 2.2.3. Imputation with Generative Models

Figure: Figure 4. The number of additional generative networks to train (left) and corresponding MACs (right) for cross-modal imputation.
Refer to caption: extracted/5606462/figures/motivation/generative_model_overhead.png

Recent multimodal FL studies explored modality imputation using generative models such as autoencoders (Baldi, ) to address missing modalities (Zheng et al., ; Zhao et al., ). These works often assume that networks capable of cross-modality transfer have been pre-trained with complete modality data prior to the primary task training. For example, with three modalities A, B, and C, there should exist networks already trained to generate A from B and C, B from A and C, and so forth. However, this assumption is often unrealistic in FL settings as the central server cannot access clients’ raw complete modality data.
Consequently, these cross-modal transfer networks must be trained locally on each client. This requirement introduces additional training burdens, particularly for clients possessing more available modalities.

To evaluate the additional training burden for generative networks in cross-modal knowledge transfer, Figure 4 shows the required number of networks to train (left) and the associated MACs (right). For instance, with two modalities A and B, a client must train two models: A to B and B to A. With three modalities, six combinations emerge (AB, AC, BA, BC, CA, CB). Since the permutations for $n$ modalities is $n\cdot(n-1)$, the complexity of the additional generative network training increases quadratically. Considering the prevalent use of CNN-based blocks followed by transpose-CNN blocks (Zheng et al., ), we calculated the number of MACs associated with these networks. As the number of modalities — and consequently, additional generative networks — increases quadratically, so do the associated MACs. This trend underscores the critical need for scalable approaches in supporting multimodal FL applications efficiently.

Motivation #2:
Current solutions struggle with scalability and efficiency when dealing with a modality number exceeding two or three, the typical range used in existing research.

## 3. System Design

Figure: Figure 5. Overview of the *FLISM* framework.
Refer to caption: extracted/5606462/figures/overview_0313_1500.png

### 3.1. Problem Statement

Consider $N$ as the total number of clients engaged in the multimodal FL task with $M$ input modalities, where $M\geq 2$. In real-world applications, it is common for clients to be unable to use all modalities due to various factors, such as device malfunctions, network conditions, or privacy concerns (See §2.1 for details). To simulate this practical scenario, we let percentage $p$% ($40\leq p\leq 80$) of clients randomly drop up to $M-1$ modalities in their training data.(^2^22Based on our experiments, we observed a significant degradation in accuracy occurs when $p\geq 40\%$.) For brevity, throughout the paper, we use *incomplete modality ratio* to refer to $p$, unless stated otherwise. Consequently, the range of available modalities of a client $c_{k}$ ($1\leq k\leq N$) varies from 1 to $M$ ($1\leq m_{k}\leq M$). Furthermore, each client $c_{k}$ possesses a multimodal dataset $D_{k}$, where $D_{k}=\{x_{i},y_{i}\}^{|D_{k}|}_{i=1}$, $x_{k}$ represents a multimodal data with $m_{k}$ modalities, and $y_{i}$ is the label corresponding to $x_{i}$.

### 3.2. FLISM Framework

We present the overview of our system, *FLISM*, a multimodal FL framework that facilitates learning with incomplete modality data while significantly enhancing computational and communication efficiency. Figure 5 illustrates the overall design of *FLISM*.
*FLISM* achieves a balance between statistical utility (accuracy) and system efficiency (communication and computation) through three key system components:
*missing-modality-robust learning* (§3.2.1), *client role assignment* (§3.2.2), and * cross-client knowledge transfer* (§3.2.3). We first delve into the intuition and rationale behind each component, followed by how these components interact as FL training proceeds (§3.2.4).

#### 3.2.1. Missing-Modality-Robust Learning

In multimodal applications, clients possess a diverse set of modalities, varying in type and quantity. Performing local training with zero-imputation for missing modalities leads to discrepancies in learned representations across clients (Kim et al., ), degrading model performance as demonstrated in §2.1. A naïve solution of training multiple unimodal models for each available modality drastically increases both communication and computational costs. Therefore, an ideal multimodal FL system should learn effective representations with incomplete modalities, while maintaining accuracy and minimizing system cost.

We design missing-modality-robust learning to achieve this goal, leveraging supervised contrastive learning (Khosla et al., ). Supervised contrastive learning was designed to improve the accuracy of supervised tasks by adding a self-supervision loss to the conventional classification loss. In image classification, an example of a self-supervision task could be applying various data augmentation methods (e.g., random flip, rotation, resize) on the same image and training a model to pull the embeddings of the various views of the same image to a similar space. In our case, where we aim to train a model that is robust against missing modalities, adopting random modality drop as a data augmentation technique becomes a natural choice. This strategy enables training a model that can map a diverse set of modalities to a similar embedding space for identical labels. By creating and training on these simulated data, we aim to enhance the model’s robustness to missing modalities.

Following this strategy, a client $c_{k}$, possessing $m_{k}$ ($2\leq m_{i}\leq M$) modalities, randomly drops up to $m_{k}-1$ modalities in each batch training. The client $c_{k}$ then performs local training by adding supervised contrastive loss (Khosla et al., ) to its main training objective, using both the original data $x$ and augmented data $\hat{x}$.

$$ (1) $l_{con}(k)=\sum_{x_{i}\in B}\frac{-1}{|P(i)|}\sum_{p\in P(i)}\log(\frac{exp(z_ {i}\cdot z_{p}/\tau)}{\sum_{a\in A(i)}exp(z_{i}\cdot z_{a}/\tau)})$ $$

In this context, $z_{i}$ represents the embedding of the input data $x_{i}$ that is derived from a batch $B$. The set $P(i)\equiv\{p\in A(i):y_{p}=y_{i}\}$ includes the indices of all positive pairs in the batch, distinct from $i$, and also encompasses the augmented data. Essentially, the loss function ensures that the embeddings of the augmented data $\hat{z}$ are closely aligned with that of the original data $z$, provided they share the identical label. This design promotes consistency in the representation space, reinforcing the model’s ability to recognize and process modality variations of the same data.

#### 3.2.2. Client Role Assignment

In multimodal applications, the contribution of each modality to model training and the resulting accuracy varies. For instance,
in autonomous driving, while LiDAR and cameras are complementary, LiDAR can provide higher quality data for object detection due to its resilience against varied lighting conditions, a capability that cameras lack (Wang and Huang, ).
Similarly, in tasks combining image and text, such as sentiment analysis (Zadeh et al., ) or movie genre classification (Arevalo et al., ), text often plays a more crucial role in achieving higher accuracy (Ma et al., , ). However, existing methods often assume a predefined understanding of each modality’s importance, an assumption that becomes impractical as new multimodal applications emerge and the spectrum of modalities expands.

To address this challenge, we introduce ModUtil, a metric that evaluates the importance of available modalities. ModUtil consists of the standard deviation of the embedding distances (DistSTD) and a contrastive loss ($l_{con}$ in Equation (1)). DistSTD stems from our intuition that the clients with more significant modalities experience higher variance in the embedding distances of their original and simulated (augmented) data. Given the stochastic nature of the modality drop within each batch, clients with more and significant modalities will face difficulty in embedding their original and simulated data to closer feature spaces. For a client $c_{k}$ with data $x_{i}$ in batch $B$, we define DistSTD as follows:

$$ (2) $DistSTD\\ (k)=\sqrt{\frac{\sum_{x_{i}\in B}(Dist(z_{i},\hat{z_{i}})-\mu)^{2}}{|B|}}$ $$

where Dist is a cosine distance to measure the distance between the embeddings of original ($z_{i}$) and simulated ($\hat{z_{i}}$) data respectively, and $\mu$ is the mean value of the embedding distances. Although DistSTD can reflect how diverse a client can simulate missing modality data, it cannot represent the quality of a model trained with simulated data. Therefore, we incorporate contrastive loss $l_{con}$ to our modality utility metric. ModUtil(k) for client $c_{k}$ can be defined as:

$$ (3) $ModUtil(k)=\frac{DistSTD(k)}{l_{con}(k)}.$ $$

Based on ModUtil, we establish two types of client roles: facilitator and learner. Facilitator mainly supports the learning process by simulating a wide range of available modality data. Learner simulates various missing modalities and primarily benefits from cross-client knowledge transfer (detailed in §3.2.3).

To distinguish between facilitator and learner, we dynamically define a threshold based on ModUtil, derived from $N_{M}$ multimodal clients:

$$ (4) $RoleThresh=\frac{1}{N_{M}}\sum_{k=1}^{N_{M}}ModUtil(k).$ $$

The role $r_{k}$ for a client $c_{k}$ can then be assigned using the following rule:

$$ (5) $\displaystyle r_{k}=\begin{cases}facilitator&\text{if $ModUtil(k)>RoleThresh$} \\ learner&\text{if $ModUtil(k)\leq RoleThresh$}.\end{cases}$ $$

#### 3.2.3. Cross-Client Knowledge Transfer

Given the diverse types and number of available modalities each client possesses, and their varying contributions to global model training, it is imperative to devise a method that can compensate for any modality knowledge gaps across all clients. For instance, if a client $c_{k}$ possesses modalities $m_{1}$ and $m_{2}$, in which client $c_{k+1}$ lacks, we must efficiently transfer such modality knowledge to client $c_{k+1}$. However, direct data sharing is not feasible due to FL principles. An alternative approach is to employ cross-modal transfer techniques to reconstruct missing modality data such as autoencoders (Zheng et al., ) or generative adversarial networks (Shang et al., ). Yet, these methods risk user privacy and introduce significant overhead by necessitating cross-modal transfer training for each unique modality combination (§2.2.3).

We introduce cross-client knowledge transfer to handle this issue. This method utilizes knowledge distillation (Hinton et al., ) to share complementary knowledge among clients without transmitting raw data or utilizing cross-modal imputation. We begin by aggregating the local models from multimodal clients that have performed missing-modality-robust learning (§3.2.1) to a single global model, called the knowledge transfer model. Note that the knowledge transfer model is distinct from the primary model, which is the only model used for the final inference. Primary model, on the other hand, aggregates all clients’ local models to ensure a fair representation of data distribution across the entire client pool in FL training.

In global round $t$, the server distributes both knowledge transfer model ($W^{K}_{t}$) and primary model ($W^{P}_{t}$) to the learner.
The learner client $c_{k}$ with data $x_{i}$ in a batch $B$ first initializes its local model weights $w_{t,k}$ to $W^{P}_{t}$. Then, it trains its local model with the knowledge distillation loss $l_{kd}$ to minimize the distance between the output of its local model with that of the knowledge transfer model as follows:

$$ (6) $\begin{split}p^{K}=SoftMax(W^{K}_{t}(x_{i})/T),\\ p^{c_{k}}=SoftMax(w_{t,k}(x_{i})/T),\end{split}$ $$

$$ (7) $l_{kd}=T^{2}\times KLD(p^{K}\,||\;p^{c_{k}})$ $$

where the $KLD$ is the Kullback–Leibler divergence (kul, ), calculated between $p^{K}$, the softmax probabilities derived from the global knowledge transfer model’s output, and $p^{c_{k}}$, the softmax probabilities obtained from the client’s local model output. $T$ represents a temperature used to soften the probability distribution of the logits, facilitating the learning of more generalizable knowledge.

To summarize, all clients use classification loss (cross-entropy loss) to train their local models for the primary task. Multimodal clients additionally incorporate contrastive loss (Equation 1), while both unimodal and multimodal learner clients integrate knowledge transfer loss (Equation 7) into their optimization objectives.

#### 3.2.4. System Flow

Figure: Algorithm 1 *FLISM* Training Flow

The training flow of *FLISM* is detailed in Algorithm 1. At the start of each global round $t$, the server assigns roles $R$ to selected clients $S$ based on their modality utility, ModUtil, (Line 3 and §3.2.2). Multimodal clients perform missing-modality-robust learning to enhance the primary model’s robustness against various missing modality patterns (Lines 7-10 and §3.2.1). Additionally, learner clients train their local models by distilling knowledge from multimodal clients alongside their primary classification tasks (Lines 9-12). In each global round $t$, the server aggregates the local models from all selected clients into the global primary model $W^{P}_{t}$, and it also integrates the local models from multimodal clients into $W^{K}_{t}$ (Lines 13-14).

## 4. Experiments

In this section, we describe the multimodal sensing datasets (§4.1), baseline methods (§4.2), and experiment setup (§4.3), including implementation details and evaluation metrics, used in our study.

### 4.1. Datasets

**Table 1. Summary of datasets.**
| Dataset | #Users | #Modals | Modality<br>Types |
| --- | --- | --- | --- |
| PAMAP2 (Reiss and Stricker, ) | 8 | 6 | ACC, GYRO |
| RealWorld<br>HAR (Sztyler and Stuckenschmidt, ) | 15 | 10 | ACC, GYRO |
| WESAD (Schmidt et al., ) | 15 | 10 | ACC, BVP,<br>ECG, EDA, EMG,<br>RESP, TEMP |

Table 1 describes the three real-world multimodal sensing datasets used in our experiments: PAMAP2 (Reiss and Stricker, ), RealWorld HAR (Sztyler and Stuckenschmidt, ) and WESAD (Schmidt et al., ).

PAMAP2 (Reiss and Stricker, ) contains data from nine users performing twelve activities, captured using Inertial Measurement Unit (IMU) sensors. We excluded data from one participant due to the presence of only a single activity data (Jain et al., ). The dataset includes readings from accelerometers and gyroscopes modalities, positioned on three different body parts: the wrist, chest, and ankle, resulting in a total of six sensing input modalities.

RealWorld HAR (Sztyler and Stuckenschmidt, ) is a multi-device multimodal dataset for human activity recognition (HAR), featuring data from fifteen participants engaged in eight activities. Each participant was equipped with seven IMU devices positioned on the head, chest, upper arm, wrist, forearm, thigh, and shin. The data from the forearm and thigh were omitted due to incomplete activity coverage. As a result, the dataset encompasses ten sensing modalities, derived from five body locations and two types of IMU sensors.

WESAD (Schmidt et al., ) is a multi-device multimodal dataset for wearable stress and affect detection (Schmidt et al., ). It encompasses data collected from fifteen participants who wore both a chestband and a wristband, capturing physiological sensor data such as Electrocardiogram (ECG), Electrodermal Activity (EDA), Electromyogram (EMG), Blood Volume Pressure (BVP), and Respiration (RESP), Skin Temperature (TEMP), in addition to motion data via an Accelerometer (ACC). The objective is to classify the participants’ emotional states into three categories: neutral, stress, and amusement. The chestband monitored ACC, ECG, EMG, EDA, TEMP, and RESP, whereas the wristband tracked ACC, BVP, EDA, and TEMP, collectively resulting in ten distinct modalities.

### 4.2. Baselines

To evaluate the performance of *FLISM*, we consider three early fusion (FedAvg (McMahan et al., ), FedProx (Li et al., ), MOON (Li et al., )) and two late fusion methods specifically designed to address incomplete modality problem in FL: FedMM (Feng et al., ), Harmony (Ouyang et al., ).

FedAvg (McMahan et al., ) represents the foundational approach to FL, enabling decentralized training without the need to share raw data. This method involves transmitting updated local model weights to a centralized server for aggregation. As a baseline framework, FedAvg is crucial for assessing the lowest achievable accuracy, especially in scenarios lacking specific mechanisms to address missing modalities. It also sets a benchmark for the minimum system cost, given that clients are only required to train and share the weights of a single model.

FedProx (Li et al., ) was proposed to address system and statistical heterogeneity. It enhances performance through the addition of a proximal term to the local training loss function, which aims to minimize the discrepancy between the global and local models. Given its effectiveness in environments with heterogeneous clients, and considering that modality heterogeneity represents a form of heterogeneity challenge, we have selected FedProx as one of our baselines.

MOON (Li et al., ), similar to FedProx, targets local data heterogeneity problem. It incorporates contrastive learning into FL to reduce the gap between the global and local model’s embeddings while increasing the disparity from the embeddings of the previous local model. MOON has demonstrated superior performance over other FL methods across different image classification tasks, showcasing its effectiveness.

FedMM (Feng et al., ) is a late fusion technique, specifically designed to tackle missing modality issue in multimodal FL applications. Initially, FedMM conducts unimodal training for each available modality across all clients. It then merges these unimodal representations through a cross-attention mechanism (Vaswani et al., ). Evaluation across a variety of tasks, supporting upto two modalities, demonstrated FedMM’s effectiveness.

Harmony (Ouyang et al., ) is another late fusion approach proposed to manage incomplete modality data in multimodal FL tasks. It structures the FL training process into two distinct stages: initial modality-wise unimodal training, followed by a second stage dedicated to multimodal fusion. Additionally, Harmony incorporates modality biases in the fusion step to address local data heterogeneity. Evaluations on real-world datasets with up to three modalities have demonstrated Harmony’s effectiveness compared to other state-of-the-art baselines.

### 4.3. Experiment Setup

Implementation. In our experiments, we use a 1D convolutional neural network (CNN) as the encoder architecture, following the design for sensing application assessments (Haresamudram et al., ). For a fair comparison, we standardized the encoder models across all methods. We set random client selection rates from 30% to 50% based on the total dataset users. Standard settings include a learning rate of 0.01, weight decay of 0.001, and a batch size of 32, with SGD as the optimizer. After a grid search to fine-tune the hyperparameters for each baseline, we adjusted the MOON’s learning rate to 0.001 and its batch size to 64. We performed all experiments with ten different seeds and reported the average values. We implemented *FLISM* and all baselines using Python 3.8 and PyTorch 1.7 (Paszke et al., ). We plan to open source upon publication.

Metrics. To assess the accuracy of our method and compare it with baseline approaches, we utilize the macro F1 score. The macro F1 score is recommended practice (PlÖtz, ) when working with imbalanced data and widely adopted in prior work (Cho et al., ). To evaluate communication overhead, we measure the total time spent on the model upload and download processes by all clients participating in FL training (Salehi et al., ).
For computation overhead, we calculate the total number of parameters trained throughout the FL training process across all clients. This provides insight into the computational complexity and resource requirements of the FL system, offering a measure of its scalability and efficiency.

## 5. Results

### 5.1. End-to-end Performance

**Table 2. Comparison of *FLISM*’s accuracy and system overhead in terms of communication and computation cost (number of trained parameters). EF denotes Early Fusion, whereas LF represents Late Fusion. Results in bold represent the highest performance, whereas those underlined denote the second-highest (runner-up) performance.**
| Dataset | PAMAP2 | WESAD | RealWorld HAR |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Type | Method | Accuracy<br>(F1-score) | Comm<br>(seconds) | #Params<br>($\times 1e6$) | Accuracy<br>(F1-score) | Comm<br>(seconds) | #Params<br>($\times 1e6$) | Accuracy<br>(F1-score) | Comm<br>(seconds) | #Params<br>($\times 1e6$) |
| EF | FedAvg (McMahan et al., ) | .709 | 430 ($1\times$) | 83 ($1\times$) | .588 | 483 ($1\times$) | 109 ($1\times$) | .738 | 356 ($1\times$ ) | 85 ($1\times$) |
| EF | FedProx (Li et al., ) | .710 | 430 ($1\times$) | 83 ($1\times$) | .587 | 483 ($1\times$) | 109 ($1\times$) | .738 | 356 ($1\times$) | 85 ($1\times$) |
| EF | MOON (Li et al., ) | .720 | 673 (1.57$\times$) | 109 (1.31$\times$) | .399 | 815 (1.69$\times$) | 161 (1.48$\times$) | .620 | 553 (1.55$\times$) | 111 (1.31$\times$) |
| LF | FedMM (Feng et al., ) | .753 | 1551 ($3.61\times$) | 313 ($3.77\times$) | .544 | 2271 ($4.70\times$) | 512 ($4.7\times$) | .797 | 1522 ($4.28\times$) | 377 (4.44$\times$) |
| LF | Harmony (Ouyang et al., ) | .698 | 849 ($1.97\times$) | 103 (1.24$\times$) | .348 | 3224 ($6.67\times$) | 305 (2.8$\times$) | .721 | 1639 ($4.69\times$) | 157 (1.85$\times$) |
| EF | *FLISM* | .770 | 648 (1.51$\times$) | 109 (1.31$\times$) | .589 | 825 (1.71$\times$) | 161 (1.48$\times$) | .778 | 547 (1.54$\times$) | 111 (1.31$\times$) |
|  |  |  |  |  |  |  |  |  |  |  |

We start our analysis with a comparison of our system, *FLISM* against baselines. We focus on model accuracy (F1-score) and system efficiency, which includes the communication cost and the total number of parameters trained (i.e., it reflects the computation cost on the client side). Table 2 presents results averaged from three missing modality scenarios with incomplete modality ratio $p$ of 40%, 60%, and 80%. We selected FedAvg (McMahan et al., ) and FedProx (Li et al., ) as benchmarks for communication and computation costs as they represent the simplest and lightest forms of FL that involve local training and exchange of single model’s weights. Although FedAvg (McMahan et al., ) and FedProx (Li et al., ) are efficient in communication and computation, their accuracy falls short, especially with higher incomplete modality ratio (details in §5.2). MOON (Li et al., ) incurs additional overhead by exchanging an extra model and incorporating contrastive loss in local training. However, its performance declines with the increase in the number and complexity of modalities, as shown in the WESAD dataset, where MOON ranks second to last (.399 and .189 lower than FedAvg in F1 score). This result indicates that MOON’s assumption, that the global model always outperforms a client’s local model, may not hold in multimodal FL with incomplete modalities.

Both FedMM (Feng et al., ) and Harmony (Ouyang et al., ) are late fusion techniques designed to handle missing modalities in multimodal FL tasks. Initially, they train multiple unimodal models for each unique modality on the client side, followed by training fused models. Consequently, this approach results in higher communication and computation costs compared with early fusion methods. We observe that the performance of these two methods varies by dataset: for activity recognition from motion sensing modalities (PAMAP2, RealWorld-HAR), FedMM and Harmony match or slightly outperform early fusion methods. In contrast, in WESAD dataset, their F1 score significantly drops (.544 and .348), underscoring the critical role of early fusion in capturing low-level correlations between sensing modalities.

On the other hand, *FLISM* consistently delivers superior performance, ranking as either the highest or second-highest across all metrics and datasets.
In summary, *FLISM* achieves a balance between accuracy and efficiency by (1) outperforming baseline methods in F1 score by an average of .051, .096, .055 for PAMAP2, WESAD, and RealWorld-HAR, respectively; (2) demonstrating enhanced system efficiency with an average increase of 1.85$\times$, 3.33$\times$, and 2.89$\times$ in communication speed, and 1.91$\times$, 2.53$\times$, and 2.40$\times$ in computation efficiency, compared to late fusion methods tackling incomplete modalities.

### 5.2. Accuracy Performance

**Table 3. Accuracy improvement of *FLISM* over baselines with various incomplete modality ratios.**
| Dataset | PAMAP2 | WESAD | RealWorld-HAR |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| $p$ | $40\%$ | $60\%$ | $80\%$ | $40\%$ | $60\%$ | $80\%$ | $40\%$ | $60\%$ | $80\%$ |  |  |  |  |  |  |  |  |  |  |
|  | Method | F1 | $\Delta$ F1 | F1 | $\Delta$ F1 | F1 | $\Delta$ F1 | F1 | $\Delta$ F1 | F1 | $\Delta$ F1 | F1 | $\Delta$ F1 | F1 | $\Delta$ F1 | F1 | $\Delta$ F1 | F1 | $\Delta$ F1 |
|  | FedAvg (McMahan et al., ) | .734 | .059 $\uparrow$ | .727 | .052 $\uparrow$ | .666 | .071 $\uparrow$ | .674 | .007 $\downarrow$ | .570 | .004 $\uparrow$ | .521 | .050 $\uparrow$ | .793 | .021 $\uparrow$ | .740 | .048 $\uparrow$ | .680 | .052 $\uparrow$ |
|  | FedProx (Li et al., ) | .732 | .061 $\uparrow$ | .731 | .048 $\uparrow$ | .666 | .071 $\uparrow$ | .673 | .006 $\downarrow$ | .571 | .003 $\uparrow$ | .518 | .080 $\uparrow$ | .797 | .017 $\uparrow$ | .738 | .050 $\uparrow$ | .678 | .054 $\uparrow$ |
|  | MOON (Li et al., ) | .742 | .051 $\uparrow$ | .725 | .054 $\uparrow$ | .692 | .045 $\uparrow$ | .444 | .223 $\uparrow$ | .393 | .181 $\uparrow$ | .359 | .167 $\uparrow$ | .664 | .150 $\uparrow$ | .631 | .157 $\uparrow$ | .565 | .167 $\uparrow$ |
|  | FedMM (Feng et al., ) | .768 | .025 $\uparrow$ | .763 | .016 $\uparrow$ | .748 | .011 $\downarrow$ | .549 | .118 $\uparrow$ | .546 | .028 $\uparrow$ | .536 | .010 $\downarrow$ | .813 | .010 $\uparrow$ | .795 | .007 $\downarrow$ | .782 | .050 $\downarrow$ |
|  | Harmony (Ouyang et al., ) | .721 | .072 $\uparrow$ | .696 | .083 $\uparrow$ | .676 | .061 $\uparrow$ | .336 | .331 $\uparrow$ | .360 | .214 $\uparrow$ | .347 | .179 $\uparrow$ | .699 | .115 $\uparrow$ | .716 | .072 $\uparrow$ | .748 | .016 $\downarrow$ |
|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |

To rigorously evaluate the *FLISM*’s accuracy compared to baseline methods, we conducted in-depth experiments under various modality drop scenarios. The results are presented in Table 3. Here, an increased value of $p$ signifies a higher incidence of clients with incomplete modalities. The final test accuracy is measured using the F1 score noted as $F1$, with improvements of the *FLISM* marked as $\Delta F1$.

*FLISM* consistently outperforms early fusion methods, achieving an average F1 score improvement of .057, .080, and .064 on the PAMAP2, RealWorld-HAR and WESAD datasets, respectively.
For the WESAD dataset, while FedAvg and FedProx may exhibit marginally superior performance at lower values of $p$, *FLISM* demonstrates enhanced capability in more demanding contexts characterized by higher $p$ values. In these scenarios, more number of clients with incomplete modalities participate in FL training, underscoring *FLISM*’s robustness in complex environments.

Furthermore, *FLISM* outperforms late fusion approaches in the majority of scenarios. While FedMM achieves marginally higher F1 scores, ranging from .007 to .050, in some instances, this advantage incurs considerable increases in communication and computation expenses, increasing by 2.57$\times$ - 2.76$\times$, and 3.00$\times$ - 3.40$\times$, respectively.

Harmony, in its initial phase, trains multiple unimodal models similar to FedMM’s approach. However, its multimodal fusion stage, which leverages clustering, underperforms, yielding the lowest F1 scores (.336, .360, and .347) on the WESAD dataset among all methods evaluated. This underperformance is attributed to Harmony’s assumption of uniform modality across clients during multimodal fusion stage, a presumption that may hold for tasks with a small number of modalities (up to three, as in Harmony’s experiments). The challenge intensifies for Harmony when faced with a greater variety and number of modalities, necessitating separate fusion training for each unique modality combination among client groups.
In contrast, *FLISM* does not make assumptions about modality uniformity, and operates flexibly across a wider range and variety of sensing modalities. Its enhanced performance in complex multimodal sensing tasks, even with heterogeneous modalities, is attributed to simulating diverse missing modality scenarios (§3.2.1) and facilitating the implicit transfer of robust representations to clients requiring complementary knowledge (§3.2.3).

### 5.3. System Efficiency

Figure: Figure 6. Comparison of *FLISM* with other baselines in terms of communication cost. *FLISM* is more communication-efficient compared to late fusion-based methods.
Refer to caption: extracted/5606462/figures/results/comm_pamap2_final.png

Figure: Figure 7. Comparison of *FLISM* with other baselines in terms of computation cost. *FLISM* is more computation-efficient compared to late fusion-based methods.
Refer to caption: extracted/5606462/figures/results/compu_pamap2_final.png

We compare the communication and computational efficiency of *FLISM* against late fusion baselines throughout all FL global rounds.
Regarding communication cost, we simulate upload and download speeds of client devices by leveraging FLASH (Yang et al., ), an extensive FL simulation framework. FLASH utilizes a dataset derived from more than 136,000 smartphones, offering a real-world basis for our simulations. Each client is assigned a random device profile before the experiments. In each round, a client’s communication speed is determined by sampling from a Gaussian distribution, based on the assigned profile’s mean and standard deviation. This method reflects a realistic scenario where each client uses their own device, though communication speeds vary due to network conditions. To calculate the communication cost, we only consider model weights transferred between the client and the server, as auxiliary information transfer is minimal, typically integer values which is negligible compared with the substantial model weights in the FL setting. For computational overhead, we calculate the total number of parameters trained by all clients throughout the FL training process. This method illuminates the FL system’s computational complexity and resource consumption, providing insights into its scalability and operational efficiency.

Figure 6 and 7 show a consistent trend in communication and computational efficiency across all datasets, underscoring the effectiveness of *FLISM*. An exception occurs when the incomplete modality ratio $p$ is 80% in the PAMAP2 dataset, with Harmony exhibiting the lowest communication and computational overhead. This lower overhead results from Harmony’s strategy of selecting clients with identical modality sets during the initial training phase, which may lead to a limited number of available clients when the client pool is restricted. This limitation is particularly pronounced in the PAMAP2 dataset, which consists of only eight clients. Therefore, a significant reduction in available clients or a lack of complete modality data can impede the unimodal model’s training process. This leads to a notable decrease in performance, as evidenced by the F1-score degradation (Table 3), despite a slight increase in system efficiency.

In other cases, analysis reveals a uniform pattern in communication and computation costs across all datasets. This trend demonstrates that FedMM and Harmony both incur considerable overhead, which becomes more severe as the number of modalities increases. This is because these methods require each client $c_{k}$ to exchange $m_{k}$ unimodal models. In contrast, *FLISM* significantly reduces overhead, showcasing an average improvement of 2.40$\times$, 2.76$\times$, and 2.78$\times$ in communication costs for the PAMAP2, WESAD, and RealWorld-HAR datasets compared to FedMM, and 1.37$\times$, 3.98$\times$, and 3.00$\times$ compared to Harmony, respectively. Moreover, *FLISM* also demonstrates an average improvement of 2.88$\times$, 3.18$\times$, and 3.41$\times$ in computation costs for the same datasets in comparison to FedMM, and 0.95$\times$, 1.90$\times$, and 1.42$\times$ when compared to Harmony, illustrating its efficiency in both communication and computational aspects.

### 5.4. Scalability Analysis

Figure: Figure 8. Scalability Analysis of *FLISM*: Comparing communication (Left) and computation overheads, measured by the number of trained parameters (Right), against late fusion baselines. Y-Axis is in Log-Scale.
Refer to caption: extracted/5606462/figures/results/simul_commu.png

As the integration of sensors into a wide range of devices continues to grow, there is an increasing demand for scalable FL systems. These systems must efficiently handle an expanding number of devices and modalities to support diverse multimodal sensing applications (§2.2.1).

To assess scalability with respect to communication and computation costs, we conducted experiments involving an increasing number of sensing modalities. The results, comparing *FLISM* to late fusion baselines, are shown in Figure 8. The X-axis denotes the number of modalities, and the Y-axis indicates the communication and computation costs (measured in the number of trained parameters), both presented on a logarithmic scale.
Note, that the multimodal datasets we utilized in our experiments had at most 10 sensing modalities, as shown in Table 1. However, real-world multimodal sensing tasks could involve a greater number of modalities, given the data may originate from a diverse range of sensor and device combinations (Schmidt et al., ; Orzikulova et al., ).
To better reflect such extensive real-world scenarios, we simulated a multimodal sensing task with modalities varying from 5 to 30. We set the total number of clients, $N$, to 100, selecting 10% for participation in each FL training round. The incomplete modality ratio $p$, was configured at 40%. The FL training spanned 20 rounds, during which each client trained their models for one local epoch.

As shown in Figure 8, both FedMM and Harmony experience significant increases in system costs as the number of modalities in a multimodal sensing task grows. Specifically, the communication cost for tasks with five versus 30 modalities increases by 1,016.89 seconds for FedMM and by 12,842.84 seconds for Harmony. Similarly, computation costs escalate, as evidenced by an increase in the number of trained parameters—188.66% for FedMM and 464.31% for Harmony.
In contrast, *FLISM* maintains a stable system cost that is almost negligible compared to late fusion baselines, without compromising accuracy. In particular, it outperforms FedMM and Harmony in efficiency, with communication improvements between 3.24$\times$ - 9.04$\times$ and 10.93$\times$ - 85.10$\times$ and in computation between 3.73$\times$ - 10.60$\times$ and 5.81$\times$ - 32.29$\times$, respectively.

In summary, our simulations, involving up to 30 sensing modalities, revealed that late fusion-based methods face significant system overheads due to the need to train separate unimodal models for each modality and then fuse them for every unique combination, with overheads escalating sharply as modalities increase. In contrast, *FLISM* provides a scalable and substantially more efficient solution, achieving this without sacrificing accuracy.

### 5.5. Component-wise Analysis

**Table 4. Ablation analysis of the *FLISM* on PAMAP2 dataset. Each component is annotated with abbreviations. Missing-Modality-Robust Learning (MMRL, §3.2.1), Cross-Client Knowledge Transfer (CCKT, §3.2.3), and Client Role Assigment (CRA, §3.2.2).**
| Version | Description | F1-score |
| --- | --- | --- |
| Ver. 1 | *FLISM* ( w/o MMRL, CCKT and CRA) | .709 |
| Ver. 2 | *FLISM* ( w/o CCKT and CRA) | .750 |
| Ver. 3 | *FLISM* ( w/o CRA) | .767 |
| Final | *FLISM* | .770 |
|  |  |  |

To understand and assess the effectiveness of each individual system component incorporated in *FLISM*, we implement its four different variations, including the complete version. The results from experiments conducted using the PAMAP2 dataset are presented in Table 4. We report the average F1-score across three different incomplete modality ratio scenarios, maintaining consistency with our experiments throughout the paper. As detailed in §3.2, *FLISM* consists of three key components: Missing-Modality-Robust Learning (MMRL, §3.2.1), Cross-Client Knowledge Transfer (CCKT, §3.2.3), and Client Role Assignment (CRA, §3.2.2).

The most basic version of *FLISM*, lacking all three key components, functions similarly to vanilla FedAvg. This version does not engage in simulation, transfer knowledge, or make adjustments based on the quality of data available at the clients.
The introduction of the MMRL component (Ver. 2) leads to a .041 increase in the F1-score. This enhancement confirms the contribution of simulating various missing modality scenarios, which enables the model to learn robust representations when faced with incomplete modality data. Further enhancements are observed with the integration of CCKT (Ver. 3), which facilitates the transfer of these simulated missing modality scenarios across clients, resulting in an additional .017 increase in the F1-score compared to the version excluding CCKT. The culmination of integrating all three components in the complete version of *FLISM* achieves the highest F1-score of .770, demonstrating the significant and distinct contributions of each component to the system’s overall effectiveness.

## 6. Related Work

Multi-modal Learning in Sensing Applications.
Multimodal learning is becoming more prevalent in real-world mobile sensing applications, including mental health assessment (Wang et al., ), object identification (Gong et al., ), stress detection (Yu and Sano, ), and eating episode detection (Shin et al., ). Consequently, there has been a recent surge in efforts aimed at enhancing model performance for applications that leverage multimodal data. In this context, COCOA (Deldari et al., ) enhances representation quality by developing a unique objective function that optimizes cross-correlation between different data types and reduces similarity among unrelated instances. Similarly, ColloSSL (Jain et al., ) leverages data from multiple devices worn by a user to extract high-quality features, treating data from devices attached to different body positions as separate modalities.
Although these works achieve higher performance by leveraging the complementary knowledge across modalities, sending raw physiological and motion sensing data to a central server to a central server might lead privacy concerns.

Multi-modal Federated Learning. Federated Learning (FL) (McMahan et al., ) is a distributed machine learning approach that allows for collaborative model training without exposing data from its own devices. Although many FL studies have improved functionality using various modalities, most multimodal FL research assumes all modalities are available. CreamFL (Yu et al., ) enables training larger server models from clients with heterogeneous model architectures and data modalities, and FedMEKT (Le et al., ) leverage representations from different modalities. However, most of the existing multimodal FL approaches assume all modalities are available for model training, despite considering statistical heterogeneity.

Missing Modalities in Federated Learning. Research on incomplete modalities in FL is explored in various applications, including autonomous driving (Zheng et al., ) and Alzheimer’s disease monitoring (Ouyang et al., ). Harmony (Ouyang et al., ) disentangles the multimodal network training in a two-stage framework. AutoFed (Zheng et al., ) propose an autoencoder-based data imputation method to fill missing data modality with the available ones. FedMM (Feng et al., ) employs attention-based fusion to integrate outputs from uni-modal models trained separately for each modality. Note that FedMM (Feng et al., ) and the initial stage of Harmony (Ouyang et al., ) involve training uni-modal models, a process that could hinder scalability as the number of modalities grows. Similarly, AutoFed (Zheng et al., ) faces limitations as it requires all combinations of autoencoders for data imputation. *FLISM* proposes a communication and computation-efficient multimodal FL training approach, enhancing scalability for larger number of modalities.

## 7. Discussions

In this section, we outline discussions and promising directions for future research.

Extension to Non-sensing Modalities.
Recently, the study of large multimodal models, incorporating modalities such as images and text, has gained significant attention (Radford et al., ; Saito et al., ). Although *FLISM* performs well in accuracy and system efficiency, it is mainly focused on multimodal sensing applications. This focus stems from observations that early fusion is systematically more efficient than late fusion. We plan to extend *FLISM* to include non-sensing modalities, such as images and audio. This could involve utilizing small pre-trained models to extract features from these modalities, aligning the extracted features, and proceeding with the *FLISM* training. This approach could broaden the applicability of *FLISM* to a wider range of multimodal integration scenarios, enhancing its versatility and effectiveness.

Runtime Handling of Incomplete Modalities.
We focused on scenarios involving static modality drops. This approach stems from our observation that dropping the entire modality throughout the FL training causes the highest accuracy degradation. However, we acknowledge that dynamic modality drops, where modalities might become unavailable at various points during application runtime, is also a critical aspect. The Missing-Modality-Robust Learning (§3.2.1), a component of *FLISM*, is designed to accommodate extensions for simulating various dynamic drop scenarios. Further development of the method to specifically cater to dynamic drop scenarios at runtime is an area for future exploration.

System Heterogeneity-Aware Client Selection. Although the *FLISM* achieves optimal balance between system efficiency and model accuracy, it overlooks individual user system utilities, such as WiFi connectivity, battery life, and CPU memory. As highlighted in our motivation, the number and type of modalities available for local training vary by user, also the system utilities can change dynamically. Building on top of the Client Role Assignment (§3.2.2), we can devise an additional client selection method that accounts for device utility to enhance convergence speed. Future research could explore adapting the method to accommodate system heterogeneity.

User Engagement and Incentive Mechanisms. In our system, *FLISM*, clients with multiple modalities perform robust learning for missing modalities, conducting simulations to enable clients with fewer modalities to benefit from these robust representations. The availability of numerous modalities offers greater opportunities for conducting these beneficial simulations. Despite necessitating an extra forward pass and thus higher computational demand compared to unimodal clients, the potential for improving the global model’s training through increased simulation activities is important. In real-world applications, motivating users to undertake more simulations, despite the higher computational costs, could substantially improve overall model performance. Implementing effective incentive mechanisms is crucial to boosting user participation and, thereby, enriching the model’s learning environment.

## 8. Conclusion

We introduced *FLISM*, a multimodal FL framework designed to address the challenges of incomplete sensing modalities. At its foundation, *FLISM* employs simulations of various scenarios with missing modalities to learn representations capable of handling incomplete data. It also identifies clients with less critical modalities, enabling cross-modality knowledge transfer. Through comprehensive evaluations with three real-world multimodal datasets and scalability tests, *FLISM* was benchmarked against five baseline models, reveals its efficacy. *FLISM* improves model performance, showing an average increase of .067 in F1-score, and achieves reductions in both communication and computational overhead, being 2.69$\times$ faster and 2.28$\times$ more efficient. Additionally, in scenarios with a larger number of modalities, *FLISM* enhances communication (3.23$\times$$\sim$85.10$\times$) and computation (3.73$\times$$\sim$32.29$\times$) efficiency.

## Peer Reviews

confidence: 3
contribution: 4
initial_rating: 6
presentation: 4
questions: - How generalizable the proposed solution is to other modalities such as image and audio?
- Can you describe how the modality variance could potentially be handled?
- Will the Modality Quality-Aware Aggregation method cause biases towards users/device with more modality and hence cause performance drop for other users?
soundness: 3
strengths: - The paper is well motivated and address an important real-world problem in FL (modality variance).
- The early fusion with zero imputation fits well in the FL paradigm. The proposed algorithm provides mechanism at both client and server end to consider the modality variance problem.
- The proposed solution is evaluated on 4 different datasets and on both model and system performance metrics.
- The paper is well organized and written. The presentation is clear and easy to follow.
summary: This paper tries to develop an efficient and scalable federated learning algorithms that handles incomplete modalities while maintaining high accuracy. Healthcare applications increasingly reply on multimodal data and there is need to utilize such data in a privacy preserving manner (.e.g, FL). But incomplete modalities brings challenges to FL. Deep imputation methods are unrealistic in FL setting and not scalable as the number of modalities increases. Intermediate fusion methods scales poorly with number of modalities as well, also incurs high communication and computation costs. Early fusion is more efficient but may lead to performance degradation due to drift etc.

The authors proposed FLISM, an early fusion based algorithm with 3 key components to address the challenges above. Modality-Invariant Representation Learning use supervised contrastive learning to learn a representation that's consistent despite varying input modalities. This addresses early fusion with zero imputation may lead to biased modality relationships. Modality Quality-Aware Aggregation uses an entropy-based metric to evaluate and weights updates from clients. Global-Aligned Knowledge Distillation use KD to align local model with global model to enhance model's ability to generalize over different modality sets.

The proposed algorithm is evaluated on 4 datasets and compared against state of the art baselines. The proposed algorithm shows good performance and system efficiency.
weaknesses: - Although assumed a multi-modal setting, the solution is only evaluated on time series data.
- The proposed solution is only evaluated under static modality drop. Whereas in reality, the modality variance is more dynamic.
- Although the authors provided empirical evidence to support entropy-based metric for client selection, they only seem to consider model performance and did not provide much discussion on fairness.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: See weakness for more details.

- How is the fusion performed? How is the heterogeneity in data structured and sampling rate handled? Please elaborate and see weakness for more details. 
- What are the major novelties compared to previous modality dropout and global-local knowledge distillation methods?
- Why is the same modality from different locations considered different? (e.g. three same modalities from 2 different locations == 10 modalities) Please compare how this is treated with previous works (e.g. Harmony).
soundness: 3
strengths: - The challenge described is clear and meaningful for multimodal learning.
- The design is well supported by the motivation and address the concern raised in the challenge.
- The authors provide detailed illustrations and easy-to-follow algorithm to clarify their technical methods.
- The method demonstrates both superior performance and significant efficiency gains compared to SOTA frameworks.
summary: Briefly summarize the paper and its contributions. This is not the place to critique the paper; the authors should generally agree with a well-written summary.

This paper introduces an efficient federated learning framework specifically designed for federated multimodal sensing applications with incomplete modalities, called FLISM. The authors mainly propose three techniques to address the missing modalities:

1. Modality-invariant representation learning. which uses supervised contrastive learning to extract features robust to missing modalities by augmenting input data through dropping random modalities and adding noises during early fusion.
2. Modality quality aware aggregation that prioritizes high-quality multimodal data from heterogeneous clients.
3. Global aligned knowledge distillation that aligns the local client models with the global server model to reduce biases introduced by limited modalities training on clients.

The authors compare FLISM with different SOTA federated multimodal learning frameworks and demonstrate improvements in F1 scores across different benchmarks. The early fusion also shows greater efficiency compared to the late fusion since only one encoder is involved regardless of the number of modalities. The authors also evaluate FLISM with an imputation method (AutoFed) to handle incomplete data and demonstrate significant improvements in performance and computational costs.  Ablation studies are also conducted to understand the performance of FLISM.
weaknesses: **Major**

- I am concerned with the novelty of the work. There have been many works on modality dropout to improve multimodal robustness. Although some of these works may be conducted in the representation space, they share the same concept of randomly dropping a subset of the modalities to improve robustness. Furthermore, some previous works have also performed knowledge distillation [1] to regularize the discrepancy between the clients. The authors should expand the related works, discuss how the proposed method is different from existing work, and emphasize the key novelties.
- The authors should improve the evaluation presentation to reduce confusion. For example, the term $p$ is used in multiple ways where p means incidence of clients with incomplete modalities for 4.2 but in later sections it represents the ratio of modalities that is missing.
- The authors should also elaborate on the experimental setup. Are the incomplete modality ratios for the training data (from the appendix)? Then, what is the incomplete modality ratio for the testing data? If the testing data is complete, what is the performance like if it contains incomplete multimodal data, which is common under the sensor deployment setting?
- The number of nodes involved for each dataset should also be specified.
- The efficiency in communication and computation cost reflects the advantage of FLISM over methods that leverage intermediate fusion. It does not show a comparison with any early fusion frameworks. Early fusion frameworks naturally have lower costs and a lower number of parameters as the number of modalities scales, so it may not be a fair evaluation if Figure 2 excludes those early fusion frameworks.
- The authors could clarify how they perform the fusion. It is unclear how the early fusion is performed. How does this generalize to the case when the time-series input modalities have different sampling rates, which could lead to different information densities? The performance of additional datasets with more diverse modalities could improve the significance of the work.

**Minor**

- Some parts of Table 1 are a bit confusing. The caption and the left column say accuracy, but the indicators in each column say F1 and delta F1.
- Appendix 5. E: Late fusion → Intermediate fusion.
- Since the setup is unclear and the code has not been released, the reproducibility of this work is also unclear.

[1] FedGKD: Toward Heterogeneous Federated Learning via Global Knowledge Distillation

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: - How are the datasets split for experiments, i.e., do each client correspond to the participants in the datasets? What about the training and test sets? Are there any test users (test clients)?
- There are three definitions of $p$ in the paper. In Eq. (6), it is defined as a probability; in Algorithm 1, it is the fraction of clients; and in the experimental section, it is defined as the incomplete modality ratio.
- Is the incomplete modality of each client dynamically changing during the FL process? For example, could client 1 have modalities A and B in the first communication round, but B and C in the second round?
soundness: 2
strengths: The paper is well organized, and the proposed methodology is interesting, including modality dropout and entropy in the context of multimodal FL. In addition, the paper conducts extensive comparisons with baselines, and the results show that the proposed method outperforms the baselines.
summary: This paper aims to address the challenge of missing modalities in multimodal FL frameworks, particularly in cases involving numerous modalities. The paper proposes the FLISM algorithm, which utilizes early fusion in multimodal FL and incorporates three key components. MIRL employs modality dropout to enhance the model’s robustness to missing modalities, MQAA performs weighted aggregation based on client entropy, and GAKD assists local learning through KL divergence. The paper compares FLISM against three SOTA multimodal FL methods and three traditional FL methods in four datasets, considering different modality missing rates, communication, and computation overhead.
weaknesses: - I feel that the level of novelty is somewhat limited. Many elements mentioned in the paper, though applied in a new context of multimodal FL, are existing techniques (e.g., modality dropout). Other aspects, such as early fusion, entropy-based weighted aggregation, and knowledge distillation using KL divergence, are also well established. I can find similar techniques just by randomly searching on Google Scholar.

[1] Improving Unimodal Object Recognition with Multimodal Contrastive Learning

[2] Contrastive Learning based Modality-Invariant Feature Acquisition for Robust Multimodal Emotion Recognition with Missing Modalities

[3] FedGKD: Towards Heterogeneous Federated Learning via Global Knowledge Distillation

[4] FL-FD: Federated learning-based fall detection with multimodal data fusion

- The reduction in the number of parameters, communication, and computation overhead presented in the paper stems from the use of early fusion. This is a contribution of early fusion (and this paper is not the first to apply early fusion in multimodal FL, see [4]) and is not directly related to the methods proposed (i.e., MIRL, MQAA, and GAKD).

- The proposed methods lack generality and are confined to one-dimensional time-series data. The early fusion approach used in the paper requires that the data dimensions of different modalities be consistent. The four datasets selected for evaluation, such as IMU, EMG, and ECG, consist of one-dimensional data vectors. The methods proposed in this paper may not work with data that have varying dimensions (e.g., images), and the paper does not discuss the feasibility or performance of reshaping other types of data.

- The paper does not clarify the roles of entropy in reflecting client modality updates and client modality quality. Is entropy indicating data quality, modality quality, or model update quality? Why not simply calculate the weight based on the number of modalities? Since in Figure 8, they are inversely correlated.

- Typical modality fusion strategies include three approaches, yet the paper neglects late fusion and thus should compare against multimodal FL frameworks that employ late fusion.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: See above.
soundness: 2
strengths: 1. The paper is well-organized.
2. The proposed method achieves considerable performance.
summary: This paper proposes FLISM to tackle incomplete modalities efficiently. FLISM is an efficient FL training algorithm with incomplete sensing modalities while maintaining high accuracy. In particular, FLISM introduces three techniques to solve the corresponding issues. It utilizes contrastive learning to learn modality-invariant representations, utilizes entropy to weight-prioritize contributions, and utilizes knowledge distillation to reduce local update shifts caused by modality differences.
weaknesses: 1. The proposed method seems to be general and not particularly related to time-series healthcare sensing. Could you add experimental comparisons with the Harmony method under the settings of the Harmony paper in the corresponding experimental section? 
2. The contributions of the proposed method appear insufficient; the combination of missing modalities and the introduced techniques are not particularly strong. For example, entropy may also represent data complexity, so the choice based on entropy might not necessarily be better. Could invariant representation learning based on comparisons lead to decreased discriminative ability, negatively affecting certain modalities, especially the missing ones? Distillation learning could perhaps be interpreted as personalization or something else. The three challenges also lack corresponding experimental evidence and analysis. In summary, the entire method seems to combine existing technologies, resembling a general engineering solution that might still be applicable in other scenarios. It would be best to add some theoretical explanations and connections to incomplete modalities. 
3. The compared early fusion methods are relatively old, traditional federated learning approaches unrelated to modality missing. Could you compare some SOTA relevant methods? 
4.  When modality data is severely missing, the performance and efficiency seem to be worse than the Harmony method. 
5. Is the proposed method effective across different backbones? It would be helpful to supplement results on newer architectures like transformers and Mamba. 
6. It would be best to increase analysis to demonstrate the robustness and specificity of the proposed method, such as visual analysis to show that the method genuinely addresses issues caused by modality missing, rather than relying on general tricks, and sensitivity analysis to demonstrate method robustness.