<figure><img src="image_1.png" /></figure>

HAL
open science

# A Survey on Multimodal Federated Learning

Silvia Grosso, Nawel Benarba, Sara Bouchenak, Roberto Esposito, Mirko Polato

<figure><img src="image_2.png" /></figure>

## ► To cite this version:

Silvia Grosso, Nawel Benarba, Sara Bouchenak, Roberto Esposito, Mirko Polato. A Survey on Multi-modal Federated Learning. PEPR-SanteNum 2025 - Journées Annuelles du PEPR Santé Numérique, Oct 2025, Lille, France, Oct 2025, Lille, France. hal-05366356

HAL Id: hal-05366356

https://hal.science/hal-05366356v1

Submitted on 14 Nov 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

Copyright
# A Survey on Multimodal Federated Learning

Silvia Grosso¹,², Nawel Benarba¹, Sara Bouchenak¹, Roberto Esposito², Mirko Polato²
¹INSA Lyon, France – ²University of Turin, Italy

<figure><img src="image_3.png" /></figure>

## Background

**Federated Learning** enables multiple clients to train a global model without sharing raw data.

- Each client updates $\theta_k$ on its local dataset $D_k$

- The server aggregates via a weighted average, such as

$$ \theta = \sum_{k=1}^{K} \frac{|D_k|}{|\bigcup_{k=1}^{K} D_k|} \cdot \theta_k $$

In **Multi-Modal Federated Learning** the main objective is to minimize the global loss function

<figure><img src="image_4.png" /></figure>

$$ \min_{\theta} \mathcal{L}(\theta) = \min_{\theta} \sum_{k=1}^{K} \omega_k \sum_{i=1}^{|D_k|} \mathcal{L}_k(\mathbf{x}_k^i, \mathbf{y}_k^i; \theta) $$

- $\{C_k\}_{k \in [K]}$ is the set of clients

- $D_k = \{(\mathbf{x}_k^i, \mathbf{y}_k^i)\}_{i=1}^{|D_k|}$ is the dataset owned by $C_k$

- $\mathbf{x}_k^i = (\mathbf{x}^{i,m_1}_k, \mathbf{x}^{i,m_2}_k, \dots, \mathbf{x}^{i,m_{M_k}})$ is the $i$-th multimodal sample associated with $\mathbf{y}_k^i$

<figure><img src="image_5.png" /></figure>

## Why MMFL for Healthcare?

- **Distributed system:** split computation, memory, and communication across sites

- **Privacy first:** keep personal and sensitive data safe

- **Multimodal intelligence:** integrate images, text, signals, and more for smarter insights

<figure><img src="image_6.png" /></figure>

## Proposed MMFL Taxonomy

<figure><img src="image_7.png" /></figure>

- Do all clients use the same modalities and model architectures?

- Who manages multimodality: clients, the server, or both?

- How many and how well are data locally annotated?

- What is the level of statistical heterogeneity across local datasets?

### Homogeneous MMFL Systems

$$ \left\{ \begin{array}{l} \mathcal{M}_k \equiv \mathcal{M}_{k'} \equiv \mathcal{M} \\ \Theta_k^m = \Theta_{k'}^m \end{array} \right. \quad \forall k, k' \in [K], \forall m \in \mathcal{M} $$

<figure><img src="image_8.png" /></figure>

### Heterogeneous MMFL Systems

$$ \left\{ \begin{array}{l} \exists k \neq k' \in [K] \\ \exists k \neq k' \in [K], \exists m \in \mathcal{M}_k \cap \mathcal{M}_{k'} \end{array} \right. \quad : \mathcal{M}_k \neq \mathcal{M}_{k'} \\ \Theta_k^m \neq \Theta_{k'}^m \end{array} \right. $$

<figure><img src="image_9.png" /></figure>

### Hybrid MMFL Systems

Same model architectures, different set of modalities

$$ \left\{ \begin{array}{l} \exists k \neq k' \in [K] \\ \forall k, k' \in [K], \forall m \in \mathcal{M}_k \cap \mathcal{M}_{k'} \end{array} \right. \quad : \mathcal{M}_k \neq \mathcal{M}_{k'} \\ \Theta_k^m = \Theta_{k'}^m \end{array} \right. $$

<figure><img src="image_10.png" /></figure>

Same set of modalities, different model architectures

$$ \left\{ \begin{array}{l} \forall k, k' \in [K] \\ \exists k \neq k' \in [K], \exists m \in \mathcal{M} \end{array} \right. \quad : \mathcal{M}_k \equiv \mathcal{M}_{k'} \equiv \mathcal{M} \\ \Theta_k^m \neq \Theta_{k'}^m \end{array} \right. $$

<figure><img src="image_11.png" /></figure>

<figure><img src="image_12.png" /></figure>

## Existing Works

- Mostly **homogeneous** or **hybrid** MMFL systems

- Heterogeneous **data sources**: electronic health records, medical imaging, wearable sensors, genomic data

- **Public datasets:** ISIC 2020, IU-XRay, BraTS 2020/2021, MIMIC-III, Sleep-EDF, WESAD, LiTS17, CheXpert, fastMRI, COVID-19 image data collection, VQA-RAD

- **Main tasks:**
  - Disease detection (e.g., melanoma, COVID-19)
  - Image segmentation (brain/liver tumors)
  - Medical report generation
  - Clinical outcome prediction
  - Neuroimage reconstruction
  - Emotion & mental state monitoring (depression, fatigue, sleep stage)

<figure><img src="image_13.png" /></figure>

## Open Challenges

### Multimodality Selection

- identify useful vs. detrimental modalities

- dynamic selection to reduce cost and balance contributions

### Model Architecture Heterogeneity

- support diverse institutional resources

- reuse local models without reconfiguration

### Communication & Computation Efficiency

- multimodal healthcare data (e.g., CT scans, ECGs, health records) is heavy to transmit and process

### Fairness

- modality-specific bias exacerbated by the multimodal interaction

- ensure performance- and contribution-aware fairness

### Interpretability

- understand modality contributions, cross-modal interactions, and decision processes

- supports trust, accountability, and debugging

### Privacy & Robustness

- adversaries may exploit multimodal interactions for inference or poisoning attacks

- explore decentralized MMFL approaches further

Published in

<figure><img src="image_14.png" /></figure>

<figure><img src="image_15.png" /></figure>