X-Fi: A Modality-Invariant Foundation Model for Multimodal Human Sensing

## Abstract

Abstract Human sensing, which employs various sensors and advanced deep learning technologies to accurately capture and interpret human body information, has significantly impacted fields like public security and robotics. However, current human sensing primarily depends on modalities such as cameras and LiDAR, each of which has its own strengths and limitations. Furthermore, existing multi-modal fusion solutions are typically designed for fixed modality combinations, requiring extensive retraining when modalities are added or removed for diverse scenarios. In this paper, we propose a modality-invariant foundation model for all modalities, X-Fi, to address this issue. X-Fi enables the independent or combinatory use of sensor modalities without additional training by utilizing a transformer structure to accommodate variable input sizes and incorporating a novel “X-fusion” mechanism to preserve modality-specific features during multimodal integration. This approach not only enhances adaptability but also facilitates the learning of complementary features across modalities. Extensive experiments conducted on the MM-Fi and XRF55 datasets, employing six distinct modalities, demonstrate that X-Fi achieves state-of-the-art performance in human pose estimation (HPE) and human activity recognition (HAR) tasks. The findings indicate that our proposed model can efficiently support a wide range of human sensing applications, ultimately contributing to the evolution of scalable, multimodal sensing technologies.

## 1 Introduction

Human sensing refers to using various sensors to capture human body information in a specific space, such as presence (Nguyen et al., ; Nanzer, ), activity (Vrigkas et al., ; Yang et al., )), and pose (Toshev & Szegedy, ; Cai et al., ). Benefiting from advanced deep learning technology, human sensing has made significant progress in both accuracy and granularity, expanding applications in many domains. For instance, video surveillance (Elharrouss et al., ) is empowered by human sensing algorithms to analyze dangerous behaviors. In the field of robotics, human pose helps robots comprehend human actions and then realize human-robot collaboration (Gao et al., ).

Currently, human sensing tasks mainly rely on vision-based modalities like cameras (Ionescu et al., ), which face inherent limitations such as reliance on illumination, privacy concerns, and insufficient 3D information (Cai et al., ). Alternative modalities like LiDAR, mmWave radar, and WiFi have be introduced to overcome these challenges (Nirmal et al., ). Each modality has distinctive advantages but also has limitations, e.g., LiDAR offers high-resolution data but is expensive (Li et al., ), mmWava radar is cost-effective and informative but lacks sensitivity to static objects (An & Ogras, ), and WiFi is ubiquitous and privacy-preserving but has low resolution (Yang et al., ; Zhou et al., ). Therefore, single-modal sensing solutions cannot fit all scenarios, and a multi-modal approach that leverages the strengths of each modality is essential for future advancements in human sensing.

Numerous methods have been proposed for multi-modal perception based on sensor fusion. They usually predefine a fixed set of modalities for a specific scenario, e.g., combining RGB and LiDAR for pose estimation in autonomous driving to enhance long-range all-day sensing. Nevertheless, in any given scenario, once the model is trained, adding or removing even one modality requires a huge effort: adjusting the network and retraining it from scratch. In the real world, we may require versatile combinations of sensor modalities according to different scenarios (Yang et al., ).
Hence, we contemplate whether it is possible to design a one-for-all solution for modality-invariant human sensing. Such a model would require training only once, allowing all sensor modalities that participated in the training process to be utilized independently or in any combination for a wide range of potential applications.

Figure: Figure 1: The left image shows existing modality-corresponding human sensing solutions, while the right image illustrates our proposed modality-invariant foundation model, X-Fi, which enables modality-invariant human sensing.
Refer to caption: x1.png

To achieve modality-invariant human sensing, we aim to design a foundation model that can dynamically handle varying numbers of modalities. First, considering the arbitrary number of input modalities, we naturally resorted to a transformer structure due to its adaptability to variable input sizes. Second, recognizing that each modality has unique characteristics, we adopt modality-specific feature extractors to capture distinctive information. Finally, to retain each modality’s distinct features during multimodal fusion, we introduced a cross-attention fusion mechanism that reduces the information loss caused by multi-modal feature compression and fusion process.
Surprisingly, this design not only supports dynamic inputs from various modalities but by training with diverse modality combinations to perform sensing tasks, it learns complementary features among modalities as well. As a result, it even achieves superior performance against conventional modality fusion methods such as feature-level and decision-level fusion.

To this end, we propose a novel modality-invariant foundation model, X-Fi, for versatile human sensing. X-Fi can take in any combinations of modalities and activate corresponding parts to extract modality-specific features. A cross-modal transformer is designed to learn a cross-modal feature and each modality information is then preserved in the representation by executing cross-attention multi-modal fusion processes to preserve distinctive modal features. As shown in Figure.2, we have a set of modality-corresponded feature encoders and key-value generators to extract, fine-tune and preserve modality-specific features. For versatile combinations of sensor modalities, corresponding feature encoders are activated to obtain modality-specific features of each modality. Subsequently, the attention-based key-value generators are trained to produce key-value pairs, which contain more task-related information derived from modality-specific features (Vaswani et al., ). For the modality-invariant multi-modal fusion process, we introduce “X-fusion” block. It consists of a cross-modal transformer and a set of modality-correlated cross-attention transformers. The cross-modal transformer is designed to adaptively learn a unified embedding across variable-sized multi-modal embedding, which comprises the features of each modality. The generated cross-modal embedding serves as the query, along with key-value pairs of each modality, are sent to the corresponding cross-attention transformer to perform attention-based feature injection. We evaluated X-Fi on human pose estimation (HPE) and human activity recognition (HAR) tasks in MM-Fi (Yang et al., ) and XRF55 (Wang et al., ), demonstrated that X-Fi surpasses previous methods by MPJPE 24.8% and PA-MPJPE 21.4% on HPE task, 2.8% on HAR task.

The main contributions are summarized as follows.
First, we proposed X-Fi, a foundation model for modality invariant human sensing, which has strong scalability that only needs to be trained once and could support dynamic inputs from various modalities for a wide range of sensing scenarios. Second, we further developed X-Fusion block to empower efficient multi-modal fusion under modality invariant training strategy by preserving modality-specific information and applying token fusion technique to inject back into the cross-modal embedding. Third, we conducted extensive experiments on the MM-Fi and XRF55 datasets, utilizing a total of six distinct modalities, and observed that our proposed X-Fi achieved state-of-the-art performance on both the HPE and HAR tasks.

## 2 Related Work

### 2.1 Human Sensing

Human sensing has been extensively explored for applications such as autonomous driving and robotics in recent years, aiming to capture, analyze, and interpret human body information through various sensors and novel deep learning frameworks (Cai et al., ). Most human sensing approaches could generally be split into three catgories: i) Vision-based approaches, where human body information is extracted from visual data captured by cameras (Shahroudy et al., ) and leveraged by deep learning architectures like convolutional structures (Chun et al., ; Pavlakos et al., ; Sun et al., ; Li et al., ) and transformer (Goel et al., ; Cai et al., ; Qiu et al., ). ii) Sensor-based approaches, using alternative modalities such as LiDAR (Ren et al., ), mmWave radar (Wang et al., ; An & Ogras, ), and WiFi CSI (Yang et al., ; Zhou et al., ) to overcome limitations of vision-based methods and expand applicability. iii) Multimodal fusion approaches, including decision-level fusion (An et al., ; Yang et al., ) and feature-level fusion (Zheng et al., ; Chen et al., ), which combine each modality’s strengths to improve sensing performance.

### 2.2 Modality-Invariant Methods

Current human sensing methods lack broad applicability. Both decision-level fusion approaches (Yang et al., ) and feature-level fusion approaches (Zheng et al., ; Chen et al., ) are typically tailored to fixed modality combinations for specific applications. For example, (Zheng et al., ) developed a specific heatmap sampling structure to fuse 2D heatmaps from RGB images with projected 2D point clouds from LiDAR. Similarly, (Chen et al., ) employed modality-specific backbones to extract grid features from images and cluster features from mmWave point clouds, combining them via a global integration module to produce fused features.

In fields beyond human sensing, several studies on multimodal fusion seek to accommodate diverse sensor inputs to a unified model for various applications. Some methods align all input modalities with a primary modality (Man et al., ; Han et al., ), like OneLLM, which employs a universal encoder and projection module to tokenize eight modalities and align the resulting multimodal tokens with language using large language models (Han et al., ), but these approaches heavily rely on the presence of the primary modality. Some other methods, such as (Xue & Marculescu, ), design a gating network to identify the active input modalities and subsequently activate the modality-combination-specific expert networks for multimodal fusion. Yet, this approach is struggled by the necessity of creating numerous expert networks to accommodate a wide range of potential modal combinations in the presence of an extensive sensor options. Other methods like (Memmel et al., ) utilize a transformer-based structure to process tokenized input and also introduce a modality-invariant training strategy by dropping modalities during training. However, this approach is only applicable when the input modalities are limited to vision-based modalities.

Figure: Figure 2: The architecture of the proposed modality-invariant foundation model, X-Fi. The X-Fi consists of a modality feature encoding module that extracts modality-specific features, and a X-Fusion modules that learns an unified cross-modal feature through modality-fusion in the Cross Modal Transformer and preserves distinctive modal features through feature injection in modality-specified Cross Attention Transformers.
Refer to caption: x2.png

## 3 Modality-Invariant Foundation Model for Human Sensing

Assume that input $X$ comprises a random combination of modalities within a specific range, including any number and type of modalities. Our objective is to learn a unified representation $Emb_{cm}$ that integrates modality-specific features from $X$ by using a modality-invariant foundation model $\mathcal{F}$. This representation can be mapped to various down-stream human sensing task spaces by using corresponding task-specific functions $g_{t_{i}}$, such that the task-specified loss function $\mathcal{L}_{t_{i}}(g_{t_{i}}(\mathcal{F}(X));y_{t_{i}})$ is minimized. Thus we propose X-Fi, a modality-invariant human sensing framework designed to map the arbitrary multimodal input $X$ to any task-specific target domain $t_{i}$, such as HPE and HAR.

As illustrated in Fig 2, given $N$ modalities in the dataset, we first prepare a set of modality-specified feature encoders ${\left\{Enc_{i}\left(\cdot;\theta_{Enc,i}\right)\right\}}_{i=1}^{N}$ to extract features ${\left\{F_{i}\in\mathbb{R}^{n_{f}\times d_{f}}\right\}}_{i=1}^{N}$ form corresponding modality inputs ${\left\{x_{i}\right\}}_{i=1}^{N}$. Then we introduce cross-attention-based multi-modal fusion design that could efficiently preserve modality-specific features in the learned unified cross-model embedding. The modality-specified key-value generators ${\left\{G_{kv,i}\left(\cdot;\theta_{G,i}^{kv}\right)\right\}}_{i=1}^{N}$ are applied to encode the modality information from ${\left\{F_{i}\right\}}_{i=1}^{N}$ into key-value pairs ${\left\{(K_{i},V_{i})\right\}}_{i=1}^{N}$. To obtain an unified cross modal embedding $Emb_{cm}\in\mathbb{R}^{n_{f}\times d_{f}}$, we use a cross modal transformer $G_{cm}\left(\cdot;\theta_{G}^{cm}\right)$ to learn from the $Emb_{mm}\in\mathbb{R}^{(n_{f}*N)\times d_{f}}$. The generated cross modal embedding $Emb_{cm}$ serves as the query $Q$, together with each key-value pair, to send into the modality-corresponded cross attention transformer ${\left\{G_{ca,i}\left(\cdot;\theta_{G,i}^{ca}\right)\right\}}_{i=1}^{N}$ in order to inject modality-specific information. Through iterative learning of the $Emb_{cm}$ and performing modality-specific information injection, the final produced $Emb_{cm}$ could be used for various downstream tasks via task-correlated Multi-layer Perceptron (MLP) head.

In the following subsections, we will decompose X-Fi by describing the modality feature encoding (sec. 3.1), the attention-based X-Fusion (sec. 3.2), and the modality-invariant training (sec. 3.3).

### 3.1 Modality Feature Encoding

In human sensing,
different modalities exhibit distinctive characteristics, requiring unique feature encoder structures to extract robust modality-specific features. For instance, images are represented as a 2D matrix and have rich spatial feature, requiring models like CNN (LeCun et al., ) and ViT (Dosovitskiy et al., ) to extract vision features. While point clouds from LiDAR consist of a set of unordered points in Euclidean space, necessitating point cloud-specific frameworks like Point Net (Qi et al., ) and Point Transformer (Zhao et al., ) to process them. Thus, we introduce modality-specific feature encoders in X-Fi to extract features from various modalities.

In X-Fi, the modality-specified feature encoder $Enc_{i}\left(\cdot;\theta_{Enc,i}\right)$ for each modality consists of a pretrained feature extractor $E_{i}\left(\cdot;\theta_{E,i}\right)$ and a linear projection head. The feature extractors are responsible for extracting informative high-dimensional representations from modal data. To accommodate diverse combinations of sensor modalities across different application scenarios, we can select any model structures as backbones for these feature extractors. Each feature extractor is modality specified, which allows us to choose a model structure that best aligns with the unique characteristics of the modality’s data, thereby ensuring the efficiency of feature extraction. For example: for point clouds from mmWave radar and LiDAR, we adopt the Point Transformer backbone (Zhao et al., ). The self-attention mechanism in Point Transformers is particularly suitable for point cloud data due to its permutation invariance and ability to dynamically focus on relevant points.

Since the data structures of the features extracted by different modality-specific feature extractors are still different across modalities, we use a set of linear projection structures to unify these features into the same dimensional space,allowing them to be used together in subsequent stages of the model. Finally, the positional encoding is added to obtain the final features.

$$ $\displaystyle\left\{Enc_{i}(\cdot;\theta_{Enc,i})\right\}_{i=1}^{N}=\left\{ \text{LinearProjection}_{i}(E_{i}(\cdot;\theta_{E,i});\theta_{LP,i})\right\}_{ i=1}^{N}$ (1) $$

### 3.2 Attention-based X-Fusion

After obtaining the modality-specific features, a natural question is posed: How can the modality-invariant foundation model effectively combine these features to obtain a unified cross-modal representation that is applicable for downstream human sensing tasks?
Thus, in this section, we will interpret the proposed X-Fusion structure which consists of a cross-modal transformer to learn an unified cross-modal feature and cross-attention multi-modal fusion modules to preserve distinctive modality features.

#### 3.2.1 cross-modal transformer

In this subsection, we aim to learn a unified cross-modal feature. The proposed structure should be capable of dynamically handling varying numbers of modality features and effectively achieving information interaction among diverse modal features. Thus, we adopt a Transformer-based structure (Vaswani et al., ) that can accept variable sizes of input. Also, the self-attention mechanism can equally learn the long-distance dependencies between any two modal features in the input, thereby effectively modeling global information for all modal features.

First, we concatenate all obtained modality-specific features to initialize a multi-modal embedding $Emb_{mm}\in\mathbb{R}^{(n_{f}*N)\times d_{f}}$. As illustrated in the Figure. 2, we then linearly map the multi-modal embedding into three vectors, named $Q_{mm},K_{mm},V_{mm}$. Next, we perform dot-product-based multi-head attention to calculate the attention value which models the dependencies among modality features.

Notably, The multi-head attention and feed-forward blocks output embedding of the same size as the input, resulting in variable-sized cross-modal embedding based on the number of modality inputs. This variability complicates the design of subsequent fusion blocks and task-specific projection heads which have fixed-size parameters. To address this issue, we apply adaptive pooling both after the multi-head attention block and within the residual connection to perform down sampling and obtain unified-sized outputs $Z_{mm}\in\mathbb{R}^{n_{f}\times d_{f}}$. Finally, we feed $Z_{mm}$ into a feed-forward structure with a residual connection to obtain the unified cross-modal embedding $Emb_{cm}\in\mathbb{R}^{n_{f}\times d_{f}}$.

$$ $\displaystyle Z_{mm}=\text{AvgPool}(\text{MultiHead}\left(Q_{mm},K_{mm},V_{mm} \right))+\text{AvgPool}(Emb_{mm})$ (2) $\displaystyle Emb_{cm}=FFN(Z_{mm})+Z_{mm}$ (3) $$

#### 3.2.2 cross-attention multi-modal fusion

However, the basic transformer design, which has a layered architecture with sequentially stacked transformer blocks, often leads to a prioritization of dominant modalities. First reason is that the self-attention mechanism in the cross-modal transformer tends to assign higher attention weights to the more informative modality-specific features (Lu et al., ), causing significant information loss from less dominant modalities. Secondly, in layered structures, different layers focus on distinct aspects of the input data, and later layers naturally exerting a greater influence on the overall output. As a result, the modalities emphasized by the later layers tend to be disproportionately prioritized. To efficiently and equally fuse features from any combination of modalities, we develop a cross-attention multi-modal fusion architecture that ensures the model pays equal attention to every input modality by incorporating modality-specific information into the cross-modal embedding. We also design an X-Fusion process that iterates on the same block, compelling the X-Fusion block to identify a unified dynamic attention allocation method that effectively preserves the features of each modality and optimally integrates them to achieve the most optimal solution.

##### Key-Value Generators.

To prepare key-value pairs that preserve modality-specific information, we design key-value generators for each modality. take the i-th modality for illustration, key-value generator $G_{kv,i}$ receives modality specific feature $F_{i}\in\mathbb{R}^{n_{f}\times d_{f}}$ extracted from feature extractor $E_{i}$ as the input. $F_{i}$ is first passed through a two-layer MLP with a hidden dimension of $d_{f}*2$ incorporating a Rectified Linear Unit (ReLU) activation (Nair & Hinton, ) and layer normalization in between. The MLP output is then projected to key $K_{i}$ and value $V_{i}$ using separate linear mapping layers. Once the key-value pair $(K_{i},V_{i})$ is generated from $F_{i}$, it remains unchanged during iterations in the X-fusion block to prevent information leakage between modalities.

##### Cross-Attention Transformers.

We employ a series of cross-attention transformers to integrate modality-specific information into the cross-modal embedding $Emb_{cm}$. Each transformer is associated with a specific modality and receives three inputs: the unified cross-modal embedding $Emb_{cm}$ that is shared across all cross-attention transformers, the key ($K$) and value ($V$) pairs obtained from modality-specific key-value generators. Take the i-th modality as an example, $Emb_{cm}$ serves as the query ($Q$) in the multi-head attention mechanism. Attention weights are computed based on $Q$ and $K_{i}$ to scale the modality-specific value $V_{i}$, yielding an intermediate result denoted as $O_{i}$. Subsequently, $Emb_{cm}$ is residual added to $O_{i}$ to retain cross-modal features, resulting in $O_{i}^{{}^{\prime}}$. The final output $F^{\prime}_{i}$ of each modality-specific cross-attention transformer is produced by passing $O_{i}^{{}^{\prime}}$ through a feed-forward structure with a residual connection. Each $F^{\prime}_{i}$ generated from modality-specific cross-attention transformer encapsulates fused information focusing on a single modality. To integrate information from all modalities, we concatenate all ${\left\{F^{\prime}_{i}\right\}}_{i=1}^{N}$ to get the updated multi-modal embedding $Emb_{mm}^{{}^{\prime}}$, which is used in the subsequent X-fusion block iteration.

$$ $\displaystyle Emb_{mm}^{{}^{\prime}}=\text{Concat}{\left\{G_{ca,i}\left(Emb_{ cm},K_{i},V_{i};\theta_{G,i}^{ca}\right)\right\}}_{i=1}^{N}$ (4) $$

### 3.3 Modality-Invariant Training

To train our modality invariant foundation model, we design an explicit training strategy with reference to (Memmel et al., ). First, to ensure that the feature extractors retain their ability to capture modality-specific features without being biased, we pre-train the parameters on the respective datasets and freeze these parameters during subsequent training. Next, to simulate versatile combinations of sensor modalities in different scenarios, we use a modality existence list containing $n$ elements to indicate whether each modality is present in the input modal combination. This modality existence list is utilized to control the presence of input modalities during each training iteration and activate the corresponding modality-specific structures in the X-Fi. In addition, each modality is assigned an independent occurrence probability $p_{i}$. By adjusting the modality-corresponded $p_{i}$, we can refine the model’s training strategy by prioritizing certain modalities. For example, because LiDAR point cloud data is inherently sparser than image data and exhibits complex features such as permutation invariance, it is necessary to include more LiDAR samples in the training process to ensure accurate model fitting. Since the training sample is controlled by the modality existence list, we can model the training samples by the distribution of the modality existence list. We denote the count of occurrences of i-th modality in m training iterations by a variable $K_{i}$, and $K_{i}\sim Binomial(m,p_{i})$. Since $K_{i}$ for each modality is independent, we can describe the distribution using a joint probability mass function $P(K_{1}=k_{1},\cdots,K_{n}=k_{n})=\prod_{i=1}^{n}\binom{m}{k_{i}}p_{i}^{k_{i}}
(1-p_{i})^{m-k_{i}}$. For each training batch, the modality existence list is sampled from a distribution to regulate the input modalities, thereby influencing the activation of modality-specific model structures.

## 4 Experiments

### 4.1 Experimental Setup

##### Datasets.

We train and evaluate our proposed X-Fi on two largest human sensing multimodal public datasets MM-Fi (Yang et al., ) and XRF55 (Wang et al., ), to assess its efficiency as a unified modality-invariant foundation model across diverse human sensing tasks, including Human Pose Estimation (HPE) and Human Activity Recognition (HAR). The details of the specific datasets utilized for each task are provided as follows. (i) HPE: MM-Fi consists of over 320k synchronized frames from 40 human subjects. Each synchronized frame consists of samples collected from 5 sensing modalities, including RGB images (I), Depth images (D), LiDAR point clouds (L), mmWave point clouds (R), and WiFi-CSI (W). The chosen annotation type is 3D whole-body keypoints, representing the positions of 17 human joints. (ii) HAR: MM-Fi and XRF55 are utilized for HAR task evaluation. MM-Fi consists of 27 action categories, including 14 daily activities and 13 rehabilitation exercises. We exclude WiFi-CSI modality from MM-Fi when addressing the HAR task, as each WiFi data sample spans only 100ms, which is insufficient to effectively capture the distinct data patterns associated with different actions. XRF55 includes 42.9K radio frequency samples collected by 3 sensing modalities, including mmWave Range-Doppler & Range-Angle Heatmaps (R), WiFi-CSI (W), and RFID phase series data (RF). XRF55 comprises 55 action classes collected from 39 human subjects, covering categories such as fitness activities and human-computer interactions.

##### Training Objective.

Different loss functions are applied as the training objectives for different human sensing tasks. For HPE, Mean-Squared-Error is chosen as the loss function, which measures the average squared euclidean distance between each of the 17 predicted whole-body key-points and their corresponding ground truth points in 3D space. For HAR, Cross-Entropy loss is applied between the predicted probability distribution and the ground truth distribution.

**Table 1: Performance comparisons of X-Fi with baseline methods on the MM-Fi dataset for HPE task. “Baseline1” denotes the decision-level fusion results and “Baseline2” denotes the feature-level fusion results. Imp denotes the improvement achieved over baseline in percentage level.**
| MM-Fi | MPJPE | PA-MPJPE |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| modality | X-Fi | Baseline1 | Imp$\downarrow$ | Baseline2 | Imp$\downarrow$ | X-Fi | Baseline1 | Imp$\downarrow$ | Baseline2 | Imp$\downarrow$ |
| I | 93.9 | 121.0 | 22.4% | 121.0 | 22.4% | 60.3 | 68.0 | 11.4% | 68.0 | 11.4% |
| D | 101.8 | 102.4 | 0.6% | 102.4 | 0.6% | 48.4 | 52.7 | 8.2% | 52.7 | 8.2% |
| L | 167.1 | 161.5 | -3.5% | 161.5 | -3.5% | 103.2 | 103.5 | 0.3% | 103.5 | 0.3% |
| R | 127.4 | 141.3 | 9.8% | 141.3 | 9.8% | 69.8 | 72.4 | 3.7% | 72.4 | 3.7% |
| W | 225.6 | 227.1 | 0.7% | 227.1 | 0.7% | 105.3 | 108.0 | 2.5% | 108.0 | 2.5% |
| I+D | 86.1 | 96.7 | 11.0% | 109.2 | 21.2% | 48.1 | 53.9 | 10.7% | 57.1 | 15.6% |
| I+L | 93.0 | 122.1 | 23.8% | 121.2 | 23.3% | 59.7 | 75.6 | 21.1% | 73.0 | 18.2% |
| I+R | 88.8 | 101.0 | 12.1% | 140.6 | 36.8% | 57.3 | 57.3 | 0.0% | 68.2 | 16.0% |
| I+W | 93.0 | 146.4 | 36.5% | 156.7 | 40.7% | 59.5 | 77.3 | 23.1% | 86.3 | 31.1% |
| D+L | 102.5 | 111.7 | 8.2% | 108.0 | 5.1% | 48.4 | 68.8 | 29.7% | 55.8 | 13.3% |
| D+R | 98.0 | 94.1 | -4.1% | 116.4 | 15.8% | 47.3 | 51.8 | 8.6% | 57.0 | 17.0% |
| D+W | 101.8 | 141.7 | 28.1% | 155.5 | 34.5% | 48.1 | 71.4 | 32.6% | 81.5 | 41.0% |
| L+R | 109.8 | 116.4 | 5.7% | 137.0 | 19.9% | 63.4 | 71.5 | 11.3% | 71.9 | 11.8% |
| L+W | 159.5 | 167.1 | 4.5% | 206.2 | 22.6% | 102.7 | 100.7 | -2.0% | 109.3 | 6.0% |
| R+W | 117.2 | 144.2 | 18.7% | 141.3 | 17.0% | 62.7 | 72.2 | 13.0% | 77.6 | 19.1% |
| I+D+L | 84.8 | 102.9 | 17.5% | 104.5 | 18.8% | 48.2 | 62.3 | 22.7% | 61.2 | 21.4% |
| I+D+R | 83.4 | 86.7 | 3.8% | 113.4 | 26.5% | 47.3 | 50.1 | 5.4% | 58.0 | 18.4% |
| I+D+W | 85.3 | 118.6 | 28.0% | 137.8 | 38.1% | 48.1 | 63.7 | 24.4% | 75.3 | 36.1% |
| I+L+R | 88.4 | 101.4 | 12.8% | 135.0 | 34.5% | 57.2 | 62.7 | 8.8% | 67.7 | 15.6% |
| I+L+W | 93.0 | 134.9 | 31.1% | 156.2 | 40.5% | 59.7 | 82.0 | 27.2% | 84.8 | 29.6% |
| I+R+W | 88.5 | 116.5 | 24.1% | 128.9 | 31.3% | 57.1 | 63.1 | 9.5% | 75.0 | 23.9% |
| D+L+R | 96.0 | 95.3 | -0.8% | 111.3 | 13.7% | 47.3 | 58.6 | 19.2% | 58.4 | 19.0% |
| D+L+W | 102.0 | 130.7 | 22.0% | 154.6 | 34.0% | 48.1 | 78.1 | 38.4% | 84.1 | 42.8% |
| D+R+W | 97.0 | 113.2 | 14.3% | 132.0 | 26.5% | 47.1 | 59.4 | 20.7% | 71.8 | 34.3% |
| L+R+W | 107.4 | 129.6 | 17.2% | 141.6 | 24.1% | 63.1 | 78.1 | 19.2% | 77.0 | 18.1% |
| I+D+L+R | 83.5 | 90.7 | 7.9% | 109.4 | 23.6% | 47.6 | 55.8 | 14.6% | 61.6 | 22.8% |
| I+D+L+W | 86.0 | 116.7 | 26.4% | 134.6 | 36.2% | 48.2 | 70.0 | 31.2% | 75.4 | 36.0% |
| I+D+R+W | 84.0 | 101.9 | 17.6% | 128.6 | 34.7% | 47.6 | 56.3 | 15.5% | 71.3 | 33.3% |
| I+L+R+W | 88.6 | 114.0 | 22.3% | 174.6 | 49.2% | 57.1 | 69.5 | 17.8% | 78.1 | 26.8% |
| D+L+R+W | 97.6 | 110.8 | 11.9% | 147.6 | 33.9% | 47.4 | 66.7 | 28.9% | 73.1 | 35.1% |
| I+D+L+R+W | 83.7 | 103.0 | 18.8% | 128.9 | 35.1% | 47.6 | 62.4 | 23.8% | 72.5 | 34.3% |

##### Baseline Formulation.

To evaluate the performance of our proposed unified modality-invariant foundation model across various human sensing tasks and modality combinations, we formulate baseline models by adopting a feature fusion approach (Das et al., ; Marcard et al., ) and a decision-level fusion approach (Yang et al., ). For the feature fusion approach, first, we leverage the pre-trained modality-specific feature extractors in X-Fi to obtain features from each modality. Subsequently, for single-modal inputs, a MLP structure is applied to project the single-modal features into the task representation space. For multi-modal inputs, we concatenate the features from each modality to create a multi-modal feature, which is then processed by a MLP head with an average pooling layer to fuse and map the multi-modal features into the unified task representation space. We train a baseline model for each modality combination. And for different tasks, we optimize the parameters of each baseline model (including the pre-trained feature extractor) using different learning objectives. For the decision-level fusion approach, we first obtain the final-layer output of each individual modality. Then, we calculate the average of the outputs from all modalities in the combination to generate the fused output. The baseline results are presented in the Table 1& 2.

##### Implementation Details.

We conduct experiments on all humans sensing tasks and datasets with the following model settings. To standardize feature representations obtained by various modality-specific feature extractors, we apply linear projection units to map each modality feature representation to 32 features, each with feature dimension of 512. The positional embedding, derived from LiDAR 3D point cloud raw data due to its inherent capture of human body spatial information, is only provided in the multi-modal embedding if the LiDAR modality is present in the input. The backbone for both the cross-modal transformer and each modality-specific cross-attention transformer consists of a 1-layer decoder-only transformer structure with 8 multi-head attention heads and a scaling factor of 0.125.
Each modality-specific key-value pair and the cross-modal embedding in X-Fusion block comprise 32 features with feature dimension of 512. The AdamW optimizer, with an initial learning rate of $1\times e^{-3}$ for HPE and $1\times e^{-4}$ for HAR, is chosen for model optimization. The training process is performed with a batch size of 16 on an NVIDA GeForce RTX 4090 GPU.

### 4.2 Quantitative Results

We compare our approach with the baseline methods regarding the performance on downstream human sensing tasks, including HPE and HAR. For convenience, we refer to the decision-level fusion baseline as “baseline1”, and the feature-level fusion baseline as “baseline2”. The results for X-Fi, as shown in Tables 1& 2, represent average values obtained from multiple experiments. These values are derived by validating the performance of a single X-Fi model across different modality combinations that is trained using the proposed modality-invariant training strategy.

##### Human Pose Estimation.

We assess the performance of X-Fi on the MM-Fi dataset (Yang et al., ) by adopting two commonly-used metrics in HPE: Mean Per Joint Position Error (MPJPE) and Procrustes Analysis MPJPE (PA-MPJPE). For single-modal inputs, X-Fi improves the averaged MPJPE and PA-MPJPE by $6.0\%$ and $5.2\%$, respectively. For the dual-modality inputs, the averaged MPJPE and PA-MPJPE are improved respectively by $14.5\%$ and $14.8\%$ on baseline1, $23.7\%$ and $18.9\%$ on baseline2. Notably, for LiDAR and WiFi modalities, when used as individual inputs, X-Fi does not achieve good performance, with MPJPE difference of $-3.5\%$ and $+0.7\%$, respectively. However, when the two modalities are combined as inputs, the performance improves significantly, surpassing the baseline1 by $4.5\%$ and the baseline2 by $22.6\%$. For the combined input of three modalities, X-Fi demonstrates an average improvement of $17.0\%$ and $28.8\%$ in MPJPE, $19.5\%$ and $25.9\%$ in PA-MPJPE. For the combined input of four modalities, the results indicate an average enhancement of $17.0\%$ and $35.5\%$ in MPJPE, $21.6\%$ and $30.8\%$ in PA-MPJPE. For all five modalities combined input, MPJPE increases by $18.8\%$ and $35.1\%$, PA-MPJPE increases by $23.8\%$ and $34.3\%$.

**Table 2: HAR performance on the MM-Fi and XRF55 datasets.**
| Modality | Baseline | X-Fi | Imp $\uparrow$ |
| --- | --- | --- | --- |
| MM-Fi |  |  |  |
| I | 25.3% | 26.5% | 1.2% |
| D | 49.9% | 48.1% | -1.8% |
| L | 63.9% | 52.7% | -11.2% |
| R | 85.0% | 85.7% | 0.7% |
| I + D | 45.2% | 45.3% | 0.1% |
| I + L | 23.8% | 35.2% | 11.4% |
| I + R | 66.9% | 73.4% | 6.6% |
| D + L | 49.7% | 51.6% | 1.9% |
| D + R | 74.1% | 79.8% | 5.7% |
| L + R | 85.6% | 88.7% | 3.2% |
| I + D + L | 43.0% | 48.7% | 5.7% |
| I + D + R | 69.1% | 70.7% | 1.6% |
| I + L + R | 68.2% | 77.8% | 9.6% |
| D + L + R | 72.9% | 80.5% | 7.6% |
| I+D+L+R | 72.6% | 72.2% | -0.4% |
| XRF55 |  |  |  |
| R | 82.1% | 83.9% | 1.9% |
| W | 77.8% | 55.7% | -22.1% |
| RF | 42.2% | 42.5% | 0.3% |
| R+W | 86.8% | 88.2% | 1.4% |
| R+RF | 71.4% | 86.5% | 15.1% |
| W+RF | 55.6% | 58.1% | 2.6% |
| R+W+RF | 70.6% | 89.8% | 19.2% |

##### Human Activity Recognition.

The performance of X-Fi on the HAR task is evaluated by classification accuracy on both the MM-Fi (Yang et al., ) and XRF55 (Wang et al., ) datasets. On the MM-Fi dataset, X-Fi performs better than the baseline when handling multi-modal inputs, yielding an average improvement of 2.8%. But when dealing with single-modality inputs, its performance declines, dropping by 2.8%. On the XRF55 dataset, X-Fi only encounters challenges with WiFi-CSI single-modality input. It outperforms the baseline on all other modality combinations, achieving an average improvement of 6.7%. Notably, while X-Fi’s performance is suboptimal when using WiFi as a single input, when combined with other modalities, WiFi not only avoids hindering performance but also introduces unique information that enhances classification accuracy. For example, when X-Fi processes mmWave radar and RFID multimodal input, it outperforms the baseline by 15.1%. With the inclusion of WiFi, this improvement increases to 19.2%.

### 4.3 Analytics

##### Impact of modality existence list.

To assess the impact of adjusting the independent existing probability of each modality in the modality existence list, we conduct experiments with different probability combinations on the XRF55 dataset for the HAR task. The XRF55 modality existence list contains the independent existence probabilities of mmWave Radar, WiFi-CSI, and RFID modalities in sequential order. As shown in Table 3, X-Fi encounters difficulties in dealing with WiFi-CSI. However, through gradually increase the exist probability of WiFi-CSI, the model is prioritised to fit WiFi-CSI modality during training, which yields significant improvement from $29.1\%$ to $55.7\%$ in handling single WiFi-CSI input. Additionally, we are surprised to find that prioritizing a particular modality does not degrade the model’s ability to extract features from the other two modalities. Consequently, the multimodal fusion results are also improved a lot.

| XRF55 | Accuracy (%) |  |  |
| --- | --- | --- | --- |
| ($P_{R}$,$P_{W}$,$P_{RF}$) | (0.5,0.5,0.8) | (0.5,0.7,0.7) | (0.5,0.9,0.6) |
| R | 82.2% | 82.4% | 83.9% |
| W | 29.1% | 37.4% | 55.7% |
| RF | 41.6% | 41.9% | 42.5% |
| R+W | 85.1% | 86.0% | 88.2% |
| R+RF | 84.5% | 84.8% | 86.5% |
| W+RF | 50.3% | 52.7% | 58.1% |
| R+W+RF | 86.4% | 87.3% | 89.8% |

Table: Table 3: Comparisons of X-Fi trained with different modality existence lists. The probabilities are shown in decimal form.

| MM-Fi | MPJPE (mm) | PA-MPJPE (mm) |  |  |
| --- | --- | --- | --- | --- |
| Method | Transformer | X-Fusion | Transformer | X-Fusion |
| I | 99.6 | 93.9 | 65.0 | 60.3 |
| L | 212.2 | 167.1 | 109.6 | 103.2 |
| R | 138.0 | 127.4 | 74.6 | 69.8 |
| W | 243.3 | 225.6 | 112.9 | 105.3 |
| L + R | 123.4 | 109.8 | 69.9 | 63.4 |
| L + W | 216.3 | 159.5 | 112.7 | 102.7 |
| I + L + W | 96.6 | 93.0 | 61.5 | 59.7 |

Table: Table 4: Comparison between transformer-based fusion and X-Fusion on the HPE task..

##### Comparison between transformer-only fusion and X-Fusion.

To assess whether the X-Fusion module’s feature injection mechanism preserves distinctive features from each modality and enhances multi-modal fusion, we replaces X-Fusion in X-Fi with a basic transformer encoder consists of four stacked blocks. We conduct experiments on these variants, with a subset of results analyzed in Table 4 and the full findings provided in the appendix 5. As shown in Table 4, X-Fusion yields better performance on most of the modality combination input. First, when handling single modality input, transformer-based fusion struggles with challenging modalities, whereas X-Fusion effectively captures the distinctive features of each modality. For example, performance with LiDAR input improves by $21.3\%$. Next, in contrast to transformer-based fusion methods, X-Fusion leverages each modality’s strengths, yielding better fusion results than single-modality inputs.
X-Fusion demonstrates superior performance compared to either the LiDAR or WiFi-CSI modality when used individually. Lastly, in the case of multi-modality inputs, the performance of transformer-based fusion is highly dependent on the presence of dominant modalities. For example, adding the RGB modality improves the performance of the LiDAR and WiFi-CSI multi-modal inputs by 119.7 mm with the transformer backbone, whereas the corresponding improvement with X-Fusion is 66.6 mm.

##### Comparison between X-Fusion and its variants.

To evaluate whether the proposed X-Fusion structure, which iterates multiple loops on the same X-Fusion block, is optimal for preserving and fusing multi-modal features, we design two variants. The first employs a layered architecture with four sequentially stacked X-Fusion blocks, where each layer learns new key-value pairs from the multimodal embedding of the previous layer. The second variant is similar, but uses the same key-value pairs across layers, learned from the embedding generated by the modality feature encoder. We conduct experiments on both variants, with full results provided in the appendix 5. On average, the first variant demonstrates a performance improvement of +1.27 mm for MPJPE and -0.7 mm for PA-MPJPE across all modality combinations. The second variant achieves an improvement of +0.93 mm for MPJPE and +0.03 mm for PA-MPJPE. Therefore, considering the computational efficiency and the minor performance differences, we retain the X-Fusion structure where an X-Fusion block is applied iteratively.

### 4.4 Qualitative Results

To intuitively illustrate the performance of X-Fi on the human sensing tasks, we visualize the estimated human skeleton generated by the model in the HPE task in Figure. 3 and employ the t-SNE method  (Van der Maaten & Hinton, ) to visualize the multimodal embedding obtained by the X-fusion block for the HAR task in Figure. 4.

##### Human Pose Estimation.

The visualization results for HPE comprise two actions, ‘picking up things’ and ‘throwing,’ each depicted through a sequence of four images. In different rows, we presented the results for the RGB and depth single-modal inputs, the fused results of the RGB and depth multimodal inputs, and the ground truth. To facilitate a clearer comparison between the fused results and the single-modal inputs results, we incorporated blue and orange dashed lines in the fused result images to represent the results of the RGB and depth single-modality inputs, respectively. Through observation, we found that the RGB modality provides more accurate estimations of leg movements, while the depth modality achieves greater accuracy in estimating hand movements. By combining both modalities into a multi-modal input, X-Fi effectively retains the leg movement accuracy from the RGB modality and the hand movement accuracy from the depth modality, resulting in fused outcomes that are closest to the ground truth.

Figure: Figure 3: Comparison of predicted human skeletons.
Refer to caption: extracted/5936647/figures/HPE_result.png

Figure: Figure 4: Comparison of multi-modal embedding distribution for HAR. The upper right corner of the image provides an enlarged view of the red-boxed area from the original image.
Refer to caption: extracted/5936647/figures/HAR_result.png

##### Human Activity Recognition.

The visualization results for HAR includes the multi-modal embedding distributions from WiFi-CSI and mmWave Radar single-modal inputs, as well as the combined WiFi-CSI and mmWave Radar multi-modal inputs. To more closely analyze the distribution of sample points, we zoomed in on a small region containing points from two distinct categories. To quantify the distribution, we used the Silhouette score and Calinski–Harabasz index as indicators of clustering quality. The Silhouette score assesses how well an object fits within its own cluster compared to others, ranging from -1 to 1, with higher values indicating better separation and cohesion. Similarly, higher Calinski–Harabasz index values reflect more distinct clusters. We observed that the multimodal input results exhibit superior feature discriminability. While the mmWave Radar single-modal results demonstrate good classification capability, a closer inspection of the zoomed-in region reveals that the multimodal input results forms tighter clusters. Furthermore, a comparison of the indicator values confirms that the multimodal results outperforms any single-modal results.

## 5 Conclusion

In this paper, we introduce X-Fi, a modality-invariant foundation model designed to overcome the limitations of current human sensing approaches, which often rely on fixed modality combinations. By leveraging a transformer architecture and a novel X-fusion mechanism, X-Fi supports flexible multimodal integration and effectively utilizes distinctive modal information. Our extensive experiments on the MM-Fi and XRF55 datasets demonstrate the model’s superior performance in both human pose estimation (HPE) and human activity recognition (HAR) tasks. These results highlight X-Fi’s potential to drive advancements in scalable, multimodal human sensing technologies.

## Appendix A Appendix

**Table 5: Full results of the comparison among transformer-based fusion, X-Fusion, and X-Fusion’s variants.**
| MM-Fi | MPJPE | PA-MPJPE |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| X-Fi Variants | Transformer<br>Fusion | X-Fusion<br>(iterative) | X-Fusion<br>(stacked) | X-Fusion<br>(stacked,<br>same kv) | Transformer<br>Fusion | X-Fusion<br>(iterative) | X-Fusion<br>(stacked) | X-Fusion<br>(stacked,<br>same kv) |
| I | 99.6 | 93.9 | 100.6 | \ul95.8 | 65.0 | 60.3 | 62.2 | \ul61.3 |
| D | 109.0 | \ul101.8 | 94.8 | 104.5 | 56.1 | 48.4 | 50.8 | \ul48.8 |
| L | 212.2 | \ul167.1 | 184.9 | 160.2 | 109.6 | 103.2 | 104.7 | \ul103.6 |
| R | 138.0 | 127.4 | 121.2 | \ul122.8 | 74.6 | 69.8 | 65.4 | \ul66.4 |
| W | 243.3 | \ul225.6 | 229.7 | 224.3 | 112.9 | \ul105.3 | 105.4 | 103.2 |
| I + D | 89.7 | 86.1 | 81.1 | \ul84.9 | 52.9 | 48.1 | 50.0 | 48.1 |
| I + L | 96.6 | 93.0 | 98.8 | \ul93.2 | 62.4 | 59.7 | 60.9 | \ul60.2 |
| I + R | 93.0 | 88.8 | 91.9 | \ul89.8 | 59.1 | 57.3 | \ul57.5 | 58.3 |
| I + W | 96.0 | 93.0 | 97.2 | \ul94.0 | 61.9 | 59.5 | 60.9 | \ul60.5 |
| D + L | \ul97.3 | 102.5 | 91.4 | 97.9 | 52.3 | \ul48.4 | 49.5 | 48.0 |
| D + R | \ul94.9 | 98.0 | 88.4 | 96.8 | 50.6 | 47.3 | 48.0 | 47.3 |
| D + W | \ul95.6 | 101.8 | 94.5 | 98.8 | 51.4 | \ul48.1 | 49.6 | 48.0 |
| L + R | 123.4 | \ul109.8 | 115.5 | 106.9 | 69.9 | \ul63.4 | 64.1 | 61.7 |
| L + W | 216.3 | \ul159.5 | 181.7 | 156.4 | 112.7 | \ul102.7 | 103.9 | 102.5 |
| R + W | 126.1 | 117.2 | 120.8 | \ul117.4 | 70.0 | 62.7 | 63.9 | 62.7 |
| I + D + L | 86.8 | 84.8 | 78.6 | \ul84.2 | 52.2 | \ul48.2 | 49.3 | 48.0 |
| I + D + R | 85.2 | \ul83.4 | 78.2 | 83.8 | 50.9 | 47.3 | 48.7 | \ul47.5 |
| I + D + W | 85.6 | 85.3 | 79.6 | \ul84.5 | 51.5 | 48.1 | 49.3 | 48.1 |
| I + L + R | 93.4 | 88.4 | 93.0 | \ul89.7 | 59.1 | \ul57.2 | 57.1 | 58.3 |
| I + L + W | 96.6 | 93.0 | 98.1 | 93.0 | 61.5 | 59.7 | \ul60.1 | 60.2 |
| I + R + W | 93.3 | 88.5 | 91.8 | \ul90.0 | 58.8 | \ul57.1 | 57.0 | 58.4 |
| D + L + R | \ul91.5 | 96.0 | 84.8 | 95.8 | 50.5 | 47.3 | 47.9 | \ul47.4 |
| D + L + W | \ul93.0 | 102.0 | 88.9 | 98.1 | 51.4 | \ul48.1 | 49.2 | 47.9 |
| D + R + W | \ul90.6 | 97.0 | 87.2 | 97.3 | 50.4 | 47.1 | 47.7 | \ul47.3 |
| L + R + W | 122.8 | \ul107.4 | 116.0 | 106.4 | 70.4 | \ul63.1 | 63.7 | 61.3 |
| I + D + L + R | 84.1 | 83.5 | 77.0 | \ul83.2 | 51.2 | 47.6 | 48.3 | 47.6 |
| I + D + L + W | 84.7 | 86.0 | 78.1 | \ul84.2 | 51.8 | \ul48.2 | 48.9 | 48.0 |
| I + D + R + W | \ul83.4 | 84.0 | 77.9 | 83.9 | 50.8 | 47.6 | 48.2 | \ul47.7 |
| I + L + R + W | 93.9 | 88.6 | 92.7 | \ul89.5 | 59.1 | \ul57.1 | 56.8 | 58.4 |
| D + L + R + W | \ul89.4 | 97.6 | 84.4 | 95.8 | 50.7 | 47.4 | 47.7 | 47.4 |
| I + D + L + R + W | \ul82.9 | 83.7 | 76.6 | 83.1 | 51.3 | 47.6 | 47.9 | 47.6 |

### A.1 Feature Extractor Pre-training.

As mentioned in the main work, to effectively extract informative high-dimensional representations from modal data, we designed modality-specific feature extractors whose backbone structures could be selected based on the data structure of each modality and the characteristics of the datasets. Meanwhile, we employ pre-training techniques to preserve the efficiency of feature extraction. Initially, for each modality, we train an entire model on the respective dataset and task. A portion of the model architecture is then repurposed as the feature extractor, with the corresponding pre-trained weights frozen during subsequent training. For instance, a ResNet-18 model was first trained on RGB images from the MM-Fi dataset to perform HPE. Subsequently, the MLP structure of the model’s output layer was discarded, retaining only the bottleneck blocks serving as the feature extractor. To identify efficient backbone structures, we conducted experiments to validate the performance of various backbone structures on each dataset.The finalized backbones are presented in the following. When utilizing MM-Fi for HPE, we employ the ResNet-18 architecture (He et al., ) for both the RGB and Depth modalities, the Point Transformer architecture (Zhao et al., ) for LiDAR, the Point Transformer architecture without transition down layers for mmWave point cloud data, and the MetaFi++ architecture (Zhou et al., ) for Wifi-CSI. For HAR using MM-Fi, the same backbone architectures are utilized, with the parameters specifically trained for HAR due to the differing feature representation requirements across tasks. When utilizing the XRF55 dataset for HAR, the ResNet-18 architecture is adopted for the RFID, WiFi, and mmWave modalities to maintain consistency with the baseline backbone networks employed in the study (Wang et al., ).

### A.2 Evaluation of modality-specific features combining methods

Our objective is to assess whether concatenation is the most effective method for combining modality-specific features into multi-modal embedding. To investigate this, we conducted a controlled experiment on the MM-Fi HPE task. In this experiment, we compared the concatenation method with the feature addition approach, where the feature values from each modality are summed and then divided by the number of modalities. As demonstrated in the Table 7, the feature addition approach failed to preserve distinctive modality features in the multimodal embeddings, resulting in average performance deteriorations of 15.7 mm on MPJPE and 5.9 mm on PA-MPJPE.

**Table 6: Ablation studies on drop out layers and positional encoding layer.**
| Ablation Studies<br>on MM-Fi HAR | Dropout Layer | Positional Encoding |  |  |
| --- | --- | --- | --- | --- |
| Accuracy | Accuracy |  |  |  |
| Methods | DPR = 0.3 | DPR = 0.0 | False | True |
| I | 26.1% | 27.4% | 25.4% | 27.4% |
| D | 46.6% | 47.3% | 47.0% | 47.3% |
| L | 12.8% | 32.9% | 31.9% | 32.9% |
| R | 80.9% | 86.0% | 85.5% | 86.0% |
| I + D | 46.3% | 45.2% | 45.9% | 45.2% |
| I + L | 27.1% | 29.8% | 30.6% | 29.8% |
| I + R | 69.2% | 72.4% | 74.4% | 72.4% |
| D + L | 46.2% | 48.4% | 49.0% | 48.4% |
| D + R | 74.7% | 80.2% | 78.6% | 80.2% |
| L + R | 79.3% | 86.7% | 87.4% | 86.7% |
| I + D + L | 46.1% | 46.1% | 47.6% | 46.1% |
| I + D + R | 67.6% | 70.8% | 69.9% | 70.8% |
| I + L + R | 66.9% | 74.4% | 75.6% | 74.4% |
| D + L + R | 72.6% | 80.1% | 78.3% | 80.1% |
| I + D + L + R | 65.8% | 70.0% | 70.7% | 70.0% |

### A.3 Ablation Studies

To demonstrate the effectiveness of our proposed modality-invariant foundation model and the corresponding optimization process, we evaluate the impact of dropout layer, positional encoding,and the optimizer choice. And the results are illustrated in Table 6& 7.

To evaluate the effect of dropout layers, we incorporated a dropout layer with a rate of 0.3 into all transformer structures within X-Fi and assessed its performance on the MMFi dataset for the HAR task. The experimental results indicated that the introduction of the dropout layer led to a 4.6% decrease in averaged accuracy. Furthermore, the dropout layer significantly impaired the model’s predictive performance, particularly when handling challenging modalities.

To assess the impact of different optimizer choices on experimental results, we conducted a set of controlled experiments using the SGD optimizer with a learning rate scheduler on the MM-Fi HPE task. The results indicated that the AdamW optimizer facilitated better convergence to a more optimal local minimum, leading to an average performance improvement of xx mm on MPJPE and x mm on PA-MPJPE.

To assess the impact of LiDAR positional encoding, we conducted an experiment by removing the positional encoding structure. The results demonstrated that positional encoding provided a slight improvement in the performance of X-Fi with single-modality input. However, when faced with multi-modal inputs, the model’s performance fluctuated, exhibiting worse results with certain modality combinations while performing better with others.

**Table 7: Ablation studies on comparing multi-modal features combininng methods and optimizer choices.**
|  | multi-modal feature combining methods | optimizer choices |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Ablation Studies<br>on MM-Fi HPE | MPJPE | PA-MPJPE | MPJPE | PA-MPJPE |  |  |  |  |
| Methods | Add | Concate | Add | Concate | SGD | AdamW | SGD | AdamW |
| I | 123.4 | 93.9 | 78 | 60.3 | 239.1 | 93.9 | 165.2 | 60.3 |
| D | 112.5 | 101.8 | 54.4 | 48.4 | 211.9 | 101.8 | 146.3 | 48.4 |
| L | 226.4 | 167.1 | 104.2 | 103.2 | 243.9 | 167.1 | 130.4 | 103.2 |
| R | 144.3 | 127.4 | 70.3 | 69.8 | 248.2 | 127.4 | 148.1 | 69.8 |
| W | 227.7 | 225.6 | 104.8 | 105.3 | 296.5 | 225.6 | 138 | 105.3 |
| I + D | 96.1 | 86.1 | 53.4 | 48.1 | 166.4 | 86.1 | 116.1 | 48.1 |
| I + L | 115.9 | 93 | 76.7 | 59.7 | 151.2 | 93 | 105.1 | 59.7 |
| I + R | 104.3 | 88.8 | 63.1 | 57.3 | 149.4 | 88.8 | 110.9 | 57.3 |
| I + W | 118.7 | 93 | 75.8 | 59.5 | 144.4 | 93 | 102.8 | 59.5 |
| D + L | 108.9 | 102.5 | 53.5 | 48.4 | 140.1 | 102.5 | 95.8 | 48.4 |
| D + R | 106.6 | 98 | 52.8 | 47.3 | 147.8 | 98 | 106.3 | 47.3 |
| D + W | 110.6 | 101.8 | 53.5 | 48.1 | 136.6 | 101.8 | 99.1 | 48.1 |
| L + R | 135.8 | 109.8 | 67.5 | 63.4 | 166.6 | 109.8 | 108.3 | 63.4 |
| L + W | 210.5 | 159.5 | 102.9 | 102.7 | 184.3 | 159.5 | 113.5 | 102.7 |
| R + W | 131.7 | 117.2 | 67.4 | 62.7 | 192.5 | 117.2 | 117.5 | 62.7 |
| I + D + L | 94.4 | 84.8 | 53.1 | 48.2 | 127 | 84.8 | 87.6 | 48.2 |
| I + D + R | 93.4 | 83.4 | 52.4 | 47.3 | 136.2 | 83.4 | 92.9 | 47.3 |
| I + D + W | 96 | 85.3 | 53 | 48.1 | 124.5 | 85.3 | 88 | 48.1 |
| I + L + R | 101.4 | 88.4 | 62.5 | 57.2 | 121.5 | 88.4 | 85.5 | 57.2 |
| I + L + W | 114.5 | 93 | 75.4 | 59.7 | 121 | 93 | 87.1 | 59.7 |
| I + R + W | 102.7 | 88.5 | 62.2 | 57.1 | 127 | 88.5 | 90.2 | 57.1 |
| D + L + R | 105.2 | 96 | 52.4 | 47.3 | 125.8 | 96 | 88.6 | 47.3 |
| D + L + W | 108.5 | 102 | 53.2 | 48.1 | 128.1 | 102 | 91 | 48.1 |
| D + R + W | 105.8 | 97 | 52.4 | 47.1 | 127.9 | 97 | 91.4 | 47.1 |
| L + R + W | 128.1 | 107.4 | 66.9 | 63.1 | 164.9 | 107.4 | 104.5 | 63.1 |
| I + D + L + R | 92.2 | 83.5 | 52.2 | 47.6 | 107.4 | 83.5 | 76.6 | 47.6 |
| I + D + L + W | 94.4 | 86 | 53 | 48.2 | 106.2 | 86 | 77.2 | 48.2 |
| I + D + R + W | 93.5 | 84 | 52.2 | 47.6 | 115.9 | 84 | 79.3 | 47.6 |
| I + L + R + W | 100.3 | 88.6 | 62.3 | 57.1 | 113.6 | 88.6 | 80.5 | 57.1 |
| D + L + R + W | 104.8 | 97.6 | 52.2 | 47.4 | 125 | 97.6 | 88.5 | 47.4 |
| I + D + L + R + W | 92.3 | 83.7 | 52.1 | 47.6 | 101.2 | 83.7 | 73.6 | 47.6 |

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: Can you clarify the number of iterations needed for the X-fusion block?

It will be interesting to compare average pooling vs. cross-attention with learned queries.

Comparison to prior methods (see weaknesses) should be included.
soundness: 3
strengths: The paper addresses an interesting and important problem of developing foundation models for multi-modal human sensing. 

There are some interesting ideas in the model design, e.g., using cross-attention to iteratively refine the average pooled multimodal features.
summary: The paper presents X-FI, a foundation model for multi-modal human sensing. X-FI allows inference with versatile combinations of sensor modalities, and supports multiple tasks including pose estimation and activity recognition. Key to X-FI is a multi-modal feature encoding module that builds on cross-attention. X-FI was evaluated on two public datasets, demonstrating some encouraging results.
weaknesses: I am not convinced by the claim on “modality invariant human sensing”, one of the key contributions of the paper. Conceptually, there are several approaches for modality invariant processing. For example, the model in Memmel et al., 2023 (a Transformer that takes tokens from any combination of modalities) can in theory be adapted for human sensing. Technically, this invariance is achieved by dropping out modalities during training, which is again discussed in Memmel et al., 2023 and also mentioned Chen et al., 2023.

The experiments miss important baselines. There is hardly a comparison to prior methods in the paper. At least, two sets of baselines should have been considered: (a) latest methods using a single modality (to demonstrate the benefit of multi-modal fusion); and (b) previous methods that also considers multimodal fusion such as Chen et al., 2023 (to highlight the benefit of the proposed method). The current baselines are rather straw man.  

The presentation of the technical components lacks clarity. Some of the designs are not well justified. 
* Eq 2: The use of average pooling is not justified. In fact, with different numbers of input modalities, each bin in average pooling may span (1) tokens from a single modality; or (2) tokens from more than one modality. It is not clear why average pooling works here. One possibility is learning a fixed number of queries and using cross-attention to project the input multi-modal feature, similar to the design of Percerver IO.
* Eq 4: It is unclear how to determine the number of iterations for the X-fusion block. I also could not find the exact number used in the experiments. 
* L229: I assume n_f is the number of tokens per modality, but it was not described in the paper.

================================

Post-rebuttal update: In the rebuttal, the authors have addressed my previous concerns, included substantial updated results, and majorly improved the paper. While I still have some concerns over the technical innovation and the use of average pooling, I feel like the paper is now beyond the bar. Therefore, I have increased my rating.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: I presented many questions and suggestions in the sections above. I think the main issue of the paper was a lack of clarity in describing the method and its implementation.
soundness: 3
strengths: Introduction:

1. The need for a flexible multimodal human sensing model is well-motivated. Particularly, the second paragraph outlining specific modalities and how their strengths and weaknesses can compensate for each other is strong.

Modality Invariant Foundation Model for Human Sensing:

2. The motivation for the cross-attention multimodal fusion (X-Fusion) is good. Attempting to combat the domination of certain modalities by incorporating modality-specific information is good. How did the authors know that the attention mechanism was weighing some modalities more than others? Did they view the transformers' attention weights and notice a trend? Did previous works report that happening? It would be great to see these preliminary results somewhere.
3. I appreciate the thought and structure that went into randomizing the modalities in "3.3 Modality-invariant training." Table 3 is good. How did you determine the final probabilities? How did you know it was necessary to include more LiDAR samples than the other modalities?

Experiments:

4. Just to clarify, in tables 1 and 2 X-Fi was only trained once and tested on various input modalities (each unique row), but the baselines were retrained on each modality combination separately. Is this true (line 365)? If this is the case, I think this should be noted in the caption and is a strength for your method.
5. I like Figure 4's clusterability scores. The visualization makes it hard to distinguish the difference between R and W+R, but I think the numbers emphasize the point that multimodal is better.
summary: This paper introduces a model to perform human sensing on various input modalities and output tasks. The method utilizes transformer-based attention to input a varying number of modalities and learn complementary features. The method can perform during test time with any combination of input modalities better than a fusion architecture trained with those modalities, which indicates a step towards developing a multi-sensor foundation model for human motion understanding.


This review follows the sections of the paper.
weaknesses: Introduction:

1. Figure 1: I am not sure what the purpose of this figure is. What are you trying to show? X-Fi can replace a bunch of modality-specific models? But within X-Fi we have all the modality-specific encoders anyways. Are you trying to indicate that you don't need to retrain models for each combination of modalities? Nothing in this figure indicates retraining. I would suggest brainstorming a more informative diagram.
2. Figure 2 is good, but could use some clarifications. Why are the last 3 modalities with dotted lines? Can you label the dark gray background boxes (which module is the X-Fusion part)? Also the text is quite small and hard to read. What does N stand for in the arrow loop?
3. I think the term "foundation model" is starting to be used loosely quite frequently, and I would appreciate if the authors could clarify in a sentence or two. Maybe define a foundation model in this context and how their method is such before referring to their method as a foundation model. This could also be done in related works, comparing your method to any related foundation models.

Related Works:

4. I think a discussion of multimodal missing modalities literature is missing. There are many works that perform multi-modal HAR in the face of missing modalities during training/testing which is a similar setup to X-Fi. Also, there is a lot of literature on transformer based fusion that could be mentioned here.
Here are some potentially relevant works:
- Sangmin Woo, Sumin Lee, Yeonju Park, Muhammad Adi Nugroho, and Changick Kim. Towards
Zihui Xue, Zhengqi Gao, Sucheng Ren, and Hang Zhao. The modality focusing hypothesis: Towards understanding cross modal knowledge distillation. arXiv preprint arXiv:2206.06487, 2022.
- Quan Kong, Ziming Wu, Ziwei Deng, Martin Klinkigt, Bin Tong, and Tomokazu Murakami. Mmact: A large-scale dataset for cross modal human action understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8658–8667, 2019.
- Qi Wang, Liang Zhan, Paul Thompson, and Jiayu Zhou. Multimodal learning with incomplete modalities by knowledge distillation. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1828–1838, 2020.
- XB Bruce, Yan Liu, and Keith CC Chan. Multimodal fusion via teacher-student network for indoor action recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35(4), pp. 3199–3207, 2021.

5. In the second sentence under Modality invariant methods. I would disagree that decision-level approaches are restricted to fixed modality combinations, and both citations provided are feature level fusion. Decision level methods may average the outputs of separate modality modals, or perform some sort of voting, so it would be easy to extend to more modalities or drop the outputs of some modalities. In fact I think that's what your baseilne1 does to my understanding.
6. I don't think the discussion in the last paragraph citing LLM methods and a visual odometry method in robotics is very relevant.

Modality Invariant Foundation Model for Human Sensing:

7. A lot of notation is introduced and it is hard to follow. Consider maybe labeling the arrows and boxes in figure 2 with the notation ($G_{ca}, G_{cm}, Emb_{cm}$ etc.). Also maybe the dimensions of the cubicle blocks so we can see what n_f and d_f is referring to. Is $\theta_{Enc}$ Just the concatenation of $\theta_{E}$ and $\theta_{LP}$? Is positional encoding shown in Equation (1)?
8. The key value generators and cross-attention transformers is unclear. It would be helpful to provide a diagram (at least in the appendix) for them. Also, when using the term transformer, I assume the input is one set of tokens and that's projected into keys, queries and values, but "in cross-attention transformer," the input is already K,Q,V's. I would suggest referring to it as just a cross-attention module. The original Transformer in Vaswani et al (the only cited Transformer paper in 3.2.1) includes the KQV projection as well as an encoder with self-attention and a decoder with self-attention/cross attention and multiple heads. Is that what your "Transformer" is doing?
9. In line 272, I don't understand why Emb_{cm} is being added back to O_i ? Again a diagram here might help.
10. I would suggest an ablation without the K,V generators. It is just two MLP's but it happens right after the modality-specific linear projections, so maybe the modality-specific projections can capture that information.

Experiments:

11. In Tables 1 and 2 I would suggest labeling the reported metrics either in the column header or caption (it sounds like MSE for HPE and Accuracy for HAR?).
12. HPE Qualitative results Figure 3. It is not apparent to me that RGB captures better leg movements and depth is better for hands. Can you circle on the images where you are observing this? Also, any thoughts on why this would be the case? Maybe even providing numbers for the errors on hand joints vs leg joints could make this result more convincing!
13. The results could be strengthened by comparing to other multimodal methods. Are there other works that perform multimodal HPE or HAR on the MM_Fi and XRF55 datasets that you can compare to?

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: 1- I suggest to have a joint loss ( L= L_I+ L_R+L_W+....L_N). In other words, to have loss for each modality and the X-Fusion output as well. It should boost the performance.  
2-   I suggest an ablation study on the number of  stacked blocks in X-Fi  block. 
3- I was expecting an experiment which gradually add new modality each time. For example the model is trained first on RGB then WiFI is  added by fixing the parameters and finetuning the model after adding the new modality.
soundness: 2
strengths: The paper presents a simple yet effective framework for multi-modal fusion from heterogenous modalities.
Two large datasets were adopted in the experiments.  
The authors conducting experiments on  different modality combinations.
summary: In this paper, the authors propose a multi-modal modality invariant foundation model to tackle the problem in the current human sensing techniques. They introduced X-Fi block to combine the feature representation from different modalities.  The authors utilized two datasets to verify the effectiveness of the proposed model on human action recognition and pose estimation. The proposed model outperformed the other methods.
weaknesses: 1- The paper needs polishing and revision. For example in the first page (line 043) "Alternative modalities like LiDAR, mmWave radar,
and WiFi have be introduced to overcome these challenges" --> it should been not be. Additionally, the authors are mixing Fig with Figure. Or capital letters within the sentence--> Line 233 "Notably, The multi-head" and Line 259.
2- The authors mentioned in the contribution and abstract the proposed model surpassed the SOTA. I did not find a table of comparison against the SOTA especially the only comparison is against baseline.
3- The authors did not provide the experimental setting. For example, MM-Fi [1] has three settings and it is not clear which setting is adopted. I recommend to adopt the three settings.


[1]Yang, Jianfei, He Huang, Yunjiao Zhou, Xinyan Chen, Yuecong Xu, Shenghai Yuan, Han Zou, Chris Xiaoxuan Lu, and Lihua Xie. "Mm-fi: Multi-modal non-intrusive 4d human dataset for versatile wireless sensing." Advances in Neural Information Processing Systems 36 (2024).

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. Can the authors provide additional details on the distinguished contribution points of the X-fusion block compared to existing cross-attention fusion approaches? 
2. Would the authors consider conducting further experiments with other cross-modal foundational models (e.g., Meta-Transformer, ImageBind) as baselines to provide a more comprehensive comparison?
3. Is it possible to include additional benchmark datasets with IMU data for HAR task?
soundness: 3
strengths: + This paper addresses the important problem of modality-invariant human sensing by designing a foundational model with flexiblity without retraining. 
+ The paper is clearly written and well-structured, effectively communicating the motivation, methodology, and results. Concepts such as modality-specific feature extraction, cross-attention fusion, and the role of the X-fusion block are clearly explained, The transformer-based architecture handles multimodal input data and X-fusion block is used for cross-modal feature fusion by using cross-attention.
summary: This paper introduces a modality-invariant foundation model that allows for the integration of various sensor modalities for human sensing. The proposed X-Fi model employs a transformer architecture with a "X-fusion" mechanism to preserve modality-specific features in the multimodal integration process. The proposed model is designed to support any combination of sensor modalities with flexibility of the input dimension without requiring additional retraining. They evaluated the proposed method on two benchmark datasets, MM-Fi and XRF55, covering six distinct modalities, with two downstream tasks: human pose estimation and human activity recognition. The experimental results showed that the proposed X-Fi model improved the performance of HPE and HAR.
weaknesses: - The paper lacks a thorough investigation of related cross-modal / modality-invariant foundational models, such as CLIP [1], PaLM-E [2], Meta-Transformer [3], ImageBind [4], IMU2CLIP [5], and zero-shot learning for HAR [6]. The authors need to investigate recent cross-modal foundational models.
- While the paper proposes the X-fusion block for multimodal fusion, the core approach of using transformers and cross-attention for feature fusion is not new. Similar attention-based fusion methods are widely used in the literature for feature-level integration. The paper could strengthen the novel contributions and clarify the novel aspects of X-fusion compared to existing methods.
- The evaluation experiments focus on comparing the proposed method with basic feature and decision-level fusion approaches. The proposed foundational model may be compared with other cross-modal foundational models.
- The evaluation was conducted on MM-Fi and XRF55 datasets, which primarily include specific modalities like cameras, WiFi, LiDAR. However, human activity recognition often involves wearable sensors like IMUs. It is better to add one or two more benchmark datasets with IMU sensors.


[1] Radford, Alec, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry et al. "Learning transferable visual models from natural language supervision." In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
[2] Driess, Danny, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid et al. "PaLM-E: An Embodied Multimodal Language Model." In International Conference on Machine Learning, pp. 8469-8488. PMLR, 2023.
[3] Zhang, Yiyuan, Kaixiong Gong, Kaipeng Zhang, Hongsheng Li, Yu Qiao, Wanli Ouyang, and Xiangyu Yue. "Meta-transformer: A unified framework for multimodal learning." arXiv preprint arXiv:2307.10802 (2023).
[4] Girdhar, Rohit, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. "Imagebind: One embedding space to bind them all." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15180-15190. 2023.
[5] Moon, Seungwhan, Andrea Madotto, Zhaojiang Lin, Aparajita Saraf, Amy Bearman, and Babak Damavandi. "IMU2CLIP: Language-grounded Motion Sensor Translation with Multimodal Contrastive Learning." In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 13246-13253. 2023.
[6] Tong, Catherine, Jinchen Ge, and Nicholas D. Lane. "Zero-shot learning for imu-based activity recognition using video embeddings." Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5, no. 4 (2021): 1-23.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: Please see the weakness part.
soundness: 2
strengths: 1. The foundation model of different human sensing modalities is a promising approach, and the author made a meaningful attempt. The training schedule of this model is interesting
summary: This paper introduces a multi-modal foundation model that integrates six distinct modalities to enhance human sensing capabilities. The proposed model demonstrates the scalability of various modality combinations and incorporates an X-Fusion block to fuse multi-modal features while preserving modality-specific information. The authors conduct extensive experiments on two multi-modal datasets to evaluate the model's performance in human pose estimation and human action recognition tasks.
weaknesses: 1. The section of problem definition is missing. This paper addresses two different tasks, which generally require different input formats of modalities. For instance, HAR requires video clips, but HPE could be video clips or frames. The clear details about the inputs and outputs are crucial for reader to understand.
2. There are some undefined notations, such as  in line 172 and  in Eq. 4.
3. The motivation regarding the module in subsection 3.2.2, which is to pay equal attention to each modality, is unconvincing. Different modalities offer varying degrees of human sensing information, and only valuable modality or information are important. The model paying more attention to dominant modalities is more intuitive. Equal attention could lead to the inclusion of irrelevant information and the potential omission of key details.
4. The cross-attention module is intended to accentuate modality-specific information. Nevertheless, the absence of ablation studies on this module precludes a conclusive evaluation of its efficacy, and there is no empirical evidence to substantiate the modality-specific features extracted into the K-V pairs. Some additional constraints are necessary. For instance, SimMMDG [1] leverages distance loss between modality-specific and modality-shared features and the orthogonal constraints is adopted in [2].
5. No ablation study on the components of proposed method.
6. The experimental results of Lidar in MM-Fi reveal inferior performance relative to mmWave in Tab. 1. This is in contrast to the results in MM-Fi under same experimental setup and backbones.  And it is also not intuitive as Lidar data, characterized by its higher point density, offers a more precise and detailed representation of the human compared to mmWave.
7. To ensure a fair comparison in Tab. 4, the number of parameters in the transformer and X-Fusion block should be equalized.

8.You employ multiple layers of cross-modal transformers to obtain cross-modal embeddings. How does the number of these iterative layers，N, impact the experimental results? Additionally, how might varying the number of transformer layers for different modality counts influence the outcomes?

9. The proposed method in the paper enforces a balance across all modalities. Could this forced balancing lead to a negative impact in tasks that primarily rely on a specific modality, especially if some modalities contain less informative data?

10. Performing computations across all modalities in each iteration could result in high computational complexity, particularly in multi-modal processing and long-sequence modeling. This increased demand could pose challenges for real-time applications, making practical deployment of the method more difficult.


[1]Dong H, Nejjar I, Sun H, et al. SimMMDG: A simple and effective framework for multi-modal domain generalization[J]. Advances in Neural Information Processing Systems, 2023, 36: 78674-78695.
[2]Jiang Q, Chen C, Zhao H, et al. Understanding and constructing latent modality structures in multi-modal representation learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 7661-7671.

Meta-review: This paper proposes X-Fi, a multi-modal foundation model designed for human sensing, capable of integrating diverse sensor modalities for tasks such as human pose estimation and activity recognition. Utilising a transformer-based architecture with the proposed X-fusion mechanism, X-Fi preserves modality-specific features during fusion, enabling flexibility in handling varying modality combinations without retraining. The model was evaluated on benchmark datasets, showing improved performance over existing methods in both pose estimation and activity recognition tasks. The paper's strengths include its innovative use of cross-attention for modality-specific feature fusion, the flexibility of its modality-invariant design, and effective experimentation with multiple modality combinations. The clear motivation, robust model design, and well-explained methodology contribute to its impact on multi-modal human sensing. As a result, all reviewers have expressed positive feedback and recommended "accept" or "marginally above the acceptance threshold".

Decision: Accept (Poster)