X-Fi: A Modality-Invariant Foundation Model for Multimodal Human Sensing
Authors: Xinyan Chen
Jianfei Yang
Venue: ICLR 2025 Poster (2025)
Keywords: human sensing
multimodal learning
Primary area: foundation or frontier models, including LLMs

Abstract: Human sensing, which employs various sensors and advanced deep learning technologies to accurately capture and interpret human body information, has significantly impacted fields like public security and robotics. However, current human sensing primarily depends on modalities such as cameras and LiDAR, each of which has its own strengths and limitations. Furthermore, existing multimodal fusion solutions are typically designed for fixed modality combinations, requiring extensive retraining when modalities are added or removed for diverse scenarios. In this paper, we propose a modality-invariant foundation model for all modalities, X-Fi, to address these issues. X-Fi enables the independent or combinatory use of sensor modalities without additional training by utilizing a transformer structure to accommodate variable input sizes and incorporating a novel "X-fusion" mechanism to preserve modality-specific features during multimodal integration. This approach not only enhances adaptability but also facilitates the learning of complementary features across modalities. Extensive experiments conducted on the MM-Fi and XRF55 datasets, employing six distinct modalities, demonstrate that X-Fi achieves state-of-the-art performance in human pose estimation (HPE) and human activity recognition (HAR) tasks. The findings indicate that our proposed model can efficiently support a wide range of human sensing applications, ultimately contributing to the evolution of scalable, multimodal sensing technologies.

Published as a conference paper at ICLR 2025
X-FI: A MODALITY-INVARIAN FOUNDATION MODEL FOR MULTIMODAL HUMAN SENSING
Xinyan Chen; Jianfei Yangi Nanyang Technological University
ABSTRACT
Human sensing, which employs various sensors and advanced deep learning tech- nologies to accurately capture and interpret human body information, has signifi- cantly impacted fields like public security and robotics. However, current human sensing primarily depends on modalities such as cameras and LiDAR, each of which has its own strengths and limitations. Furthermore, existing multimodal fusion solutions are typically designed for fixed modality combinations, requiring extensive retraining when modalities are added or removed for diverse scenarios. In this paper, we propose a modality-invariin X-Fi, to address these issues. X-Fi enables the independent or combinatory use of sensor modalities without additional training by utilizing a transformer structure accommodate variable input sizes and incorporating novel "X-fusion" mech- anism to preserve modality-specitff features during multimodal integration. This approach not only enhances adaptability but also facilitates the learning of com- plementary features across modalities. Extensive experiments conducted on the MM-Fi and XRF55 datasets, employing six distinct modalities, demonstrate that X-Fi achieves state-of-the-art performance in human pose estimation (HPE) and human activity recognition (HAR) tasks. The findings indicate that our proposed model can efficiently support a wide range of human sensing applications, ulti- mately contributing to the evolution of scalable, multimodal sensing technologies.
foundation model for all modalities.
INTRODUCTION
Human sensing refers to using various sensors to capture human body information in specific space, such as presence (Nguyen et al., 2016; Nanzer, 2017), activity (Vrigkas et al., 2015; Yang et al,,, 2023)), and pose (Toshev & Szegedy, 2014; Cai et al., 2022). Benefiting from advanced deep learning technology, human sensing has made significant progress in both accuracy and granularity. expanding applications in many domains. For instance, video surveillance (Elharrouss et al., 2021) is empowered by human sensing algorithms to analyze dangerous behaviors. In the field of robotics, human sensing helps robots comprehend human actions and realize human-robot collaboration (Gao et al., 2019).
Currently, human sensing tasks mainly rely on vision-based modalities like cameras (lonescu al., 2013), which face inherent limitations such as reliance on illumination, privacy concerns, and in- sufficient 3D information (Cai et al., 2022). Alternative modalities like LiDAR, mm Wave radar, and WiFi have been introduced to overcome these challenges (Nirmal et al., 2021). Each modality has distinctive advantages but also has limitations, e.g., LiDAR offers high-resolution data but is expensive (Li et al., 2022), mm Wava radar is cost-effective and informative but lacks sensitivity to but has low reso-
static objects (An & Ogras, 2021), and WiFi ubiquitous and privacy-preserte lution (Yang al., 2023; Zhou et al., 2023). Therefore, single- mogll sensing solutions cannot fit all scenarios, and a multi-mode modal approach that leverages the strengths each modality is essential for future advancements in human sensing.
Numerous methods have been proposed for multi-modal perception based on sensor fusion. They usually predefine a fixed set of modalities for specific scenario, e.g., combining RGB and LiDAR for pose estimation in autonomous driving to enhance long-range all-day sensing. Nevertheless, in any given scenario, once the model is trained, adding or removing even one modality requires
"Codes are available at: https:/ /xwp/cceaccc.cccc github.io/X-EE..... 1-iX-Ei 'Correspondigg Author (jaafel.yang@nnu.dd..s)))

---

Published as a conference paper at ICLR 2025
Transins
Appllcations
East bldies
T
Looke
B
X-Fi
Insrard Teaning
Helogrncs
KBB Vap
Uodel
Figure I:
The left image depicts current human sensing solutions that are specifically designed and trained for fixed modality combinations, while the right image illustrates our proposed modality- invariant foundation model, X-Fi, which can be trained once and adapted to various scenarios.
huge effort: adjusting the network and retraining it from scratch. In the real world, we may require versatile combinations of sensor modalities according to different scenarios (Yang et al., 2024). Hence, we contemplate whether it is possible to design a one-for-all solution for modality- invariant human sensing. Such a model would require training only once, allowing all sensor modalities that participated in the training process to be utilized independently or in any combination for wide range of potential applications.
To achieve modality-invarian human sensing, we aim to design a foundation model that can dy- namically handle varying numbers of modalities. First, considering the arbitrary number of input modalities, we naturally resort to a transformer structure due to its adaptability to variable input sizes (Vaswani et al., 2017). Second, recognizing that each modality has unique characteristics, we adopt modality-specinf each modality's distinct features during multimodal fusion, introduce cross- cross-attention fusion mechanism that reduces the information loss caused by multimodal feature compression and fu- sion process. Surprisingly, this design not only supports dynamic inputs from various modalities but by training with diverse modality combinations to perform sensing tasks, it learns complemen- tary features among modalities as well. As result, it even achieves superior performance against conventional modality fusion methods such as feature-level and decision-level fusion.
feature extractors to capture distinctive information. Finally, to retain
To this end, we propose a novel modality-invariinn foundation model, X-Fi, for versatile human sensing. X-Fi can take in any combination of modalities and activate corresponding parts to extract mppelity- features. A cross-modal transformer designed to learn cross-moda feature and each modality information is then preserved in the representation by executing cross-attention multi- modal fusion processes to preserve distinctive modal features. As shown in the Figure 2, we have set of preserve modality-speciinn features. For versatile combinations of sensor modalities, corresponding feature encoders are activated to obtain modality-specifte features of each modality. Subsequently, the key-value generators are trained to produce key-value pairs, which contain more task-related information derived from modality-specifie features. For the modality-invariant multimodal fu- sion process, we introduce "X-fusion" block. It consists of cross-modal transformer and a set of modality-correlatedd cross-attention modules. The cross-modal transformer is designed to adap- tively learn unified cross-moda embedding across variable-sioe comprises the features of each modality. The generated cross-mode embedding serves as the query, along with key-value pairs of each modality, are sent to the corresponding cross-attention module to perform attention-bas based feature injection. We evaluated X-Fi on human pose estimation (HPE) and human activity recognition (HAR) tasks on MM-Fi (Yang et al., 2024) and XRF55 (Wang et al., 2024) datasets, demonstrated that X-Fi surpasses previous methods by MPJPE 24.8% and PA-MPJPE 21.4% HPE task, 2.8% on HAR task.
feature encoders and key-value generators to extract, fine-tune and
multi-moda embedding, which
The main contributions are summarized as follows. First, we proposed X-Fi, a foundation model for modality invariant human sensing, which has strong scalability that only needs to be trained once and could support dynamic inputs from various modalities for a wide range of sensing scenarios. Sec- ond, we further developed X-Fusion block to empower efficient multi-modal fusion under modality invariant training strategy by preserving modality-speciffe information and applying token fusion technique to inject back into the cross-modal embedding. Third, we conducted extensive experi- ments on the MM-Fi and XRF55 datasets, utilizing a total of six distinct modalities, and observed that our proposed X-Fi achieved state-of of-the-art performance on both the HPE and HAR tasks.
2

---

Published as a conference paper at ICLR 2025
2
RELATED WORK
2.1 HUMAN SENSING
Human sensing has been extensively explored for applications such as autonomous driving and robotics in recent years, aiming to capture, analyze, and interpret human body information through various sensors and novel deep learning frameworks (Cai et al., 2022). Human sensing approaches are typically divided into three categories: i) Vision- based approaches, where human body informa- tion is extracted from visual data captured by cameras (Shahroudy et al., 2016) and leveraged by deep learning architectures like convolutional structures (Pavlakos et al., 2018; Sun et al., 2019; Li et al., 2020) and transformer (Goel et al., 2023; Cai et al., 2024). ii) Sensor-based approaches, using alternative modalities such as LiDAR (Ren et al., 2024), mm Wave radar (Wang et al., 2021; An & Ogras, 2022), and WiFi CSI (Yang et al., 2023; Zhou et al., 2023) to overcome limitations of vision- based methods and expand applicability. iii) Multimodal fusion approaches, including decision-level fusion (An al., 2022; Yang al., 2024) and feature-level fusion (Zheng al., 2022; Chen et al., 2023), which combine each modality's 's strengths to improve sensing performance.
2.2 MODALITT- METHODS
Current human sensing methods lack broad applicability. Feature-level fusion approaches (Zheng et al., 2022; Chen et al., 2023) are typically tailored to fixed modality combinations for specific applications. For example, (Chen et al., 2023) employed modality-specific backbones to extract grid features from images and cluster features from mm Wave point clouds, combining them via a global integration module to produce fused features. Decision-level fusion approaches (Yang et al., 2024; Yaday et al., 2023) are easy to adapt to various modality combinations, but the performance could be hindered by less informative modalities, e.g. WiFi.
In fields beyond human sensing, several studies on multimodal fusion seek to accommodate diverse sensor inputs to a unified model for various applications. Some approaches apply cross-modal distil- lation techniques to seek the shared and the supplementary information across multi calliiimodlities (Xue al., 2022; Kong et al., 2019; Wang et al., 2020; Bruce et al., 2021). Some methods align multi- modalities features in a joint embedding space with a unified model structure (Man et al., 2023; Radford et al., 2021; Driess et al., 2023; Zhang et al., 2023; Girdhar et al., 2023; Moon et al., 2023; Tong et al., 2021; Wu et al., 2023). However, these approaches require retraining or fine-tuning to adapt to various scenarios. In recent works, foundation models, which are large-scale, pre-trained models that learn unified representations from diverse inputs, have gained attention for their capacity to be fine-tuned for a wide range of tasks (Devlin, 2018; Touvron et al., 2023). Extending this con- cept, invvaiit representation learning have been proposed to handle dynamic multimodal combinations across various scenarios. (Xue & Marculescu, 2023) designs a gating network to dynamically activate the modality- cottiyaaionannnineeiiiii expert networks for multimodal fusion and (Memmel et al., 2023) introduces a modality- invariant training strategy by dropping modalities dur- ing training. However, these approaches face challenges due to the need for numerous networks and configurations to handle diverse modality combinations in the presence of extensive sensor options.
MODALITY-INVARIAT FOUNDATION MODEL FOR HUMAN SENSING
Assume the input X comprises a random combination of modalities within a specific range, includ- ing any number and type of modalities. Our objective is to learn unified representation Zem that in- tegrates modality-specifit features from X by using a modality-invanl -invariat foundation model F. This representation can be mapped to various down-stream human sensing task spaces t₂ by using corre- sponding task-specific functions Gt., such that the task-specified loss function Lr, (9t, (F(X)): y..) is minimized. In this paper, the M/ refers to 3D human joints for HPE task and human activity classes for HAR task. Thus we propose X-Fi, a modality-invarian human sensing framework designed to map the arbitrary multimodal input X any task-specific target domain f..
As illustrated in the Figure 2, given & modalities in the dataset, we first prepare a set of modality- specified feature encoders {E, (-: (=) to extract features {f: e R xd, | from correspond- ing modality inputs {z, ) where ny indicates the number of features extracted for each modality and dy indicates the dimension of each feature. Then we introduce a gross-attention-batt modal fusion design that could efficiently preserve modality-specific features in the learned unified
multi-
3

---

Published as a conference paper at ICLR 2025
Modality Feature Encoders E
Modality Key Value Generators Giu
Application Heads
Kapolks
Fortar Satuatir
Dep Kep
Seliiiaiioo Hoar
(X,
Other Tarks
X-Fusion
Diphi Crean Kahmic Nodule
Loguss
Coros Modil shene
Oox
Figure 2: The architecture of the proposed modality invarriat foundation model, X-Fi. X-Fi consists modality feature encoders and an X-Fusion module, which includes pross-moda transformer and modality-specified cross-attention modules. The modalities with dotted lines represent inactivate modalities in the given scenario. The N in X-Fusion block represents the number of iterations.
cross-model embedding. The modality-specified key-value generators {G (:: (3, } i=1 are ap- plied to encode the modality information from {f into key-value pairs {(K,, vi) To obtain an unified cross-modal embedding Fem E R xdy, we use cross-modal transformer Gem (: ()") to learn from the Zmm E [[(==h)xd,, The generated cross modal embedding Zem serves as the query Q, together with each key-value pair, to send into the modality-specific cross-attention mod- ules (Goo, (-: (,, / in order to inject poodality-speciffe information. Through iterative learning of the Zem and performing modality-specifie information injection, the final produced žem could be Multi-layer Perception (MLP) head.
used for various downstream tasks via
In the following subsections, will decompose X-Fi by describing the modality feature encoding (sec. 3.1), the attention- based X-Fusion (sec. 3.2), and the modality-invarian training (sec. 3.3).
3.1 MODALITY FEATURE ENCODING
In human sensing, different modalities exhibit distinctive characteristics, requiring unique feature encoder structures to extract robust modality-specifft feature. For instance, images are represented a 2D matrices and have rich spatial features, requiring models like CNN (LeCun al., 1998) and ViT (Dosovitskiy, 2020) to extract vision features. While point clouds from LiDAR consist of set of unordered points in Euclidean space, necessitating point cloud-specific frameworks like Point Net (Qi et al., 2017) and Point Transformer (Zhao et al., 2021) to process them. Thus, we introduce modelity-specifie feature encoders in X-Fi to extract features from various modalities.
In X-Fi, the -peecffi feature encoder E, (-: (g,) for each modality consists of a pretrained feature extractor F, (-: (p.)) and linear projection head. The feature extractors are responsible for extracting informative nigh-dimensional representations from modal data. To accommodate diverse combinations of sensor modalities across different application scenarios, we can select any model structures backbones for these feature extractors. Each feature extractor is modality specified, which allows us to choose a model structure that best aligns with the unique characteristics of the modality's 's data, thereby ensuring the efficiency of feature extraction. For example: for point clouds from mm Wave radar and LiDAR, we adopt the Point Transformer backbone (Zhao et al., 2021). ) The self-attention mechanism in Point Transformers is particularly suitable for point cloud data due to its permutation invariance and ability to dynamically focus on relevant points.
Since the data structures of the features extracted by different modality-specine feature extractors still different across modalities, we use a set of linear projection structures to unify these features into the same dimensional space, allowing them to be used together in subsequent stages of the model. Finally, positional encoding is added to obtain the final features.
(E.(::g.))) =
(Fl::
(1)
4

---

Published as a conference paper at ICLR 2025
3.2 ATTENTION-BASEI X-FUSION
After obtaining the modality-specific features, a natural question is posed: How can the modality- invariant foundation model effectively combine these features to obtain a unified cross-modal repre- sentation that is applicable for downstream human sensing tasks? Thus, in this section, we will in- terpret the proposed X-Fusion structure which consists of cross-modas modsl transformer to learn unified cross-modal features and cross-attention fusion modules to preserve distinctive modality features.
3.2.1 CROSS-MODAL TRANSFORMER
In this subsection, aim to learn unified cross-modal feature. The proposed structure should be capable of dynamically handling varying numbers of modality features and effectively achiev- ing information interaction among diverse modal features. Thus, we adopt Transformer-basee structure (Vaswani et al., 2017) that can accept variable sizes of input. Also, the self-attention mech- anism can equally learn the long-distance dependencies between any two modal features in the input, thereby effectively modeling global information for all modal features.
First, we concatenate all obtained coddality-speciffe features initialize a multi colli-modal embedding Fmom E [R(ysk))dd As illustrated in the Figure 2, we then linearly map the multi-moda embed- ding into three vectors, named Q @mm K mmaz Vmen- Next, we perform dot-product-basee brosect-based multi-head attention to calculate the attention value which models the dependencies among modality features.
Notably, the multi-head attention and feed-forward blocks output embedding of the same size as the input, resulting in variable-sized cross-modal embedding based on the number of modality inputs. This variability complicates the design of subsequent fusion blocks and task-specific projection heads which have fixed-size parameters. To address this issue, we apply adaptive pooling both after the multi-head attention block and within the residual connection to perform downsampling and obtain unified-sizes outputs 0mm R Xdy, Finally, we feed Oym into feed-forward structure with residual connection to obtain the unified modal embedding Zem Rdy
mmm CogPPoolMMultHHeee (Q Kmm, Vmm)) + AvgPool(zmnl
(2)
Zem FFFN (0 + Oym
(3)
The adaptive average pooling performs the dimension reduction for the multimodal feature and the sequential feature, which are summed up for the following layer. Then the random modality drop- ping in training ensures that all possible combinations of modalities are learned by X-Fi, implicitly aligning the feature of each modality. Therefore the final feature can be modallit-innariann demon- strated by massive empirical studies using 38 modality combinations on two large-scale datasets.
3.2.2 CROSS-ATTENTION MOLTI-MODAL FUSION
However, the basic transformer design, which has a layered architecture with sequentially stacked transformer blocks, often prioritize certain modalities during training and the inference performance will rely on the existence of the dominant modalities. The reason is that the self-attention mechanism in the cross-modal module tends foeus the more intuitive and informative modality-speciffe features during training (Lu et al., 2019). Such focus results in significant information loss from less dominant modalities, compromising robust and equitable performance across different scenar- ios where dominant modalities may be distorted or missing (Ma et al., 2022; Memmel et al., 2023). To address this, we propose cross-attention fusion modules that reintegrate overlooked modality- specific information into the cross-modal embeddings, thereby encouraging effective learning from less-informative modalities during training. Rather than impeding the features extraction from more informative modalities, such approach enhances the model's ability to capture complementary fea- tures across modalities, improving robustness across diverse application scenarios. We also design an X-Fusion process that iterates on the same block, compelling the X-Fusion block to identify unified dynamic attention allocation method that effectively preserves the features of each modality and optimally integrates them to achieve the most optimal solution.
Key- Value Generators. To prepare key-value pairs that preserve modality-specific information,
we design key-value generators for each modality. As illustrated in the Figure 7, key-v generator Gkv. receives modality specific feature fs E R extracted from feature encoder E, as the input. fo is first passed through a two-layer MLP with a hidden dimension of y incorporating Rectified I Linear Unit (ReLUU) activation (Nair & Hinton, 2010) and layer normalization in between. The MI.P
5

---

Published as a conference paper at ICLR 2025
output is then projected to key K; and value Vi using separate linear mapping layers. Once the key- value pair (K₃, V2)) is generated from ft. it remains unchanged during iterations in the X-fusion block to prevent information leakage between modalities.
Cross-Attention Modules. We employ series of cross-attention modules to integrate modality- specific information into the cross-modal embedding Zem- Each transformer is associated with a specific modality and receives three inputs: the unified cross-modal embedding 2em that is shared across all cross-attention modules, the key (K) and value (V) pairs obtained from modality-specine key-value generators. Take the i-th modality as an example, as illustrated in the Figure 7, Rem serves as the query (Q) in the multi-head attention mechanism. Attention weights are computed based on Q and K; to scale the modality-specinf value V,, yielding an intermediate result denoted as Oca,á- Subsequently, Zem is residual added to Oca,i to retain cross-modal features, resulting in Oza.i- The final output Jr of each modality-specific cross-attention module is produced by passing oza, through a feed-forward structure with a residual connection. Each f' generated from the modality- specific cross-attention module encapsulates fused information focusing on a single modality. To integrate information from all modalities, we concatenate all (f1) to get the updated multi-modal embedding zmm which is used in the subsequent X-fusion block iteration.
Concat Goo, (zems Ki, Vi; ( OG.)
(4)
mm
3.3 MODALITY-INVARIAN TRAINING
To train our modality invariant foundation model, we design an explicit training strategy with refer- ence to (Memmel et al., 2023). First, to ensure that the feature extractors retain their ability to cap- ture modality-speciffe features without being biased, we pre-train the parameters on the respective datasets and freeze these parameters during subsequent training. Next, to simulate versatile com- binations of sensor modalities in different scenarios, we use a modality existence list containing n elements to indicate whether each modality is present in the input modal combination. This modality existence list is utilized to control the presence of input modalities during each training iteration and activate the corresponding modality-specific structures in the X-Fi. In addition, each modality is signed an independent occurrence probability pt. By adjusting the modality-correspondee pt, we can refine the model's ss training strategy by prioritizing certain modalities. For example, because LiDAR point cloud data is inherently sparser than image data and exhibits complex features such as permu- tation invariance, it necessary include more LiDAR samples in the training process to ensure accurate model fitting. Since the training sample is controlled by the modality existence list, we can model the training samples by the distribution of the modality existence list. We denote the count of occurrences of i-th modality in training iterations by variable Ki, and K; ~ Binomial (m. pt). Since K; for each modality is independent, we can describe the distribution using a joint probability mass function P(K₁ k1, ... Kn = kn) = II. (r) the modality existence list is sampled from a distribution to regulate the input modalities, thereby influencing the activation of modality-specific model structures.
p (1 - p,) For each training batch,
4 EXPERIMENTS
4.1
EXPERIMENTAL. SETUP
Datasets. We train and evaluate our proposed X-Fi on the two largest human sensing multimodal public datasets, MM-Fi (Yang et al., 2024) and XRF55 (Wang al., 2024), to assess its efficiency as a unified modality-invarian foundation model across diverse human sensing tasks, including Human Pose Estimation (HPE) and Human Activity Recognition (HAR), The dataset details for each task are as follows: (i) HPE: MM-Fi consists of over 320k synchronized frames from 40 human subjects. Each synchronized frame consists of samples collected from 5 sensing modalities, including RGB images (I), Depth images (D), LiDAR point clouds (L), mm wave point clouds (R), and WiFi-CSI (W). The chosen annotation type is 3D whole-body key points, representing the positions of 17 human joints. (ii) HAR: MM-Fi and XRF55 are utilized for HAR task evaluation. MM-Fi consists of 27 action categories, including 14 daily activities and 13 rehabilitation exercises. We exclude WiFi-CSI modality from MM-Fi when addressing the HAR task, as each WiFi data sample spans only 100ms, which is insufficient to effectively capture the distinct data patterns associated with different actions. XRF55 includes 42.9K radio frequency samples collected by 3 sensing modalities,

---

Published as conference paper at ICLR 2025
Metrics
MPJPE
PA-MPIPE
MM-Fi
X-Fi
Baseline
Impt
Baseline2
Impp
X-Fi
Basclinel
Impt
Baseline2
Impt
I
93.9
121.0
22.4%
121.0 0
22.4%
60.3
68.0
11.4%
68.0
11.4%
D
101.8
102.4
0.6%
102.4
0.6%
48.4
52.7
8.2%
52.7
8.2%
L
167.1
161.5
-3.5%
161.5
-3.5%
103.2
103.5
0.3%
103.5
0.3%
R
127.4
141.3
9.8%
141.3
9.8%
72.4
3.7%
72.4
3.7%
W
225.6
227.1
0.7%
227.1
0.7%
105.3
108.0
2.5%
108.0
2.5%
I+D
96.7
11.0%
109.2
21.2%
48.1
53.9
10.7%
57.1
15.6%
I+L
93.0
122.1
23.8%
121.2
23.3%
59.7
75.6
21.1%
73.0
18.2%
I+R
88.8
101.0
12.1%
140.6
36.8%
57.3
57.3
0.0%
68.2
16,0%
1AW
93.0
146.4
36.5%
156.7
40.7%
39.5
77.3
23.1%
86,3
31.1%%
D+L
102.5
111.7
8.2%
108.0
5.1%
48.4
68.8
29.7%
55.8
13.3%
D+R
98.0
94.1
-4.1%
116.4
15.8%
47.3
51.8
8.6%
57.0
17.0%
D+W
101.8
141.7
28.1%
155.5
34.5%
48.1
71.4
32.6%
81,5
41.0%
L+R
109.8
116.4
5.7%
137.0
19.9%
63.4
71.5
11.3%
71.9
11 11.8%
LAW
159.5
167.1
4.5%
206.2
22.6%
102.7
100.7
-2.0%
109.3
6.0%
ROW
117.2
144.2
18.7%
141.3
17.0%
62.7
72.2
13.0%
77.6
19.1%
I+D+L
84.8
102.9
17.5%
104.5
18.8%
48.2
62.3
22.7%
61.2
21.4%
1+DRR
83.4
86.7
3.8%
113.4
26.5%
47.3
50.1
5.4%
58.0
1+++W
85.3
118.6
28.0%
137.8
38.1%
48.1
63.7
24.4%
75.3
36.1%
I+L+R
88.4
101.4
12.8%
135.0
34.5%
57.2
62.7
8.8%
67.7
15.6%
IL+WW
93.0
134.9
31.1%
156.2
40.5%
39.7
82.0
27.2%
84.8
29.6%
[+RW
88.5
116.5
24.1%
128.9
31.3%
57.1
63.1
9.5%
75.0
23.9%
D+LAR
95.3
-0.8%
111.3
13.7%
19.2%
58.4
D+L+W
102.0
130.7
22.0%
154.6
34.0%
48.1
78.1
38.4%
84.1
42.8%
D+R+W
97.0
113.2
14.3%
132.0
26.5%
47.1
59.4
20.7%
71.8
34.3%
L+R-W
107.4
129.6
17.2%
141.6
24.1%
63.1
78.1
19.2%
77.0
18.1%
I4D+L+R
83.5
90.7
7.9%
109.4
23.6%
47.6
55.8
14.6%
61.6
22.8%
IDD+L+W
86.0
116.7
26.4%
134.6
36.2%
48.2
70.0
31.2%
75.4
36.0%%
IDDR+W
84,0
101.9
17.6%
128.6
34.7%
15.5%
71.3
1+L+R+W
88.6
114.0
22.3%
174.6
49.2%
57.1
69.5
17.8%
78.1
26.8%
D+L+R+W
97.6
110.8
11.9%
147.6
33.9%
47.4
66.7
28.9%
73.1
35.1%
1+D+L+R-W
83.7
103.0
18.8%
128.9
35.1%
47.6
62.4
23.8%
72.5
34.3%
Table 1: Performance comparisons X-Fi with baseline methods on the MM-Fi dataset for HPE task. "Baselinel" denotes the decision-leve fusion results and "Baseline2" denotes the feature-level fusion results. Imp denotes the improvement achieved over baseline in percentage level.
including mm Wave Range-Dopplle Doppler & Range-Angle Heatmaps (R), WiFi-CSI (W), and RFID phase series data (RF). XRF55 comprises 55 action classes collected from 39 human subjects, covering categories such fitness activities and human-computer interactions. We follow the S1 Random Split for MM-Fi and the original split setting for XRF55 as outlined in their respective papers.
Training Objective. Different loss functions are applied as the training objectives for different human sensing tasks. For HPE, Sean-Squared-FFroo is chosen as the loss function, which measures the average squared euclidean distance between each of the 17 predicted whole-body key points and their corresponding ground truth points in 3D space. For HAR, Cross-Entroppy loss is applied between the predicted probability distribution and the ground truth distribution.
Baseline Formulation. To evaluate the performance of our proposed unified modality-invanl invariant foundation model across various human sensing tasks and modality combinations, we formulate baseline models by adopting feature fusion approach (Das et al., 2021; Mareard al., 2016) and decision-level fusion approach (Yang et al., 2024). For the feature fusion approach, X-Fi uses pre- trained modality-specifft extractors to obtain modality features. Single-modal inputs are projected into the task space via an MLP, while multi-modal inputs concatenated and fused using MLP with average pooling to map them into the unified task representation space. The baseline models are retrained each modality combination separately. For different tasks, retrain baseline models' weights (including the pre-trained feature extractor) using different learning objectives. For the decision-level fusion approach, Then, we calculate the average of the outputs from all modalities in the combination to generate the fused output. The bascline results are presented in the Table 1&
first obtain the final-layer output of each individual modality.
Implementation Details. We conduct experiments all human sensing tasks and datasets with the following model settings. To standardize feature representations obtained by various modality- specific feature extractors, tation to n, = 32, each with a feature dimension of d, = 512. The positional embedding, derived from LiDAR 3D point cloud raw data due to its inherent capture of human body spatial information,
apply linear projection units to map each modality feature represen-

---

Published as a conference paper at ICLR 2025
is only provided in the multi-modal embedding if the LiDAR modality is present in the input. The backbone for both the cross-modal transformer and each poodallit-speccfffe cross-attention module consists of 1 1-layer decoder-only transformer structure with 8 multi-heat attention heads and scaling factor of 0.125. Hach modality-speciff key-value pair and the cross-modal embedding in X-Fusion block comprise 32 features with feature dimension of 512. The number of iterations on the X-Fusion block is set to a default of 4 in our experiments. The Adam optimizer, with an initial for HAR, is chosen for model optimization. The
learning rate of e- for HPE and training process is performed with a batch size of 16 on an NVIDIA GeForce RTX 4090 GPU.
4.2 QUANTITATIVE RESULTS
We compare our approach to baseline methods in downstream hu- man sensing tasks, including HPE and HAR. X-Fi results are ob- tained by evaluating single X-Fi model. only trained once using the proposed modality-invarian training strategy, across various modality combinations during testing. And the baseline results are retrained on each modality combination separately. For con- venience, we refer the decision-level fusion baseline "base- linel", 1,, and the feature-level fusion baseline as "baseline2", The results for X-Fi, as shown in Tables 1& represent average val- ues obtained from multiple experiments.
Table 2: HAR accuracy (%) on the MM-Fi and XRF55 datasets.
Modality
Baseline
X-Fi
Imp 1
MM-Fi
25.3
26.5
1.2%
D
49,9
48.1
-1.8%
L
63.9
52.7
-11.2%
R
85.0
85.7
0.7%
D
45.2
45.3
0.1%
I+L
23.8
35.2
11.4%
66.9
73.4
6.6%%
D+L
49.7
51.6
1.9%
74.1
79.8
5.7%8
Human Pose Estimation. We assess the performance of X- Fi on the MM-Fi dataset (Yang al., 2024) by adopting two commonly-used metrics in HPE: Mean Per Joint Position Er- ror (MPJPE) and Procrustes Analysis MPJPE (PA-MPJPE), For single-modal inputs, X-Fi improves the averaged MPJPE and PA- MPJPE by 6.0% and 5. 5.2%, respectively. For the dual-modality inputs, the averaged MPJPE and PA-MPJPE are improved re- spectively by 14.5% and 14.8% on baselinel, 23.7% and 18.9% on baseline2. Notably, for LiDAR and WiFi modalities, when used as individual inputs, X-Fi does not achieve good perfor- mance, with MPJPE difference of -3.5% and +0.7%, respec- tively. However, when the two modalities are combined, the performance improves significantly, surpassing the baselinel by 4.5% and the baseline2 by 22.6%. For the triple-modality inputs. X-Fi demonstrates an average improvement of 17.0% and 28.8% in MPJPE, 19.5% and 25.9% in PA-MPJPE. For the combined input of four modalities, the results indicate an average enhancement of 17.0% and 35.5% in MPJPE, 21.6% and 30.8% in PA-MPIPE. For all five modalities combined input, MPJPE increases by 18.8% and 3%, PA-MPJPP increases by 23.8% and 34.3%.
L+ R
85.6
88.7
3,2%
48.7
5.7%
D+
69.1
70.7
1.618
L+L+R
68.2
77.8
9.6%%
D+L+R
72,9
80.5
7.6%
TAD+LAR
72.6
72.2
-0.4%
XRF55
R
82.1
83.9
1.9%
W
77.8
55.7
-22.1
RF
42.2
42.5
0.3%
R+W
86,8
88.2
1.4%
R+RF
71.4
86.5
15.1%
W+RF
55.6
58.1
2.6%
R+W+RF
70.6
89.8
19.2%
Human Activity Recognition. The performance of X-Fi on the HAR task is evaluated by classi- fication accuracy on both the MM-Fi (Yang et al., 2024) and XRF55 (Wang et al., 2024) datasets. On the MM-Fi dataset, X-Fi performs better than the baseline when handling multimodal inputs, yielding an average improvement of 2.8%. But when dealing with single-modality inputs, its per- formance declines, dropping by 2.8%. On the XRF55 dataset, X-Fi only encounters challenges with WiFi-CSI single-modality input. It outperforms the baseline on all other modality combinations, achieving an average improvement of 6.7%. Notably, while X-Fi's performance is suboptimal when using WiFi as a single input, when combined with other modalities, WiFi not only avoids hindering performance but also introduces unique information that enhances classification accuracy. For ex- ample, when X-Fi processes mm Wave radar and RFID multimodal input, it outperforms the bascline by 15.1%. %. With the inclusion of WiFi, this improvement increases to 19.2%.
4.3 ANALYTICS
Impact of modality existence list. To assess the impact of adjusting the independent existing probability of each modality in the modality existence list, we conduct experiments with different probability combinations on the XRF55 dataset for the HAR task. The XRF55 modality existence list contains the independent existence probabilities of mm Wave Radar, WiFi-CSI, and RFID modal- ities in sequential order. As shown in Table 3, X-Fi encounters difficulties in dealing with WiFi-CSI. However, through gradually increasing the existing probability of WiFi-CSI, the model is priori-
8

---

Published as a conference paper at ICLR 2025
tized to fit WiFi-CSI modality during training, which yields significant improvement from 29.1% to 55.7% in handling single WiFi-CSI input. Additionally, we are surprised to find that prioritizing particular modality does not degrade the model's ability to extract features from the other two modalities. Consequently, the multimodal fusion results are also improved a lot.
XRF55
Accuracy (%)
MM-Fi
MPJPE (mm)
PA-MPJPE (mm)
(RR.Pu.PRF)
(0.5.0.5,088)
(0.5,0.7.0.7)
(0.5,0.9.0.6)
Method
VO Transformer
X-Fi
VO Transformer
X-Fi
82.2
82.4
83.9
99.6
93.9
65.0
60.3
29.1
55.7
212.2
167.1
109.6
103.2
RF
41.6
41.9
42,5
R
138.0
127.4
74.6
69.8
R+W
85.1
86.0
88.2
243.3
225.6
112.9
R+RF
84.5
84.8
86.5
1. R
123.4
109.8
69.9
63,4
W+RF
50.3
52.7
58.1
L+ W
216.3
159.5
112.7
102.7
R+W-RF 86.4
87.3
89.8
I+L+ W 96.6
93.0
61.5
59.7
Table 3: Comparisons X-Fi trained with different modality
Comparison between the SOTA method (Visual Odometry Transformer and X-Fi on the HPE task.
Table 4:
existence lists. The probabilities are shown in decimal form.
Comparison between existing SOTA (Visual Odometry Transformer) and X-Fi. To demon- strate the efficiency of X-Fi on modality-invarian human sensing tasks, we conduct comparison with existing SOTA method (Memmel et al., 2023) that utilize basic transformer encoder consisting of four stacked blocks to performance modality fusion. A subset of comparison results are analyzed in Table 4 and the full findings are provided in the appendix 5. As shown in Table 4, X-Fusion most of the modality combination input. First, when handling single modality input, transformer- based fusion struggles with challenging modalities, whereas X-Fusion effectively captures the distinctive features of each modality. For example, performance with LiDAR input improves by 21.3%. Next, in contrast to transformer-based fusion methods, X-Fusion lever- ages each modality's strengths, yielding better fusion results than single-modality inputs. X-Fusion demonstrates superior performance compared to either the LiDAR or WiFi-CSI modality when used individually. Lastly, in the case of multi-modality inputs, the performance of transformer-bast sion is highly dependent on the presence of dominant modalities. For example, adding the RGB modality improves the performance of the LiDAR and WiFi-CSI multi-modal inputs by 119.7 mm with the transformer backbone, whereas the corresponding improvement with X- X-Fusion is 66.6 mm.
yields better performance
fu-
Comparison between X-Fusion and its variants. To assess the effectiveness of the X-Fusion structure, which iterates multiple loops within the same block for multimodal feature fusion, we design two variants. The first employs a layered architecture with four sequentially stacked X- Fusion blocks, where each layer learns new key-value pairs from the multimodal embedding of the previous layer. The second variant is similar but uses the same key-value pairs across layers, learned from the embedding generated by the modality feature encoder. We conduct experiments on both variants, with full results provided in the appendix 5. On average, the first variant demonstrates performance improvement of +1 +1.27 mm for MPJPF and -0.7 mm for PA-MPJPE across all modality combinations. The second variant achieves improvement of 0.93 mm for MPJPE and 0.03 mm for PA-MPJPE. Considering the computational efficiency and the minor performance differences, retain the X-Fusion structure where an X-Fusion block is applied iteratively.
4.4 QUALITATIVE RESULTS
To intuitively illustrate the performance of X-Fi on the human sensing tasks, we visualize the esti- mated human skeleton generated by the model in the HPE task in the Figure 3 and employ the t-SNE method (Van der Maaten Hinton, 2008) to visualize the multimodal embedding obtained by the X-fusion block for the HAR task in the Figure 4.
Human Pose Estimation. The visualization results for HPE comprise two actions, 'picking up things' and 'throwing', each depicted through a sequence four images. In different rows, we pre- sented the results for the RGB and depth single-modal inputs, the fused results of the RGB and depth multimodal inputs, and the ground truth. To facilitate a clearer comparison between the fused results and the single- modal inputs results, images to represent the results of the RGB and depth mnnll-mmdallity inputs, respectively. Through observation, we found that the RGB modality provides more accurate estimations of leg movements, while the depth modality achieves greater accuracy in estimating hand movements. By combining both modalities into multimodal input, X-Fi effectively retains the leg movement accuracy from the RGB modality and the hand movement accuracy from the depth modality, resulting in fused outcomes that are closest to the ground truth.
incorporated blue and orange dashed lines in the fused result

---

Published as a conference paper at ICLR 2025
Desth
BGB
Ground Truth
(a): Picking things
(b): Throwing
Figure 3: Comparison of predicted human skeletons.
St Pasectio Nuaite 23
5d soutte Score-0-43
WIF-CSI (W)
MmWave Radar (R)
WiFi-C5I + MmWave Radar (W+R)
Comparison of multi-modal embedding distribution for HAR. The upper right corner of the image provides enlarged view of the red-boxed area from the original image.
Figure 4:
Human Activity Recognition. The visualization results for HAR include the multimodal embed- ding distributions from WiFi-CSI and mmWave Radar single-modal inputs, as well as the combined WiFi-CSI and mmWave Radar multi-modal inputs. To more closely analyze the distribution of sample points, we zoomed in on small region containing points from two distinct categories. To quantify the distribution, we used the Silhouette score and Calinski of clustering quality. The Silhouette score assesses how well an object fits within its own cluster compared to others, ranging from -1 1, with higher values indicating better separation and cohe- sion. Similarly, higher Calinski-Harabasz index values reflect more distinct clusters. We observed that the multimodal input results exhibit superior feature discriminability. While the mm mawa Radar single modal results demonstrate good classification capability, closer inspection of the zoomed-in in region reveals that the multimodal imput results form tighter clusters. Furthermore, a comparison of the indicator values confirms that the multimodal results outperform any single-mon moglee results.
index as indicators
CONCLUSION
In this paper, we introduce X-Fi, a modality-invariant foundation model designed to overcome the limitations current human sensing approaches, which often rely on fixed modality combinations. By leveraging transformer architecture and novel X-fusion mechanism. X-Fi supports flexible multimodal integration and effectively utilizes distinctive modal information. Our extensive exper- iments the MM-Fi and XRF55 datasets demonstrate the model's superior performance in both human pose estimation (HPE) and human activity recognition (HAR) tasks. These results highlight X-Fi's potential to drive advancements in scalable, multimodal human sensing technologies.
10

---

Published as a conference paper at ICLR 2025
Acknowledgement. This work is supported by Start-up Grant from Nanyang Technological Uni- versity and jointly funded by the Singapore MOE Tier-1 research grant.
REFERENCES
Sizhe An and Umit Y Ogras. Mars: mmwave-based assistive rehabilitation system for smart health- care. ACM Trunsactions Embedded Computing Systems (TECS), 20(5s):1-22, 2021.
Sizhe An and Umit Y Ogras. Fast and scalable human pose estimation using mmwaae point cloud. In Proceedings of the 59th ACM/IEEE Design Automation Conference, pp. 889-894, 2022.
Sizhe An, Yin Li, and Umit Ogras. mri: Multi-modal 3d human pose estimation dataset using mmwave, rgb-d, and inertial sensors. Advances in Neural Information Processing Systems, 35: 27414-27426, 2022.
XB Bruce, Yan Liu, and Keith CC Chan. Multimodal fusion via reacher-student network for indoor action recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 3199-3207, 2021.
Zhongang Cai, Daxuan Ren, Ailing Zeng, Zhengyu Lin, Tao Yu, Wenjia Wang, Xiangyu Fan, Yang Gao, Yifan Yu, Liang Pan, et al. Humman: Multi-moda 4d human dataset for versatile sensing and modeling. European Conference on Computer Vision, pp. 557-577. Springer, 2022.
Zhongang Cai, Wanqi Yin, Ailing Zeng, Chen Wei, Qingping Sun, Wang Yanjun, Hui En Pang, Haiyi Mei, Mingyuan Zhang, Lei Zhang, et al. Smpler-x: Scaling up expressive human pose and shape estimation. Advances Neural Information Processing Systems, 36, 2024.
Anjun Chen, Xiangyu Wang, Kun Shi, Shaohao Zhu, Bin Fang, Yingfeng Chen, Jiming Chen, Yuchi Huo, and Qi Yc. Immfusion: Robust mmwave-rgb fusion for 3d human body reconstruction in all weather conditions. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 2752 2758. IEEE, 2023.
Avigyan Das, Pritam Sil, Pawan Kumar Singh, Vikrant Bhateja, and Ram Sarkar. Mmhar -imhar-ensemnet:: A multi-modal human activity recogmition model. IEEE Sensors Journal, 21(10): 15691-166166991766 2021. doi: 10.1 109/ISEN.2000.3334550
Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
Alexey Dosovitskiy, An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 11.22000..1999,,, 2020.
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, al. Palm-e: An embodied multi- modal language model. arXiv preprint arXiv:2303.0333 2023.
Omar Elharrouss, Noor Almaadeed, and Somaya Al-Maadeed. A review of video surveillance systems. Journal of Visual Communication and Image Representation, 77:103116, 2021.
Qing Gao, Jinguo Liu, Zhaojie Ju, and Xin Zhang. Dual-hand detection for human noman-roboot interaction by parallel network based on hand detection and body pose estimation. IEEE Transactions on Industrial Electronics, 66(12):9663-967, 2019.
Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15180-15190, 2023.
Shubham Goel, Georgios Pavlakos, Jathushan Rajasegaran, Angjoo Kanazawa, and Jitendra Malik. Humans in 4d: Reconstructing and tracking humans with transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 14783-14794 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
11

---

Published as a conference paper at ICLR 2025
Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE transactions pattern analysis and machine intelligence, 36(7): 1325-11331111 1339, 2013.
Quan Kong, Ziming Wu, Ziwei Deng. Martin Klinkigt, Bin Tong, and Tomokazu Murakami. Mmact: A large-scale dataset for cross modal human action understanding. In Proceedings of the IEFE/CVF International Conference on Computer Vision, pp. 8658-8667, 2019.
Yann LeCun, Léon Bottou, Yoshua Bengio. and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2222 1998.
Jialian Li, Jingyi Zhang, Zhiyong Wang, Siqi Shen, Chenglu Wen, Yuexin Ma, Lan Xu, Jingyi Yu, and Cheng Wang. Lidarcap: Long-range marker-less human motion capture with lidar point clouds. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 20502-20512, 2022.
Xinyu Li. Bing Shuai, and Joseph Tighe. Directional temporal modeling for action recognition. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VI 16, pp. 791 Springer, 2020.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolin- guistic representations for lision-and-langus tasks. Advances neural information processing systems, 32, 2019.
Mengmeng Ma, Jian Ren, Long Zhao, Davide Testuggine, and Xi Peng. Are multimodal transform- ers robust to missing modality? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18177-18186, 2022.
Yunze Man, Liang-Yee Yan Gui, and Yu-Xiong Wang. Bev-guided multi-modality fusion for driving perception. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 21960-21 21969, 2023.
Timo Marcard, Gerard Pons-Moll, and Bodo Rosenhahn. Human pose estimation from video and imus. IEEE Transactions on Pattern Analysis and Machine Intelligence, 3833)11533-1547, 2016. doi: 10. 001109/TPAMI.221122223399
Marius Memmel, Roman Bachmann, and Amir Zamir. Modality-invariant visual odometry for em- bodied vision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21549-21559, 2023.
Seungwhan Moon, Andrea Madotto, Zhaojiang Lin, Aparajita Saraf, Amy Bearman, and Babak Damavandi. Imu2clip: Language-groundee motion sensor translation with multimodal con- trastive learning. In Findings of the Association for Computational Linguistics: EMNLP 2023,
pp. 13246-1 13256-13253, 2023.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), pp. 807-814, 2010.
Jeffrey A Nanzer. A review of microwave wireless techniques for human presence detection and classification. IEEE Transactions Microwave Theory and Techniques, 6565): 2017.
Duc Thanh Nguyen, Wanqing Li, and Philip O Ogunbona. videos: A survey. Pattern Recognition, 51: 31:148-172 175, 2016.
Human detection from images and
Isura Nirmal, Abdelwahed Khamis, Mahbub Hassan, Wen Hu, and Xiaoqing Zhu. Deep learning for radio-based human sensing: Recent advances and future directions. IEEE Communications Surveys & Tutorials, 23(2):995-1019 2021.
Georgios pavlakos, Luyang Zhu, Xiaowei Zhou. and Kostas Daniilidis. Learning to estimate 3d human pose and shape from a single color image. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 459-468 2018.
12

---

Published as a conference paper at ICLR 2025
Charles R Qi. Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 652-660) 2017.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. PMLR, 2021.
Yiming Ren, Xiao Han, Chengfeng Zhao, Jingya Wang, Lan Xu, Jingyi Yu, and Yuexin Ma. Livehps: Lidar-based scene-level human pose and shape estimation in free environment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 128 1281-1291, 2024.
Amir Shahroudy, Jun Liu, Tian-Tsong Ng, and Gang Wang. Ntu rgb+ d: A large scale dataset for 3d human activity analysis. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1010-1019, 2016.
Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5693-5703 2019.
Catherine Tong. Jinchen Ge, and Nicholas D Lane. Zero-shot learning for imu-based activity recog- nition using video embeddings. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 5(4):1-23, 2021.
Alexander Toshev and Christian Szegedy. Deeppose: Human pose estimation via deep neural net- works. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 1653-1660, 2014.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière. Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(1)), 2008.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
Michalis Vrigkas, Christophoros Nikou, and Ioannis A Kakadiaris. A review of human activity recognition methods. Frontiers in Robotics and AI, 2:28, 2015.
Fei Wang, Yizhe Lv, Mengdie Zhu, Han Ding, and Jinsong Han. Xrf55: A radio frequency dataset for human indoor action analysis. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 8(1 8(1):1-34, 2024.
Qi Wang, Liang Zhan, Paul Thompson, and Jiayu Zhou. Multimodal learning with incomplete modalities by knowledge distillation. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1528-18366 2020.
Yuheng Wang, Haipeng Liu, Kening Cui, Anfu Zhou, Wensheng Li, and Huadong Ma. m-activity: Accurate and real-time human activity recognition via millimeter wave radar. In ICASSP 2021- 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp- 8298 8302. IEEE, 2021.
Jiannan Wu, Yi Jiang, Bin Yan, Huchuan Lu, Zehuan Yuan, and Ping Luo. Uniref++: Segment every reference object in spatial and temporal spaces. arXiv preprint arXiv:2312.157155 2023.
Zihui Xue and Radu Marculescu. Dynamic multimodal fusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2575-25854 2023.
Zihui Xue, Zhengqi Gao, Sucheng Ren, and Hang Zhao. The modality focusing hypothesis: Towards understanding crossmodal knowledge distillation. arXiv preprint arXiv:2206. 06487, 2022.
13

---

Published as a conference paper at ICLR 2025
Santosh Kumar Yadav, Muhtashim Rafiqi, Egna Praneeth Gummana, Kamlesh Tiwari, Hari Mohan Pandey, and Shaik Ali Akbara. novel two stream decision level fusion of vision and iner- tial sensors data for automatic multimodal human activity recognition system. arXiv preprint arXiv:2306. 15765, 2023.
Jianfei Yang, Xinyan Chen, Han Zou, Chris Xiaoxuan Lu, Dazhuo Wang, Sumei Sun, and Lihua Xie. Sensefi: A library and benchmark on heep-leaming-empowee wifi human sensing, Patterns, (3), 2023.
Jianfei Yang, He Huang, Yunjiao Zhou, Xinyan Chen, Yuecong Xu, Shenghai Yuan, Han Zou, Chris Xiaoxuan Lu, and Lihua Xie. Mm-fi: Multi-modal non-intrusive 4d human dataset for versatile wireless sensing. Advances in Neural Information Processing Systems, 36, 2024.
Yiyuan Zhang, Kaixiong Gong, Kaipeng Zhang, Hongsheng Li, Yu Qiao, Wanli Ouyang, and Xi- angyu Yue. Meta-transformer: A unified framework for multimodal learning. arXiv preprint 1xXiv23307.1002. 2023.
Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 16259-16268, 2021.
Jingxiao Zheng, Xinwei Shi, Alexander Gorban, Junhua Mao, Yang Song, Charles Qi, Ting Liu, Visesh Chari, Andre Cornman, Yin Zhou, al. Multi-modal 3d human pose estimation with 2d weak supervision in autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4478-4487, 2022.
Yunjiao Zhou, He Huang, Shenghai Yuan, Han Zou, Lihua Xie, and Jianfei Yang. Metafi++: Wifi- enabled transformer-basee human pose estimation for metaverse avatar simulation. IEEE Internet of Things Journal, 10(16):1411 14128----81111 4128-11136,1,1111111111111111111 2023.
14

---

Published as a conference paper at ICLR 2025
A
APPENDIX
MM-Fi
MPJPR
PA-MPIPE
Visual Odometry Transforner
X-Fusion (stacked, kv)
Visual Odometry Transfomerr
X-Fusion (stacked,
X-Fusion (iterative)
X-Fusion (stacked)
X-Fusion (itcrative)
X-Fusion (stacked)
Methods
kv)
I
99.6
93.9
100.6
95.8
65.0
60.3
62.2
61.3
D
109.0
101.8
94.8
104,5
56.1
48.4
50.8
48.8
L
212.2
167.1
184.9
160.2
109.6
103.2
104.7
103.6
138.0
127.4
122.8
65.4
W
243.3
225.6
229.7
224.3
112.9
105.3
105.4
103.2
I+D
89.7
86.1
81.1
84.9
$2.9
48.1
50.0
48.1
I+L
96.6
93.0
98.8
93.2
62.4
59.7
60.9
60.2
I+ R
93.0
88.8
91.9
89.8
59.1
$7.3
57.5
58.3
96.0
93.0
94.0
61.9
60,9
D+L
102.5
91.4
97.9
$2.3
49.5
48.0
D+ R
94.9
98.0
88.4
96.8
50.6
47.3
48.0
47.3
D W
95.6
101.8
94.5
98.8
51.4
48.1
49,6
48.0
L+R
123.4
109.8
115.5
106.9
69.9
63.4
64.1
216.3
159.5
156.4
112.7
102.7
103.9
102.5
R+W
126.1
117.2
120.8
117.4
70.0
62.7
63.9
62.7
D+D+L
86.8
84.8
78.6
84.2
$2.2
48.2
49.3
48.0
I+D+R
85.2
83.4
78.2
83.8
50.9
47.3
48.7
47.5
I+D+W
85.6
85.3
79.6
84.5
$1.5
48.1
49,3
48.1
I+L+R
93.4
88.4
93.0
59.1
57.1
[+L+W
96.6
93.0
98.1
93.0
61.5
59.7
60.1
60.2
1+R+
93.3
88.5
91.8
90.0
58.8
57.1
57.0
58.4
D+L+R
91.5
96.0
84.8
95.8
50.5
47.3
47.9
47.4
D+L+W
93.0
D
90.6
97.0
87.2
97.3
50.4
47.1
47.7
47.3
L+R+W
122.8
107.4
116.0
106.4
70.4
63.1
63.7
61.3
I+D+L+R
84.1
83,5
77.0
83.2
51.2
47.6
48.3
47.6
I+D+L+W
84.7
86.0
78.1
84.2
51.8
48.2
48.9
48.0
I+D+R+N
83.4
84.0
83.9
50.8
48.2
47.7
++L+R+W
93.9
88.6
92.7
89.5
59.1
56.8
58.4
D+L+R+W
89.4
97.6
84.4
95.8
50.7
47.4
47.7
47.4
1+++++R+
82.9
83.7
76.6
83.1
51.3
47.6
47.9
47.6
Table 5: Full results of the comparisons among Visual-Odometry Transformer based fusion X- Fusion, and X-Fusion's variants.
A.1 FEATURE EXTRACTOR PRE-TRAINING
mentioned the main work, to effectively extract informative high-dimensional representations from modal data, we designed modality-specifft feature extractors whose backbone structures could be selected based on the data structure of each modality and the characteristics the datasets. Mean- while, we employ pre-training techniques to preserve the efficiency of feature extraction. Initially, for each modality, we train an entire model on the respective dataset and task. portion of the model architecture is then repurposed the feature extractor, with the corresponding pre-trained weights frozen during subsequent training. For instance, a ResNet-18 18 model was first trained on RGB im- ages from the MM-Fi dataset perform HPE. Subsequently, the MLP structure of the model's output layer was discarded, retaining only the bottleneck blocks serving as the feature extractor. To identify efficient backbone structures, we conducted experiments validate the performance of various backbone structures on each dataset. The finalized backbones are presented below. When utilizing MM-Fi for HPE, we employ the ResNet-18 architecture (He et al., 2016) for both the RGB and Depth modalities, the Point Transformer architecture (Zhao et al., 2021) for LiDAR, the Point Transformer architecture without transition down layers for mm mave point cloud data, and the MetaFi++ architecture (Zhou et al., 2023) for Wifi-CSI. For HAR using MM-Fi, the same back- bone architectures are utilized, with the parameters specifically trained for HAR due to the differing feature representation requirements across tasks. When utilizing the XRF55 dataset for HAR, the ResNet- 18 architecture is adopted for the RFID, WiFi, and mm Wave modalities to maintain consis- tency with the baseline backbone networks employed in the study (Wang al., 2024).
15

---

Published as a conference paper at ICLR 2025
Ablation Studies on MM-Fi HAR
Dropout Layer
Positional Encoding
Accuracy
Accuracy
Methods
DPR 0.3
DPR 0.0
Falsc
Truc
1
26.1%
27.4% %
25.4%
27.4%
D
46.6%
47.3 %
47.0%
47.3%
L
12.8%
32.9 %
31.9%
32.9%
R
80.9%
86.0 %
85.5%
86.0%
I+D
46.3%
45.2%
45.9%
45.2%
I+L
27.1%
29.8% %
30.6 %
29.8%
I+R
69.2%
72.4 %
74.4% %
72.4%
D+L
46.2%
48.4% %
49.0%
48.4%
D+R
74.7%
80.2 %
78.6%
80.2%
L+R 79.3%
86.7 %
87.4%
86.7%
I+D+L
46.1%
46.1 %
69.9% 47.6%
46.15 1%
I+D+R
67.6%
70.8 %
70.8% %
I+L+R
66.9%
74.4% %
75.6 %
74,4%
D+L+R 72.6%
80.1 %
78.3%
80.1 %
I+D+L+R 65.8%
70.0 %
70.7
70.0%
Table Ablation studies on drop out layers and positional encoding layer.
A.2 EVALUATION OF MODALITY-PPECIFIC FEATURES COMBINING METIIODS
Our objective is assess whether concatenation is the most effective method for combining modality-specific features into multimodal embedding. To investigate this, we conducted a con- trolled experiment on the MM-Fi HPE task. In this experiment, we compared the concatenation method with the feature addition approach, where the feature values from each modality are summed and then divided by the number of modalities. As demonstrated in the Table 11, the feature addition approach failed to preserve distinctive modality features in the multimodal embeddings, resulting in average performance deteriorations of 15.7 mm on MPJPE and 5.9 mm on PA-MPIPE.
A.3 COMPARISON WITH SOTA MULTIMODAL FUSION APPROACHES
To demonstrate the the multimodal fusion effectiveness of our proposed X-Fi, we conduct compar- isons with SOTA multimodal fusion approaches that based on fixed modality combinations, e.g. ImmFusion (Chen et al., 2023) and Meta- Transformer(Zhagg et al., 2023). The experiments are con- ducted the MM-Fi dataset(Yan al., 2024) for the HPE task. For ImmFusion, model using the modality combination from the original paper (I+D+R) and then adapt the method include all modalities (I+D+R+L+W), For Meta- Transformer. we use the Teta-Transformer encoder as the backbone. Initially, we fine-tune the pretrained encoder weights (from the original repository) and train a regression head on all modalities (I+D+R+L+W). Next, freeze the en- coder weights and fine-tune the regression head on Rgh+Depth+LiDAR modalities, obtaining the Veeta-raanformer- (ifformmeerBBG((--DLL)) result. Finally, we fine-tune both the encoder and the regression head on the three modalities produce the Metaa-Tranffrmert -ransfomer-B666((--DL 116T((+D+L) result. The results shown the Table demonstrate X-Fi's superior performance.
first train the
A.4 ABLATION STUDIES
To demonstrate the effectiveness of our proposed moodality-invriian foundation model and the cor- responding optimization process, evaluate the impact of the dropout layer, positional encoding, Key- Value generators, the optimizer choice, the number of iterations on X-Fusion block, and the number of layers in cross-modal transformer. The results illustrated in the Table 6, 8,& 11.
Dropout Layer To evaluate the effect of dropout layers, incorporated dropout layer with rate of 0.3 into all transformer structures within X-Fi and assessed its performance on the MMFi dataset for the HAR task. The experimental results indicated that the introduction of the dropout
16

---

Published as a conference paper at ICLR 2025
Modalities
Methods
MPJPE (mm)
PA-MPIPE (mm)
Modalities
Methods
MPJPE (mm)
PA-MPIPE (mm)
Meta- BettaTransformer-B 16F
105.2
70.3
I+D+L
Meta- Beta-Traasfomme--B166.
96.2
68.1
X-Fusion
84.8
48.2
ImmFusion X-Fusion
207.3
146.8
I+ D+D
83.4
47.3
ImmFusion Meta-Transformer-B116
107.9
63.7
++D+L+R+W
91.4
64.9
X-Fusion
83.7
47.6
Table Comparison between X-Fi and SOTA multimodal fusion approaches, e-g. ImmFusion and Tett-Trannformer.
layer led to a 4.6% decrease in average accuracy. Furthermore, the dropout layer significantly im- paired the model's predictive performance, particularly when handling challenging modalities.
Optimizer Choice. To assess the impact of different optimizer choices on experimental results, conducted a set of controlled experiments using the SGD optimizer with a learning rate scheduler on the MM-Fi HPE task. The results indicated that the Adam optimizer facilitated better convergence to more optimal local minimum, leading to an average performance improvement of 50.9 mm on MPJPE and 44.6 mm on PA-MPJPE.
Positional Encoding. To assess the impact of LiDAR positional encoding. we conducted an ex- periment by removing the positional encoding structure. The results demonstrated that positional encoding provided slight improvement the performance of X-Fi with single-modality input. However, when faced with multi- modal inputs, the model's performance fluctuated, exhibiting worse results with certain modality combinations while performing better with others.
Kaluu Generators. To assess the impact Key- Generators. we conducted an exper- iment by removing the KeluValue Generators and directly use modality-specific features to inject information in the cross-attention module. The ablation study results shown in the Table 8 proves that utilizing key-value generators to obtain key-value pairs is more efficient than directly using initial modality features as k-v.
Ablation Studies on XRF55 HAR
Number of Iterations (#)
Key-Value Generators (with/without)
Depth of cross-modal transformer (#)
Methods
# 2
# =
without
with
#
1
80.6%
83.9%
82.0%
83.9%
82.8%
83.9%
W RF
24.2%
55.7 %
26.8%
55.7% %
54.8%
55.7 %
38.8%
42.5 %
40.0%
42.5%
40.5%
42.5%
R + W
84.1%
88.2 %
85.0%
88.2% %
87.1%
88.2%
R RF
84.1%
86.5 %
84.7%
86.5% %
84.2%
86.5 %
W +
48.0%
58.1 %
50.2%
58.1 %
54.6%
58.1
R W RF
86.3%
89.8%
86.7%
89.8 %
87.2%
89.8%
Table 8: Ablation studies on Key-Vaue Value generators, number of iterations on X-Fusion block, and number layers in cross-modal transformer.
Number of iterations on X-Fusion block. To assess the impact of the number of iterations on X-Fusion block, conducted an experiment by adjusting the number. The ablation study results shown in the Table demonstrate that when the number of iterations is set to 2, the model finds difficulties in digging and preserving features from weak modalities. We need the number of
17

---

Published as a conference paper at ICLR 2025
iterations the X-Fusion block at least repeatedly inject information from weaker modali- ties, enabling the model to more robustly all participating modalities during training.
Number of layers in cross-modal transformer. To assess the impact of the number of layers in cross-modal transformer, we have added the ablation of the number of the layers (depth) in cross- modal transformers on the XRF55 HAR task. The default depth in our experiments is and the depth we choose in the comparison experiment is 4. The results shown in the Table 8 indicate that adding the number of the layers the cross-modal transformer will not yield better performance.
A.5 HPE QUANTITATIVE RESULT INVESTIGATION
In order to have a more detailed look at the HPE quantitative result and show the observation that RGB modality captures better leg movements and depth modality is better for the arm, we have zoomed in the areas and the picture after enlargement reason for such observation, we explore the characteristics of the rgb data and the depth data in MM- Fi. We find that due to the arm's 's movement typically being faster than the legs, combined with the often blurry, while the legs
been added the Figure 5. To seek the
insufficient frame rate of the camera, the arms in the RGB samples appear very clear. This results in poor RGB-based estimation performance for the arms. In contrast, in depth images, the arm contours making their contours indistinct, which leads to poor depth- depth-based estimation performance for the legs. We also include two images samples from rgb and depth in the Figure 6.
very clear, but the legs almost blend into the background,
arm
BmM
am
Depth
RGB
Ground Truth
log
Figure 5: The detailed comparison among the rgb HPE result, the depth HPE result, and the ground truth.
RGB Image Sample
Depth Image Sample
Figure 6: The comparison between a rgb training sample and a depthtraining sample.
18

---

Published as a conference paper at ICLR 2025
A.6 CONTINUAL LEARNING ON X-FI
In order to validate the continual learning performance of X-Fi, we carry out experiment which gradually add modality each time. We first trained X-Fi model from seratch with RGB and Depth modalities. Then we added mm nawavv modality and fine-tuned the model based on the pretrained parameters. The results is shown the Table
MM-Fi
MPJPE (mm)
PA-MPJPE (mm)
train with Rgb and Depth
Fine-tune with mm Wave
train with Rgb and Depth
Fine-tune with mave
Methods
117.9 108.5
105.2
71.7
64.3
D
130.2
56.2
52.8
R
128.1
72.4
I+D
99.2
106.4
59.2
53.6
I+R
92.8
58.7
D+R
115.7
50.3
I+D+R
100.8
51.8
Table 9: Continual Learning performance of X-Fi.
Model Structure
Number of Parameters
X-Fusion block
17,903,667
(5 modalities)
X X-Fusion block
14,748,723
(4 modalities)
X- X-Fusion block
11,593,779
(3 modalities)
X Fusion block
8,438,835
(2 modalities)
X-Fusion block
5,283,891
(1 modality)
VO Transformer fusion block
12,636,723
Table 10: Comparison of number of parameters used in VO Transformer fusion block and X-Fusion block.
Porception
Linear
Miod
Forward +
Rcm
Vilue Linear meceing
Key-Value Generator Gkv.i
Cross-Attention Module G
Figure 7: The detailed architecture of the proposed key-value generators and cross-attention mod- ules.
19

---

Published as a conference paper at ICLR 2025
multi-modal feature combining methods
optimizer choices
Ablation Studies
MM-Fi HPE
MPJPE
PA-MPJPE
MPJPE
PA-MPJPE:
Methods
Add
Concate
Add
Concate
SGD
AdamW
SGD
Adam
123,4
93.9
78
60.3
239.1
93.9
165.2
60.3
D
112.5
101.8
54.4
48.4
211.9
101.8
146.3
48.4
226.4
167.1
104.2
103.2
243.9
167.1
130.4
103.2
R
144.3
127.4
70.3
69.8
248.2
127.4
148.1
69.8
W
227.7
225.6
104.8
105.3
296.5
225.6
138
105.3
I+D
96.1
86.1
53.4
48.1
166,4
86.1
116.1
48.1
I+L
115.9
93
76.7
59.7
151.2
93
105.1
59.7
R
104.3
88.8
63.1
57.3
149,4
88.8
110.9
57.3
I+ W
118.7
93
75.8
59.5
144.4
93
102.8
59.5
D
108.9
102.5
53.5
48.4
140.1
102.5
95.8
48.4
D
106.6
98
52.8
47.3
147.8
98
106.3
47.3
D+W
110.6
101.8
53.5
48.1
136.6
101.8
99.1
48.1
L+R
135.8
109.8
67.5
63.4
166,6
109.8
108.3
63.4
L+W
210.5
159.5
102.9
102.7
184.3
159.5
113.5
102.7
W
131.7
117.2
67.4
62.7
192.5
117.2
117.5
62.7
D+D+L
94.4
84.8
53.1
48.2
127
84.8
87.6
48.2
D
93.4
52.4
136.2
83.4
92.9
47.3
I+D+ W
96
85.3
53
48.1
124.5
85.3
88
48.1
I+L+R
101.4
88.4
62.5
57.2
121.5
88.4
85.5
57.2
I+L+
114.5
75.4
59.7
121
87.1
59.7
I R W
102.7
88.5
62.2
57.1
127
88.5
90.2
57.1
D+L+R
105.2
96
52.4
47.3
125.8
96
88.6
47.3
D+L+W
108.5
102
53.2
48.1
128.1
102
91
48.1
D+R+V
105.8
52.4
127.9
91.4
47.1
L+R+ W
128.1
107.4
66.9
63.1
164.9
107.4
104.5
63.1
I+ + L R
92.2
83.5
52.2
47.6
107.4
83.5
76.6
47.6
I+D L W
94.4
86
53
48.2
106.2
86
77.2
48.2
++D+R+ W
93.5
84
52.2
47.6
115.9
84
79.3
47.6
I+L+R +
100.3
88.6
62.3
57.1
113.6
88.6
80.5
57.1
D+L+R+V W
104,8
97.6
52.2
47.4
125
97.6
88.5
47.4
++D+L+R+NV
92.3
83.7
52.1
47.6
101.2
83.7
73.6
47.6
Table 11: Ablation studies on comparing multi-modal features combininng methods and optimizer choices.
Figure 8: The illustration of the training processes we experimented with: cross-attention--ased dimension reduction (left image) and the addition contrastive loss (right image).
20


Reviews:
[{"review_id": "vgpijDYWHo", "reviewer": "Reviewer_H5r3", "summary": "The paper presents X-FI, a foundation model for multi-modal human sensing. X-FI allows inference with versatile combinations of sensor modalities, and supports multiple tasks including pose estimation and activity recognition. Key to X-FI is a multi-modal feature encoding module that builds on cross-attention. X-FI was evaluated on two public datasets, demonstrating some encouraging results.", "questions": "Can you clarify the number of iterations needed for the X-fusion block?\n\nIt will be interesting to compare average pooling vs. cross-attention with learned queries.\n\nComparison to prior methods (see weaknesses) should be included.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper addresses an interesting and important problem of developing foundation models for multi-modal human sensing. \n\nThere are some interesting ideas in the model design, e.g., using cross-attention to iteratively refine the average pooled multimodal features.", "weaknesses": "I am not convinced by the claim on “modality invariant human sensing”, one of the key contributions of the paper. Conceptually, there are several approaches for modality invariant processing. For example, the model in Memmel et al., 2023 (a Transformer that takes tokens from any combination of modalities) can in theory be adapted for human sensing. Technically, this invariance is achieved by dropping out modalities during training, which is again discussed in Memmel et al., 2023 and also mentioned Chen et al., 2023.\n\nThe experiments miss important baselines. There is hardly a comparison to prior methods in the paper. At least, two sets of baselines should have been considered: (a) latest methods using a single modality (to demonstrate the benefit of multi-modal fusion); and (b) previous methods that also considers multimodal fusion such as Chen et al., 2023 (to highlight the benefit of the proposed method). The current baselines are rather straw man.  \n\nThe presentation of the technical components lacks clarity. Some of the designs are not well justified. \n* Eq 2: The use of average pooling is not justified. In fact, with different numbers of input modalities, each bin in average pooling may span (1) tokens from a single modality; or (2) tokens from more than one modality. It is not clear why average pooling works here. One possibility is learning a fixed number of queries and using cross-attention to project the input multi-modal feature, similar to the design of Percerver IO.\n* Eq 4: It is unclear how to determine the number of iterations for the X-fusion block. I also could not find the exact number used in the experiments. \n* L229: I assume n_f is the number of tokens per modality, but it was not described in the paper.\n\n================================\n\nPost-rebuttal update: In the rebuttal, the authors have addressed my previous concerns, included substantial updated results, and majorly improved the paper. While I still have some concerns over the technical innovation and the use of average pooling, I feel like the paper is now beyond the bar. Therefore, I have increased my rating.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qz6viPa9g2", "reviewer": "Reviewer_meFd", "summary": "This paper introduces a model to perform human sensing on various input modalities and output tasks. The method utilizes transformer-based attention to input a varying number of modalities and learn complementary features. The method can perform during test time with any combination of input modalities better than a fusion architecture trained with those modalities, which indicates a step towards developing a multi-sensor foundation model for human motion understanding.\n\n\nThis review follows the sections of the paper.", "questions": "I presented many questions and suggestions in the sections above. I think the main issue of the paper was a lack of clarity in describing the method and its implementation.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Introduction:\n\n1. The need for a flexible multimodal human sensing model is well-motivated. Particularly, the second paragraph outlining specific modalities and how their strengths and weaknesses can compensate for each other is strong.\n\nModality Invariant Foundation Model for Human Sensing:\n\n2. The motivation for the cross-attention multimodal fusion (X-Fusion) is good. Attempting to combat the domination of certain modalities by incorporating modality-specific information is good. How did the authors know that the attention mechanism was weighing some modalities more than others? Did they view the transformers' attention weights and notice a trend? Did previous works report that happening? It would be great to see these preliminary results somewhere.\n3. I appreciate the thought and structure that went into randomizing the modalities in \"3.3 Modality-invariant training.\" Table 3 is good. How did you determine the final probabilities? How did you know it was necessary to include more LiDAR samples than the other modalities?\n\nExperiments:\n\n4. Just to clarify, in tables 1 and 2 X-Fi was only trained once and tested on various input modalities (each unique row), but the baselines were retrained on each modality combination separately. Is this true (line 365)? If this is the case, I think this should be noted in the caption and is a strength for your method.\n5. I like Figure 4's clusterability scores. The visualization makes it hard to distinguish the difference between R and W+R, but I think the numbers emphasize the point that multimodal is better.", "weaknesses": "Introduction:\n\n1. Figure 1: I am not sure what the purpose of this figure is. What are you trying to show? X-Fi can replace a bunch of modality-specific models? But within X-Fi we have all the modality-specific encoders anyways. Are you trying to indicate that you don't need to retrain models for each combination of modalities? Nothing in this figure indicates retraining. I would suggest brainstorming a more informative diagram.\n2. Figure 2 is good, but could use some clarifications. Why are the last 3 modalities with dotted lines? Can you label the dark gray background boxes (which module is the X-Fusion part)? Also the text is quite small and hard to read. What does N stand for in the arrow loop?\n3. I think the term \"foundation model\" is starting to be used loosely quite frequently, and I would appreciate if the authors could clarify in a sentence or two. Maybe define a foundation model in this context and how their method is such before referring to their method as a foundation model. This could also be done in related works, comparing your method to any related foundation models.\n\nRelated Works:\n\n4. I think a discussion of multimodal missing modalities literature is missing. There are many works that perform multi-modal HAR in the face of missing modalities during training/testing which is a similar setup to X-Fi. Also, there is a lot of literature on transformer based fusion that could be mentioned here.\nHere are some potentially relevant works:\n- Sangmin Woo, Sumin Lee, Yeonju Park, Muhammad Adi Nugroho, and Changick Kim. Towards\nZihui Xue, Zhengqi Gao, Sucheng Ren, and Hang Zhao. The modality focusing hypothesis: Towards understanding cross modal knowledge distillation. arXiv preprint arXiv:2206.06487, 2022.\n- Quan Kong, Ziming Wu, Ziwei Deng, Martin Klinkigt, Bin Tong, and Tomokazu Murakami. Mmact: A large-scale dataset for cross modal human action understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8658–8667, 2019.\n- Qi Wang, Liang Zhan, Paul Thompson, and Jiayu Zhou. Multimodal learning with incomplete modalities by knowledge distillation. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1828–1838, 2020.\n- XB Bruce, Yan Liu, and Keith CC Chan. Multimodal fusion via teacher-student network for indoor action recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35(4), pp. 3199–3207, 2021.\n\n5. In the second sentence under Modality invariant methods. I would disagree that decision-level approaches are restricted to fixed modality combinations, and both citations provided are feature level fusion. Decision level methods may average the outputs of separate modality modals, or perform some sort of voting, so it would be easy to extend to more modalities or drop the outputs of some modalities. In fact I think that's what your baseilne1 does to my understanding.\n6. I don't think the discussion in the last paragraph citing LLM methods and a visual odometry method in robotics is very relevant.\n\nModality Invariant Foundation Model for Human Sensing:\n\n7. A lot of notation is introduced and it is hard to follow. Consider maybe labeling the arrows and boxes in figure 2 with the notation ($G_{ca}, G_{cm}, Emb_{cm}$ etc.). Also maybe the dimensions of the cubicle blocks so we can see what n_f and d_f is referring to. Is $\\theta_{Enc}$ Just the concatenation of $\\theta_{E}$ and $\\theta_{LP}$? Is positional encoding shown in Equation (1)?\n8. The key value generators and cross-attention transformers is unclear. It would be helpful to provide a diagram (at least in the appendix) for them. Also, when using the term transformer, I assume the input is one set of tokens and that's projected into keys, queries and values, but \"in cross-attention transformer,\" the input is already K,Q,V's. I would suggest referring to it as just a cross-attention module. The original Transformer in Vaswani et al (the only cited Transformer paper in 3.2.1) includes the KQV projection as well as an encoder with self-attention and a decoder with self-attention/cross attention and multiple heads. Is that what your \"Transformer\" is doing?\n9. In line 272, I don't understand why Emb_{cm} is being added back to O_i ? Again a diagram here might help.\n10. I would suggest an ablation without the K,V generators. It is just two MLP's but it happens right after the modality-specific linear projections, so maybe the modality-specific projections can capture that information.\n\nExperiments:\n\n11. In Tables 1 and 2 I would suggest labeling the reported metrics either in the column header or caption (it sounds like MSE for HPE and Accuracy for HAR?).\n12. HPE Qualitative results Figure 3. It is not apparent to me that RGB captures better leg movements and depth is better for hands. Can you circle on the images where you are observing this? Also, any thoughts on why this would be the case? Maybe even providing numbers for the errors on hand joints vs leg joints could make this result more convincing!\n13. The results could be strengthened by comparing to other multimodal methods. Are there other works that perform multimodal HPE or HAR on the MM_Fi and XRF55 datasets that you can compare to?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mfhLhLyjfZ", "reviewer": "Reviewer_Rmx6", "summary": "In this paper, the authors propose a multi-modal modality invariant foundation model to tackle the problem in the current human sensing techniques. They introduced X-Fi block to combine the feature representation from different modalities.  The authors utilized two datasets to verify the effectiveness of the proposed model on human action recognition and pose estimation. The proposed model outperformed the other methods.", "questions": "1- I suggest to have a joint loss ( L= L_I+ L_R+L_W+....L_N). In other words, to have loss for each modality and the X-Fusion output as well. It should boost the performance.  \n2-   I suggest an ablation study on the number of  stacked blocks in X-Fi  block. \n3- I was expecting an experiment which gradually add new modality each time. For example the model is trained first on RGB then WiFI is  added by fixing the parameters and finetuning the model after adding the new modality.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper presents a simple yet effective framework for multi-modal fusion from heterogenous modalities.\nTwo large datasets were adopted in the experiments.  \nThe authors conducting experiments on  different modality combinations.", "weaknesses": "1- The paper needs polishing and revision. For example in the first page (line 043) \"Alternative modalities like LiDAR, mmWave radar,\nand WiFi have be introduced to overcome these challenges\" --> it should been not be. Additionally, the authors are mixing Fig with Figure. Or capital letters within the sentence--> Line 233 \"Notably, The multi-head\" and Line 259.\n2- The authors mentioned in the contribution and abstract the proposed model surpassed the SOTA. I did not find a table of comparison against the SOTA especially the only comparison is against baseline.\n3- The authors did not provide the experimental setting. For example, MM-Fi [1] has three settings and it is not clear which setting is adopted. I recommend to adopt the three settings.\n\n\n[1]Yang, Jianfei, He Huang, Yunjiao Zhou, Xinyan Chen, Yuecong Xu, Shenghai Yuan, Han Zou, Chris Xiaoxuan Lu, and Lihua Xie. \"Mm-fi: Multi-modal non-intrusive 4d human dataset for versatile wireless sensing.\" Advances in Neural Information Processing Systems 36 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DPVjiCdnco", "reviewer": "Reviewer_u1Rk", "summary": "This paper introduces a modality-invariant foundation model that allows for the integration of various sensor modalities for human sensing. The proposed X-Fi model employs a transformer architecture with a \"X-fusion\" mechanism to preserve modality-specific features in the multimodal integration process. The proposed model is designed to support any combination of sensor modalities with flexibility of the input dimension without requiring additional retraining. They evaluated the proposed method on two benchmark datasets, MM-Fi and XRF55, covering six distinct modalities, with two downstream tasks: human pose estimation and human activity recognition. The experimental results showed that the proposed X-Fi model improved the performance of HPE and HAR.", "questions": "1. Can the authors provide additional details on the distinguished contribution points of the X-fusion block compared to existing cross-attention fusion approaches? \n2. Would the authors consider conducting further experiments with other cross-modal foundational models (e.g., Meta-Transformer, ImageBind) as baselines to provide a more comprehensive comparison?\n3. Is it possible to include additional benchmark datasets with IMU data for HAR task?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ This paper addresses the important problem of modality-invariant human sensing by designing a foundational model with flexiblity without retraining. \n+ The paper is clearly written and well-structured, effectively communicating the motivation, methodology, and results. Concepts such as modality-specific feature extraction, cross-attention fusion, and the role of the X-fusion block are clearly explained, The transformer-based architecture handles multimodal input data and X-fusion block is used for cross-modal feature fusion by using cross-attention.", "weaknesses": "- The paper lacks a thorough investigation of related cross-modal / modality-invariant foundational models, such as CLIP [1], PaLM-E [2], Meta-Transformer [3], ImageBind [4], IMU2CLIP [5], and zero-shot learning for HAR [6]. The authors need to investigate recent cross-modal foundational models.\n- While the paper proposes the X-fusion block for multimodal fusion, the core approach of using transformers and cross-attention for feature fusion is not new. Similar attention-based fusion methods are widely used in the literature for feature-level integration. The paper could strengthen the novel contributions and clarify the novel aspects of X-fusion compared to existing methods.\n- The evaluation experiments focus on comparing the proposed method with basic feature and decision-level fusion approaches. The proposed foundational model may be compared with other cross-modal foundational models.\n- The evaluation was conducted on MM-Fi and XRF55 datasets, which primarily include specific modalities like cameras, WiFi, LiDAR. However, human activity recognition often involves wearable sensors like IMUs. It is better to add one or two more benchmark datasets with IMU sensors.\n\n\n[1] Radford, Alec, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry et al. \"Learning transferable visual models from natural language supervision.\" In International conference on machine learning, pp. 8748-8763. PMLR, 2021.\n[2] Driess, Danny, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid et al. \"PaLM-E: An Embodied Multimodal Language Model.\" In International Conference on Machine Learning, pp. 8469-8488. PMLR, 2023.\n[3] Zhang, Yiyuan, Kaixiong Gong, Kaipeng Zhang, Hongsheng Li, Yu Qiao, Wanli Ouyang, and Xiangyu Yue. \"Meta-transformer: A unified framework for multimodal learning.\" arXiv preprint arXiv:2307.10802 (2023).\n[4] Girdhar, Rohit, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. \"Imagebind: One embedding space to bind them all.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15180-15190. 2023.\n[5] Moon, Seungwhan, Andrea Madotto, Zhaojiang Lin, Aparajita Saraf, Amy Bearman, and Babak Damavandi. \"IMU2CLIP: Language-grounded Motion Sensor Translation with Multimodal Contrastive Learning.\" In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 13246-13253. 2023.\n[6] Tong, Catherine, Jinchen Ge, and Nicholas D. Lane. \"Zero-shot learning for imu-based activity recognition using video embeddings.\" Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5, no. 4 (2021): 1-23.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9j7dxU0xA2", "reviewer": "Reviewer_rbmV", "summary": "This paper introduces a multi-modal foundation model that integrates six distinct modalities to enhance human sensing capabilities. The proposed model demonstrates the scalability of various modality combinations and incorporates an X-Fusion block to fuse multi-modal features while preserving modality-specific information. The authors conduct extensive experiments on two multi-modal datasets to evaluate the model's performance in human pose estimation and human action recognition tasks.", "questions": "Please see the weakness part.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The foundation model of different human sensing modalities is a promising approach, and the author made a meaningful attempt. The training schedule of this model is interesting", "weaknesses": "1. The section of problem definition is missing. This paper addresses two different tasks, which generally require different input formats of modalities. For instance, HAR requires video clips, but HPE could be video clips or frames. The clear details about the inputs and outputs are crucial for reader to understand.\n2. There are some undefined notations, such as  in line 172 and  in Eq. 4.\n3. The motivation regarding the module in subsection 3.2.2, which is to pay equal attention to each modality, is unconvincing. Different modalities offer varying degrees of human sensing information, and only valuable modality or information are important. The model paying more attention to dominant modalities is more intuitive. Equal attention could lead to the inclusion of irrelevant information and the potential omission of key details.\n4. The cross-attention module is intended to accentuate modality-specific information. Nevertheless, the absence of ablation studies on this module precludes a conclusive evaluation of its efficacy, and there is no empirical evidence to substantiate the modality-specific features extracted into the K-V pairs. Some additional constraints are necessary. For instance, SimMMDG [1] leverages distance loss between modality-specific and modality-shared features and the orthogonal constraints is adopted in [2].\n5. No ablation study on the components of proposed method.\n6. The experimental results of Lidar in MM-Fi reveal inferior performance relative to mmWave in Tab. 1. This is in contrast to the results in MM-Fi under same experimental setup and backbones.  And it is also not intuitive as Lidar data, characterized by its higher point density, offers a more precise and detailed representation of the human compared to mmWave.\n7. To ensure a fair comparison in Tab. 4, the number of parameters in the transformer and X-Fusion block should be equalized.\n\n8.You employ multiple layers of cross-modal transformers to obtain cross-modal embeddings. How does the number of these iterative layers，N, impact the experimental results? Additionally, how might varying the number of transformer layers for different modality counts influence the outcomes?\n\n9. The proposed method in the paper enforces a balance across all modalities. Could this forced balancing lead to a negative impact in tasks that primarily rely on a specific modality, especially if some modalities contain less informative data?\n\n10. Performing computations across all modalities in each iteration could result in high computational complexity, particularly in multi-modal processing and long-sequence modeling. This increased demand could pose challenges for real-time applications, making practical deployment of the method more difficult.\n\n\n[1]Dong H, Nejjar I, Sun H, et al. SimMMDG: A simple and effective framework for multi-modal domain generalization[J]. Advances in Neural Information Processing Systems, 2023, 36: 78674-78695.\n[2]Jiang Q, Chen C, Zhao H, et al. Understanding and constructing latent modality structures in multi-modal representation learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 7661-7671.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)