HoloLLM: Multisensory Foundation Model for Language-Grounded Human Sensing and Reasoning
Authors: Chuhao Zhou
Jianfei Yang
Venue: NeurIPS 2025 poster ()
Keywords: Multimodal Large Language Model
Human Sensing and Reasoning
IoT Sensors
Primary area: deep_learning
TL;DR: Propose HoloLLM, the first work to align MLLM with rare sensing modalities to achieve seamless human perception and reasoning with language grounded.

Abstract: Embodied agents operating in smart homes must understand human behavior through diverse sensory inputs and communicate via natural language. While Vision-Language Models (VLMs) have enabled impressive language-grounded perception, their reliance on visual data limits robustness in real-world scenarios with occlusions, poor lighting, or privacy constraints. In this paper, we introduce HoloLLM, a Multimodal Large Language Model (MLLM) that integrates uncommon but powerful sensing modalities, such as LiDAR, infrared, mmWave radar, and WiFi, to enable seamless human perception and reasoning across heterogeneous environments. We address two key challenges: (1) the scarcity of aligned modality-text data for rare sensors, and (2) the heterogeneity of their physical signal representations. To overcome these, we design a Universal Modality-Injection Projector (UMIP) that enhances pre-aligned modality embeddings with fine-grained, text-aligned features from tailored encoders via coarse-to-fine cross-attention without introducing significant alignment overhead. We further introduce a human-VLM collaborative data curation pipeline to generate paired textual annotations for sensing datasets. Extensive experiments on two newly constructed benchmarks show that HoloLLM significantly outperforms existing MLLMs, improving language-grounded human sensing accuracy by up to 30\%. This work establishes a new foundation for real-world, language-informed multisensory embodied intelligence.

HoloLLM: Multisensory Foundation Model for Language-Grounded Human Sensing and Reasoning
Chuhao Zhou', Jianfei Yang
MARS Lab, Nanyang Technological University
(chuhao0028e, jianfei. . edu.
Project Page: http://mww....iui.oooooo////oooa
Code: https//giiabbboomm//tttnA/////bL
Abstract
Embodied agents operating in smart homes must understand human behavior through diverse sensory inputs and communicate via natural language. While Vision-Language Models (VLMs) have enabled impressive language-grounded perception, their reliance on visual data limits robustness in real-world scenarios with occlusions, poor lighting, or privacy constraints. this paper, we introduce HoloLLM, Multimodal Large Language Model (MLLM) that integrates uncom- mon but powerful sensing modalities, such LiDAR, infrared, mm Wave radar, and WiFi, to enable seamless human perception and reasoning across heterogeneous environments. We address two key challenges: (1) the scarcity of aligned modality- text data for rare sensors, and (2) the heterogeneity of their physical signal repre- sentations. To overcome these, we design a Universal Moddality-Injectinn Projector (UMIP) that enhances pre-aligned modality embeddings with fine-grained, text- aligned features from tailored encoders via coarse-to-fin cross-attention without introducing significant alignment overhead. We further introduce a human- VLM collaborative data curation pipeline to generate paired textual annotations for sens- ing datasets. Extensive experiments on two newly constructed benchmarks show that HoloLLLM significantly outperforms existing MILMMs, improving language- grounded human sensing accuracy by up 30%. This work establishes foundation for real-world, language-informed multisensory embodied intelligence.
1 Introduction
Embodied agents in smart homes, e.g., household robots and intelligent appliances, have garnered increasing attention in recent years [1-4]. To interact effectively with humans and execute real-world tasks, agents must understand human behavior and be capable of engaging in natural language communication. This necessitates the development of models that scamlessly integrate rich human perception with advanced language understanding and generation capabilities.
Vision-Languall Models (VLMs) [5-8] have emerged as promising tools for enabling language- conditioned perception and reasoning. However, the visual modality alone struggles to operate in the real world, e.g., low-light environments, occlusions, and privacy-sensitive scenarios. In contrast, humans naturally rely on multiple sensory modalities, such as vision, audition, and olfaction, to perceive and adapt to diverse environments. Inspired by this biological principle, embodied agents can benefit from incorporating alternative sensing modalities, including LiDAR, infrared cameras, mm mmWave radar, and WiFi signals, to achieve more robust and comprehensive perception. Each of these modalities brings distinct advantages: LiDAR enables high nigh-precision 3D reconstruction [9], infrared cameras support perception in darkness [10], and mm nawe radar and WiFi are resilient to
CCorresooniigg Author yangg ntu.edu.sg)
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Secwrity: Detect the intruder
IoooLLMM: Action Recognition
Night Time
Hoaltheare: Detect fallen person
Predict: y (running)
HoloLLM: Action O4
Challenges
What is Jack deinn?
Low light. Wide rang
Modalites
Jack 1s Tallen down.
Rader, Infrrred, WiFi
Chailenges
RGB, Depth, LIDAR
Occlusion, Privacy
Housewrr:: Carry heavy objects
Modalines
HoloLLM, Action Caption
Rodar, WiFi
RGB, Depti, LiDAR, Infrared
strogeling carry
objects, need
his help.
Chalfenger
RGB
Depih
LiDAR
Vine grained Action Reasoning
Modalites
ROBR, Depdiy LIDAR
Time
Radar
Infrared
WiFi
Radar, Infrared, WiFi
Figure I: HoloLLM achieves seamless and language -orunnde-rdddddd human perception and reasoning with challenges, e.g. illumination and privacy,
complementary sensing modalities. It overcomes with superior performance on human action recognition, question answering (QA), and captioning tasks, which enables embodied agents to work intelligently across diverse scenarios.
visual occlusions and lighting variations [11]. As illustrated in Fig. 1, vision alone fails to detect a fallen individual behind an obstruction, while radar- and WiFi-based modalities remain effective. Hence, we believe Multimodal Large Language Models (MLLMs) that integrate diverse sensor inputs can provide excellent adaptability and reliability in complex, real-world environments.
However, integrating sensing modalities into an MLLM is non-trivial due to two key challenges. First, we must align sensing modalities with text using limited training data. For common modalities such as RGB or depth images, millions of web-sourced 'modality-text data pairs are available [12-14], which enable large-scale pre-training of multimodal projectors effectively align common modalities with text. In contrast, other sensing modalities (c.g., mm Wave and WiFi signals) are not available online, with only few thousand samples collected in labs [11, 10]. The data scarcity complicates the alignment between sensing modalities and text. Secondly, modality- feature encoder, but it is challenging to learn heterogeneous characteristics of these rare sensing modalities. To model the physical world, different sensors are designed to leverage distinct physical designs ge.g., wavelengths and frequencies) at multiple granularities, of which the sensing data show extreme heterogeneity and significantly differ from common modalities, leading to huge challenges for existing transformer-basee constitute the question investigated in this work: can enable MLLM to learn data-scarce and heterogeneous sensing modalities for language-ground
alignment requires a robust
encoders to learn robust representations [10, 15]. These two challenges
human perception and reasoning?
To this end, our key idea comes from two aspects. Due to data scarcity, directly aligning sensing modalities with text through large-scale pre-training is infeasible. To address it, we propose to generate initial embeddings for each modality pre-aligned with text, without additional training. Then, only minor data and fine-tuning are sufficient to achieve appropriate alignment. Regarding the heterogeneity of sensing modalities, previous studies [16, 17] show that tailored encoders are more effective than normal transformers to extract modality-specific features. However, directly aligning raw modality-speciff features with text from scratch demands massive training data and incurs additional alignment overhead. As a result, we take raw features as references and integrate text-aligned features into multimodal embeddings via an iterative process: querying and fusing text-aligned features into the multimodal embeddings through cross-attention, and projecting the embeddings into the LLM semantic space to form the enhanced queries for the next iteration.
In this paper, we propose T HoloLLM, an MILM for seamless human perception and reasoning across common and sensing modalities. Specifically, we adopt a CLIP vision encoder aligned with the text modality to generate pre-aligned initial embeddings for each modality. Tailored encoders are then designed to fully explore fine-grained, modality- discriminative features. To avoid additional alignment overhead, we propose the Universal modality features integration. UMIP takes the initial embeddings coarse queries to adaptively
Projector (UMIP) for progressive
2

---

identify and integrate fine tne-grained, text-aligned modality features via coarse to--intoeeeeeeeee cross-attention Furthermore, we introduce human- VLM collaborative data curation pipeline generate textual annotations for existing multimodal human sensing datasets: MM-Fi [11] and XRF55 [10]. A comprehensive evaluation of state-of-the-ar art MI.LMs is conducted, establishing the first benchmark for multimodal human perception and reasoning grounded in sensing modalities.
In summary, our contributions are threefold:
We propose HoloLLM, the first work align MLLM with rare sensing modalities to achieve seamless human perception and reasoning with language grounded.
We propose Universal to deal with the data-scarce feature learning and modality-tex alignment, respectively.
Projector (UMIP) and moddllity-speciffe encoders
To evaluate HoloLLM, we design human- VLM collaborative data curation pipeline to construct the text-paired multimodal dataset. Then, the first multisensory benchmark for human sensing established with different settings and baselines. The Holo LL shows superior performance, improving existing MLLM by around 30% on some QA tasks.
2 Related Work
Multimodal Human Sensing. Human sensing aims to perceive, analyze, and understand human actions, which is essential human-agent interaction. Beginning with human sensing based on RGB and depth frames [18], various other sensing modalities, such LiDAR [19], mmWave [20, 21], WiFi-CSI [17, 15], and RFID [10], have been progressively introduced to address limitations, including lighting variations, occlusions, and privacy concerns. To achieve more comprehensive human perception, methods that explore complementary information across modalities become dominant [22, 23, 11, 16]. With the development of LLM, many works seek to leverage the high-level semantics and strong generalization capability of language to perform zero-shot human sensing tasks [24], which are extended multimodal inputs for more comprehensive recognition [25]. However, existing models cannot reason and generate responses based on perceived information, limiting their capacity to engage in language-bass based interaction with humans.
Multimodal--eex Alignment. Extending LLMs to other modalities to form MLLMs enables them to follow human instructions based multimodal inputs such as images [5, 8]. vidcos [26], audio [27]. and point clouds [28]. Aligning multimodal inputs with text is the key challenge for MLLMs. Existing methods primarily rely building multimodal projectors. Typically, Linear / MLP projectors [5, 26] directly project the multimodal inputs into the LLM text space. Despite the simplicity, the number of multimodal tokens will increase significantly when high-resolution inputs are presented. To alleviate this issue, Resampler / Q-Former projectors [8, 27] utilize a fixed number of learnable queries to align the most task-relevant multimodal information with the text. Recently, hybrid projectors have been developed [6, 7J. which serve as trade-offs between MLP and Q-Former projectors, to fully preserve all information within multimodal inputs while minimizing the number of tokens generated. Nevertheless, these projectors commonly need large-scale 'modality-txx'' data pairs for pre-training, which are unavailable for sensing modalities.
3 Method
In this part, we first formulate our problem (Sec 3.1), then delve into the details of the UMIP (Sec 3.2). Followed by an introduction of the two-stage training strategy for HoloLIM (Sec 3.3). Finally, we present the data curation pipeline for our multisensory benchmarking (Sec 3.4).
3.1 Problem Formulation
The objective of MLLMs is to obtain multimodal tokens 7", which are not only aligned with text tokens tent but also preserve modality-speciif features within the inputs T This work aims to learn an MLLM with data-scarce and heterogeneous sensing modalities, which presents a greater challenge, large-scale pre-training of multimodal projectors infeasible.
An MILLM typically consists of three components: (1) A multimodal Encoder F!(-) that converts the inputs X from modality into sequence of multimodal embeddings " (2) A projector P(-)
3

---

Universal Modalin- Modalin-Injecttion Projectur (UMIPH
CLS Head
Xaiáeo
RGB
YOUF
OL
Projector
Label 9
Recoguittin
Tokenizer
Universal Encoder
FFN
Large Lauguage Modd
Action Od
Xtader
Radar Tokenizer
down sample
Cross Attention
Flatten
What the action of the human?? Relax.
MLP
Self Attention
x
WiFi
Flatten
Action Caption
Tokenizer
Q
Q: Please deseribe human action.
Modality-Speeiffe
A: In lab, person is tuning his wrists ankles......
ResNet
PoiniNet
MetaFi
Tailored Encoders
Trainable
Froozen
Figure 2: Architecture of HoloLLM. Given multimodal inputs XT, HoloLLM utilizes modality- specific tokenizers and universal encoder extract pre-aligned initial embeddings Y CLIP. Mean- while, pre-trained tailored encoders are applied to explore modality features Y The UMIP then transforms Y CLIP and Y into coarse queries Q" and fine-grained keys and values K" /V By iteratively enhancing the queries via coarse-to-fing cross-attention and projecting them to the LLM text space, the aligned multimodal tokens Zn fully enriched by modality features can be achieved.
to align Y with the LLM text space, forming aligned multimodal tokens ZTd = P(Y (3) An LLM LLM(;.), which takes 7 and the text tokens Aezt corresponding to human instructions as inputs, to output the responses A
{a,) i=1 = LLM Zlext) in an auto-regressive approach: II p(a; T Zlext A Ai), S is the length of responses generated by LLM.
p(A)
3.2 Universal Modality-Injectioo Projector
As shown in Fig. 2, we propose efficient projector, named the Universal Modality-Injechien Projector (UMIP). To overcome the data scarcity, we attempt to generate initial embeddings pre- aligned with the text for each modality, without the need for extra training. Specifically, we take the CLIP vision encoder [29] as a unified multimodal encoder ECLIP(.) to obtain initial embeddings Y
CLIP for modality m:
Yn
CLIP ECLIP (X")) E ggmlxd...
(1)
where nu and dm denote the number and dimension of the embeddings. Benefiting from extensive image-text contrastive pre-training, the CLIP vision encoder achieves superior alignment with text and offers transferability to other modalities [30]. Consequently, the YLLP can be considered inherently pre-aligned with text, and minor data and fine-tuning are sufficient align them with text.
However, our experiments (Tab. and Tab. 2) reveal that the initial embeddings CLIP lack sufficient discriminability. Previous studies [16. 17, 10] show that dedicated convolutional encoders outperform transformer counterparts in capturing heterogeneous spatial features from sensing modalities grounded in radio frequency, such WiFi signals. To this end, we design convolutional tailored encoder ETp (.) for each modality to capture the heterogeneous modality features Y
Y T
MLP"
ETT X
E Bl X.lmm Xdm.
(2)
where MLP'"(.) is a MLP projector to align the feature dimension of (.) to that of CLIP dms hm and wm, the height and width of the feature map.
The significant heterogeneity of Y makes directly aligning them with the text inefficient, increasing the demand for training data. Therefore, we only take Y as references to provide fine-grained modality-specifi features and convert them into candidate keys and values: K V E I Ap x dmm Meanwhile, the pre-aligned initial embeddings YCLIP are downsampled to form the queries:
Q" nT AvgPooll YoLIP) Y E xd...
(3)

---

where AvgPool is ID adaptive average pooling and n'mn < nm limits the number of queries to fixed value. The queries QTT only contain the coarse- geanseegaaindd modality features, which are enhanced via:
(() SelfAtt ((-1))
(() = CrossAtt Q KT V
(1) (F)((Q))) 1= 1=1....,I,
where 2,0) = Q". SelfAtt(-), CrossAtt(:; -) and (FN()) are self-attention, cross-attention, and feedforward layers in each block of UMIP, respectively, and L is the number of blocks.
Our UMIP follows iterative process to produce discriminative multimodal tokens that ade- quately aligned with the text. In each block UMIP, the coarse-to-fin cross-attention adaptively identifies the text-aligned modality features from V and injects them into the queries Q" for enhancement. The updated queries are projected into the text space via the feedforward layer, which serves enhanced queries for the next block. Finally, the multimodal tokens from the last block n MLP((Q) can be aligned with the LLM text space while sufficiently enriched by modality- specific features. Here, MI)((-) maps the dimension of CLIP (1024) that of the LLM (4096).
LLM
LLM
LLM
RGB Projector
Rocarr Projector
WiFi Projector
Universal Projector
Query
Modulit--njsstton
Universal
Projector
Vay/Value
KayiValue
Key/Value
RGB Enguder
Univenal Encoder
Universal Encuder
RGB
Radar Engoder
WiFi
Encoder
Encuder
Encuder
RGB
Rodar
WiFi
RGB
Radar
WIFI
Radar
WiFi
Medality-Seccffie Projector
Universal Projector
Universal Medality-nnjaetion Projector
Figure 3: Comparison between UMIP and other projectors: (a) Modality-Specific Projector [28, 27,
31, 32], (b) Universal Projector [30], and (c) Universal Modality-njeection Projector (Ours).
Discussion prior art. We compare UMIP with state-of-the-ar multimodal projectors in Fig. 3. Specifically, most existing works [28, 27, 31, 32] adopt modeeity-specific encoders and projectors (Fig. (a)), which commonly requires substantial 'modality-text' data pairs for pre-training. As shown in Fig. 3 (b), OneLLM [30] attempts to handle various modalities via unified framework that consists of a universal encoder and projector. However, without dedicated design for capturing heterogeneous spatial features, the universal encoder struggles to obtain sufficiently discriminative multimodal tokens. Different from existing works, UMIP only utilizes the universal encoder to generate initial embeddings for each modality (Fig. 3 (c)). These embeddings are then progressively enhanced by fine-graine fine-grained, text-aligned features from tailored encoders.
3.3 Training Strategy and Objectives
To effectively train HoloL.MM, we propose a two-stage training strategy: (1) pre-training tailored encoders extract modality-speciff features, and (2) fine-tuning the HoloL to learn discriminative multimodal tokens that are appropriately aligned with the text space.
During the first stage, task-specific objective, i.e., the classification loss for the Human Action Recognition (HAR) task, utilized to pre-train tailored encoders:
(5) where ET is the tailored encoder for modality m, X is the i-th sample in the dataset, and is the corresponding action label. C....)) and Classifier()) refer to the cross-entropy loss and the classifier to predict the action label, respectively.
L C (EB((asssiierr(E) (X." )), ()))
5

---

After vre-training all tailored encoders are frozen. The modality tokenizers and the UMIP are then fine-tuned by the combination of task siseiipccfiic and next-token prediction objectives:
L2 = (C(Caaasifierr (LLM (Z," ) c.) + Lnezt:
(6)
where Z and Zient denote the multimodal tokens and the instruction text tokens for the i-th sample, Linar is the cross-entropy loss of next-token prediction for both Action QA and Action Caption tasks.
Senalo te
hender or foor
Bigia aud
Corarddd Truis
Reaile
Duerko::
Du Contrra
Uncher Coplonnndd
Archen prettdogg relorigg Nurbordd
Aecu aike...
Auciptioe of andic
eftamana
Stakk
Copae:: loyn salira
th arice....
oener peoally eoers prrtude
(a) Action QA generation
(b) Action Caption generation
Figure 4: Data curation pipeline for (a) Action question answering (QA) and (b) Action Caption.
3.4 Data Curation for Multisensory Language-grounder Benchmarking
To perform real-world tasks in diverse smart home scenarios, embodied agents must understand human behavior from multisensory inputs and engage in inguuage-grondder communication. Therefore, we establish the first multisensory benchmark for evaluating human perception and reasoning capabilities in HoloLLM, encompassing action recognition, question answering (QA), and captioning tasks. However, textual descriptions for sensing modalities are not available in public datasets and online sources. Inspired by other modalities such as point clouds [33] and IMU [34], we select two multimodal human sensing datasets, MM-Fi [11] and XRF55 [10], and refer to the vision modality to
generate action QA and caption data. As shown in Fig. 4, we take the MM-Fi dataset as an example
to illustrate our data curation pipeline; more details can be found in Appendix A.
Action QA generation. In this work, Action QA is conditioned on options. To ensure precision and diversity, questtions are first annotated by human experts and rewritten by GPT-4o [35], resulting in a list of 15 questions. For sample from each modality, question ( ("Question') )) is randomly sampled from the list. Moreover, all action categories of the MM-Fi dataset, along with the action label of the data sample, are appended to the question to generate the "Action List" (options) and the 'Answer'. Formally, an Action QA sample can be formulated as: (Question, Action List, Answer).
Action Caption generation. We propose human- VLM collaborative pipeline to generate captions for sensing modalities. Specifically, a fixed question (' ("Question': Pleesse give detailed descriptions of human's action.) is designed for all caption tasks. Then, we uniformly select a small set of samples across various environments, subjects, and action categories to form the 'In-context Caption Samples'. For other data samples, L.LaVA captions (' ('Caption") via an in-context learning manner. Finally, an Action Caption sample can be formulated as: (Question, Caption), which is shared among all modalities of a data sample.
[26] is adopted to automatically generate the
4 Experiment
4.1 Experimental Settings
Datasets.
We utilize two multimodal human-sensing datasets MM-Fi [11] and XRF55 [10] with generated textual descriptions. Specifically, MM-Fi consists of 5 modalities: Video (V), Depth images (D), LiDAR (L), mm Wave Radar (M), and WiFi-CSI (W). Besides, XRF55 also contains 5
modalities: Video (V), Depth images (D), Infrared images (I), RFID signals (R), and WiFi-CSI (W).
Benchmarks. To comprehensively evaluate various MLLMS across diverse scenarios, we design three experimental settings: (1) Random Split (Random), (2) Cross-Subject Split (CrossSub), and (3)
Cross-Environment Split (CrossEnv). Specifically, 'Random' involves a random split of all samples

---

with ratio of 3:1, and 'CrossSub' / "CrossEnv' selects samples from nonoverlapping human subjects environments for training and testing. For quantitative evaluation, we use the accuracy for Action Recognition and Action QA, and the METEOR [36] metric for Action Caption.
Implementation Details. Following OneLLM [30], we take CLIP VIT Large pre-trained on LAION [12] to provide initial multimodal embeddings, and the LLaMA2-7B [37] as our LLM. For tailored encoders, we take Resnet18 [38] for Vision, Depth and Infrared, PointNet [39] for LiDAR and mm Wave, ID Temporal Resnet18 8 [10] for RFID, and MetaFi [17] for WiFi modalities. The
UMIP contains L = 8 blocks, with 64 query tokens for Vision, Depth, mm Wave, and Infrared, 256 query tokens for LiDAR and WiFi (XRF55), and 16 query tokens for RFID and WiFi (MM-Fi).
Please refer to Appendix for more details on our experimental settings.
Sottings
Models
Sources
Types
Human Action QA (Accuracy)
Action Caption (METEOR)
Avg
M
Avg
Tokenpacker
arXiv 24
Proj
2.
2.2
2.5
2.6
1.9
2.4
8.9
8.9
8.9
8.9
K.9
K.9
Honeybee
CVPR'24
2.3
1.7
Random
OneLLM
CVPR 24
Proj
3.6
3.5
4.7
3.5
4.2
3.9
15.5
15.3
13.3
15.99
16.6
15.3
ImageBind
CVPR'23
HoloLLM
Proj
99.8
99.7
95.8
84.2
52.8
86.5
30.8
31.1
29.6
27.4
23.0
28.4
Tokenpacker
arXiv'24 '24
3.0
3.2
6.6
CrossSub
Honeyhee
CVPR 24
Proj
1.8
1.8
1.9
2.2
2.0
1.9
10.3
10.3
10.3
10.3
10.2
10.3
OneLLM
CVPR'24
3.8
2.6
10.3
ImageBind
CVPR'23
Eac
76.9
43.3
45.5
6.8
7.7
36.0
25.8
21.0
20.9
15.3
16.5
19.9
HoloLLM
98.0
98.9
66.5
71.9
30.6
30.5
29.5
24.9
26.4
Tokenpacker
arXiv'24 24
Proj
5.0
5.0
4.7
4.2
4.3
4,6
3.9
3.9
3.8
3.8
3.7
3.8
CrossFax
Honeyhee
OneLLM
CVPR 24
CVPR124
Proj
Proj
4.2
2.0
8.0
1.1
7.8
1.9
4.0
5.0
1.7
15.5
13.1
2.2
5,1
10.4
9.3
ImageBind
CVPR'23
7.6
16.7
17.5
HoloLLM
Proj
79.5
91.6
61.4
41.4
8.2
56.4
25.7
27.5
24.5
19.6
15.9
22.6
Table 1: Evaluation of Human Action QA and Caption tasks on MM-Fi [11] across three settings. The Accuracy (%) and METEOR (%) are adopted for Action QA and Caption, respectively
Settings
Madels
Sources
Types
Human Action (Acouracy)
Action Caption (METEOR)
V
D
R
Avg
D
[
W
Avg
Tokenpacker
arXiv'24
1.2
1.1
7.7
Random
Honeyhee
CVPR 24
Proj
1.4
1.6
1.5
1.6
1.5
1.5
10.0
10.1
10.3
10.1
10.0
10.1
OneLLM
CVPR'24
2.0
1.9
1.8
14.5
ImageBind
CVPR 23
Enc
62.2
22.2
79.0
5.3
10.0
35.8
19.3
13.0
24.3
12.3
12.7
16.3
Ho
94.5
92.3
92.6
27.1
11.2
63.5
34.2
34.8
34.7
15.5
14.0
26.6
Tekenpacker
arXiv'24 24
Proj
1.2
1.1
1.4
1.2
1.2
1.2
9.2
9.1
9.1
9.2
9.3
9.2
CrossSub
Honeyhee
CVPR'24
9.3
OneLLM
CVPR
Proj
1.9
1.5
2.0
2.3
2.1
2.0
13.4
13.5
14.5
9.2
13.1
12.7
ImageBind
CVPR'23
13.3
11.1
20.4
3.8
10.7
15.8
18.3
12.7
14,5
15.2
HoloLLM
Proj
44.3
42.1
38.3
3.4
3.6
26.3
22.3
23.1
22.8
11.8
13.7
18.7
Tokenpacker
arXiv'24
7.2
CrossEns
Honeybee
OneLLM
CVPR 24
CVPR
Proj
1.4
1.5
1.4
1.6
1.7
1.2
1.1
1.3
11.2
11.2
11.0
11.5
11.0
11.3
11.1
ImageBind
CVPR 23
Eac
4.7
4.9
16.9
2.8
2.6
6.4
13.1
14.3
16.9
12.7
12.0
13.8
HoloLLM
25.9
12.4
17.1
Table 2: Evaluation of Human Action QA and Caption tasks XRF55 [10] across three settings. The Accuracy (%) and METEOR (%) are adopted for Action QA and Caption, respectively
Conadab
CanoEar
Depla
Deyci
Depti
Daphi
Deyth
Dayti
L00
Piice LEDAR
Viace LEDAM
Viico REID
Visko XFID
WiFi
WiFs
WiF5
Ava BobiLMM
WiFii
WiFi
WiFi
Ang steloL.M
Tokongueker
DIvELLA MMLE
Tekoeppocke
OheLLMM
Irageliind
Figure 5: Evaluation of Human Action Recognition on MM-Fi [17] and XRF55 [10] across three benchmarks in terms of Accuracy (Better zoom in).

---

4.2 Main Results
We compare HoloLLM with ofte-of-the-ar MILMS across three tasks: Action QA, Action Caption, and Action Recognition. Specifically, we divided existing methods into Encoder based (Image- Bind [31]) and Projector-based (Honeybee [6]. Tokenpacker [7]. OneLLM [30]) methods. The Encoder-based methods aim to align multimodal embeddings from various encoders via contrastive learning, while the Projector-basee methods focus on designing effective projectors for alignment. For fair comparison, all methods are fine-tuned on MM-Fi and XRF55 datasets our MoloLLM.
Action QA and Action Caption. The results on MM-Fi and XRF55 datasets are summarized in Tab. 1 and Tab. 2. For all three benchmarks. HoloLLM outperforms other MLLMs by a large margin on almost all modalities (indicated in bold). Specifically. Tokenpacker and Honeybee only use a modality-shared projector, which cannot effectively capture modality-specifie features aligned with text. By introducing learnable queries for each modality to the projector, OneLLM and ImageBind can better align multimodal embeddings with the text space while exploring the modality features. Moreover, ImageBind employs dedicated encoders for depth and infrared modalities, along with a stronger vision encoder (CLIP VIT-H), enabling more effective capture of modality-specinf features and improved performance. Instead of using simple learnable queries, HoloLLM designs tailored encoders to adequately capture fine-grained modality-speciffe features, which are adaptively injected into the aligned multimodal tokens via UMIP. The results demonstrate that multimodal alignment and discriminability are equally critical for human perception and reasoning based on sensing modalities.
Action Recognition. As shown in Fig. 5, HoloLLM outperforms other MLLMs on Action Recog- nition compared across most modalities. For some sensing modalities, such WiFi and RFID, performances on par with other MLLMs are observed under "CrossSub" or "CrossEnv" settings. Some sensing modalities are highly sensitive to different subjects or environments, making it chal- lenging to achieve cross- cross-subject or cross- gross-environment generalization. More efforts should be made toward building large-scale sensing datasets to further promote their generalization capability.
Action Recognition
Action QA
Action Cuption
MMFI
D
M
L
Avg
V
Avg
D
M
L
W
Avg
Baseline
9.7
9.5
13.6
6.5
7.4
10.3
5.9
3.3
14.4
4.8
4.7
6.2
15.3
15.1
17.4
13.9
16.0
15.9
TailorEnooder
83.1
91.8
58.2
50.4
8.8
58.5
71.8
73.4
48.4
28.0
11.3
46.6
25.9
25.4
24.0
20.3
13.8
21.9
UMIP
80.6
93.2
61.0
53.0
59.5
79.5
91.6
27.5
24.5
19.6
15.9
XRFSS
v
D
1
R
W
Avg
V
D
R
W
Avg
V
D
1
R
W
Avg
Baseline
4.4
3.8
18.2
2.6
3.4
6.5
3.7
2.3
2.4
322
3.8
3.3
12.0
1211
13,0
11.1
13.3
12.3
ThilkrFnooder
27.5
9.1
31.2
1.7
14.4
23.5
23.0
20.2
11.7
20.1
10.4
12.3
14.9
+UMIP
28.9
12/4
28.3
1.7
3.7
15.0
25.9
8.6
22.1
3.6
2.7
12.4
19.8
14.7
17.1
10.8
13.7
15.2
Table 3: Ablations results. We conduct experiments on both MM-Fi and XRF55 under "Cross-
Environment" setting to show the contribution of the key components.
4.3 Ablation Results
We conduct the ablation study on both MM-Fi and XRF55 datasets under the "CrossEnv" setting to show the effectiveness and generalization capability of key components. Specifically, 'Baseline' only adopts the CLIP ViT-L. to extract multimodal embeddings and utilizes Q-former [8] with modality-specifff learnable queries as the projector. Following OneLLM, the size of the learnable queries is 10x14
1024 The results are summarized in Tab. 3.
Ablation Study on Tailored Encoders. To evaluate the effectiveness of tailored encoders, we replace the CLIP VIT-L. in the 'Baseline" with them and use the same Q-former for multimodal alignment (*#TailoredFneodder) As shown in Tab. 3, introducing tailored encoders significantly improves performance across all modalities on both datasets. This demonstrates that fine- gnainedi -ppecit discriminative features are crucial in human perception and reasoning tasks, and tailored encoders can capture the features more effectively than the modality-shared unified encoder.
Ablation Study on UMIP. Furthermore, applying UMIP (++UMIP") leads to performance improve- ment, especially for Action QA, which requires deeper understanding of language-based human instructions and action categories. The results indicate that the multimodal tokens generated by UMIP achieve better alignment with text, thereby enhancing human action reasoning. Moreover, by
8

---

adaptively enhancing the pre-aligned initial embeddings using fine- tie-grained, text-aligned modality features, UMIP can provide multimodal tokens with stronger discriminability.
Baseline
ObeLLM
HolALLM (Curs)
HoloLLM we UMPP
Video (V)
Ho (Oun)
musWaae (M)
(b)
(a) Visualization of aligned tokens from action categories (denoted by different colors) generated by Baseline, OneLLM [30], and MoloLLM for 'Video' and 'mm Wave' modalities. (b) Visualization of multimodal tokens from 2 action categories (denoted by different colors) for 'mm nave' (circles), 'WiFi' (pentagrams), and 'Text' (triangles) modalities generated by HoloL.LM without or with UMIP
Figure 6: Visualization results by tSNE [40].
Use only on ward or human action.
What Ja the hamut's action? only une and
What Is the hanan's actice?
devccizz Lie scl.ion
Kick
ihut Uhe fummn deing slide??
Creference video)
left side.
1
lunge toward
Tuppteg up-
The video ahova tuo begn, ose wast strks had kind argunent.
What action naras persioriing
bext artiuity is the nanar doing withon the video?
what the action of the Tunen in the videot
The man vidte som tveoiing efjectt to stanling
seee larging to the left wiile raistig kis arws tordszetally.
(a) Infrared
(b) LIDAR
(c) Wavee
(d) WiFi-CSI
Figure 7: Qualitative results on sensing modalities. All examples are from the testing set of MM-Fi and XRF55 datasets under the "Cvios-Environmeen" setting.
4.4 Qualitative Results
To intuitively show the effectiveness of HoloLLM, we visualize the aligned multimodal and text tokens of different models in Fig. 6 and show the qualitative results for sensing modalities in Fig. 7.
Visualization Analysis of Multimodal and Text Tokens We randomly select 5 action categories from the MM-Fi dataset and visualize the aligned multimodal tokens generated by 'Baseline', OneLLM [30], and HoloLLM. As shown in Fig. 6 (a), only the multimodal tokens generated by HoloLLM are well grouped based on action categories across all modalities. This shows that tailored encoders are essential for capturing modality-specific discriminative features, while HoloLLM effectively preserves them via UMIP. Besides, we present the tokens of two sensing modalities (mm Wave and WiFi) and the tokens of text captions for two action categories in Fig. 6 (b). Compared with only adopting the tailored encoders and Q-former ('HoloLLM w/o UMIP"), the multimodal tokens generated by HoloL.LM can better align with the ground-truth text captions for different action categories. It intuitively shows that our UMIP helps achieve better multimodal alignment with text.

---

Qualitative Results. We give some qualitative results of HoloLLM in the "CrossEnv" setting in Fig. These results show that HoloLLM can perform Action QA and Action Caption tasks across sensing modalities in diverse environments. More results and analysis are detailed in Appendix C.
Conclusion and Limitations
In this work, we present HoloLLM, an MILM that integrates rare but powerful sensing modalities to enable seamless human perception and reasoning across heterogeneous real-world scenarios. Based on limited data, we propose the Universal Modality Injection Projector (UMIP) to efficiently align sensing modalities with the text via only minimal fine-tuning. Besides, the modality- discriminative features are adequately explored by tailored encoders and adaptively injected into the aligned multimodal tokens through UMIP. Thanks to UMIP, Ho shows significant improvements compared to other state-of-the-art MLLMs on our multisensory language-groundes benchmarks.
Limitation and Future work. As the first attempt to human perception and reasoning based on sensing modalities, our work is limited to human action recognition, question answering, and caption. However, MLLMs needed to support more tasks to meet the requirements real-world applications, such task planning and agent action generation, which will be explored in our future work.
Acknowledgments and Disclosure of Funding
This work is supported by a Start-up Grant from Nanyang Technological University and jointly funded by the Singapore Ministry of Education (MOE) under a Tier-1 research grant.
References
[1] Jianfei Yang, Han Zou, Hao Jiang, and Lihua Xie. Device-free occupant activity sensing using wifi-enabled iot devices for smart homes. IEEE Internet of Things Journal, 5(5):3991-4002, 2018.
[2] Jianfei Yang, Xinyan Chen, Han Zou, Dazhuo Wang, Qianwen Xu, and Lihua Xie. Efficientfi: Toward large-scale lightweight wifi sensing via esi compression. IEEE Internet of Things Journal, 1086-1333) 2022.
[3] Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Pannag R Sanketi, Quan Vuong, Openvla: An open-source vision-languagc-action model. In Annual Conference on Robot Learning, 2024.
[4] Manling Li, Shiyu Zhao, Qineng Wang, Kangrui Wang, Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Erran Li Li, Ruohan Zhang, et al. Embodied agent interface: Benchmarking Iims for embodied decision making. Advances in Neural Information Processing Systems (NeurIPS),
0004482-00034,,,-1100000000 2024.
[5] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in Neural Information Processing Systems (NeurIPS), 36:34892-34916 2023.
[6] Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. Honeybee: Locality- enhanced projector for multimodal 1lm. JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13817-13827, 2024.
[7] Wentong Li, Yuqian Yuan, Jian Liu, Dongqi Tang, Song Wang, Jie Qin, Jianke Zhu, and Lei Zhang. Tokenpacker: Efficient visual projector for multimodal 1lm. arXiv preprint arXiv::400 2407.0239, 2024.
[8] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. International Conference on Machine Learning (ICML), pages 19730-19742, 2023.
10

---

[9] Jialian Li. Jingyi Zhang, Zhiyong Wang, Siqi Shen, Chenglu Wen, Yuexin Ma, Lan Xu, Jingyi Yu, and Cheng Wang. Lidarcap: Long-range marker-less 3d human motion capture with lidar point clouds. In IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 20502-20512 2022.
[10] Fei Wang, Yizhe Lv, Mengdie Zhu, Han Ding, and Jinsong Han. Xrf55: A radio frequency dataset for human indoor action analysis. ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 8(1):1-34, 2024.
[11] Jianfei Yang. He Huang, Yunjiao Zhou, Xinyan Chen, Yuecong Xu, Shenghai Yuan, Han Zou, Chris Xiaoxuan Lu, and Lihua Xie. Mm-fi: Multi-modas non non-intrusive 4d human dataset for versatile wireless sensing. Advances in Neural Information Processing Systems (NeurIPS), 36, 2024.
[12] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems (NeurIPS), 35:25278-2551 -52994, 2022.
[13] Max Bain, Arsha Nagrani, Güil Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In IEEE/CVF International Conference on Computer Vision (ICCV), pages 1728-1732 1738, 2021.
[14] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Annual Meeting of the Association for Computational Linguistics, pages 2556-25655 2018.
[15] Yunjiao Zhou, He Huang, Shenghai Yuan, Han Zou, Lihua Xie, and Jianfei Yang. Metafi++: Wifi-enabled transformer-bases human pose estimation for metaverse avatar simulation. IEEE Internet of Things Journal, 10(16):14128-111 2023.
[16] Xinyan Chen and Jianfei Yang. X-fi: A modality-ivvarian foundation model for multimodal human sensing. arXiv preprint arXiv:2410.10166 2024.
[17] Jianfei Yang. Xinyan Chen, Han Zou, Chris Xiaoxuan Lu, Dazhuo Wang, Sumei Sun, and Lihua Xie. Sensefi: A library and benchmark on heap-leaniin-emmpowere wifi human sensing. Putterns, 4(3), 2023.
[18] Zhongang Cai, Daxuan Ren, Ailing Zeng, Zhengyu Lin, Tao Yu, Wenjia Wang, Xiangyu Fan, Yang Gao, Yifan Yu, Liang Pan, et al. Humman: Multi-modal 4d human dataset for versatile sensing and modeling. In European Conference on Computer Vision (ECCV), pages 557-577, 2022.
[19] Yiming Ren, Xiao Han, Chengfeng Zhao, Jingya Wang, Lan Xu, Jingyi Yu, and Yuexin Ma. Livehps: based scene-level human pose and shape estimation in free environment. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 1281-1291, 2024.
[20] Yuheng Wang, Haipeng Liu, Kening Cui, Anfu Zhou, Wensheng Li, and Huadong Ma. m- activity: Accurate and real-time human activity recognition via millimeter wave radar. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages IEEE, 2021.
[21] Sizhe An and Umit Y Ogras. Fast and scalable human pose estimation using mmwave point cloud. In ACM/IEEE Design Automation Conference, pages 889-894, 2022.
[22] Jingxiao Zheng, Xinwei Shi, Alexander Gorban, Junhua Mao, Yang Song, Charles R Qi, Ting Liu, Visesh Chari, Andre Comman, Yin Zhou, et al. Multi-modal 3d human pose estimation with 2d weak supervision autonomous driving. IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4478 1788-4487,220 2022.

---

[23] Anjun Chen, Xiangyu Wang, Kun Shi, Shaohao Zhu, Bin Fang, Yingfeng Chen, Jiming Chen, Yuchi Huo, and Qi Ye. Immfusion: Robust mmwave-rgb fusion for 3d human body reconstruction in all weather conditions. In IEEE International Conference on Robotics and Automation (ICRA), pages 2752-2758, 2023.
[24] Haoxuan Qu Yujum Cai, and Jun Liu. Lims are good action recognizers. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18395-18406, 2024.
[25] Yunjiao Zhou, Jianfei Yang. Han Zou, and Lihua Xie. Tent: Connect language models with iot sensors for zero-shot activity recognition. arXiv preprint
2023.
[26] Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713, 2024.
[27] Zijia Zhao, Longteng Guo, Tongtian Yue, Sihan Chen, Shuai Shao, Xinxin Zhu, Zehuan Yuan, and Jing Liu. Chatbridge: Bridging modalities with large language model as a language catalyst. arXiv preprini 2rXiv:2305.16100 2023.
[28] Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiangmiao Pang, and Dahua Lin. Pointllm: Empowering large language models to understand point clouds. In European Conference on Computer Vision (ECCV), pages 131-147, 2024.
[29] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), pages 8748-8763, 2021.
[30] Jiaming Han, Kaixiong Gong, Yiyuan Zhang, Jiaqi Wang, Kaipeng Zhang, Dahua Lin, Yu Qiao, Peng Gao, and Xiangyu Yue. Onellm: One framework to align all modalities with language. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26584- 26595, 2024.
[31] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space bind them all. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15180- 15190, 2023.
[32] Bin Zhu, Bin Lin, Munan Ning. Yang Yan, Jiaxi Cui, WANG HongFa, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, et al. Languagebind: Extending video-language pretraining to n-modality by language-base based semantic alignment. In International Conference on Learning Representations (ICI.R), 2024.
[33] Tiange Luo, Chris Rockwell, Honglak Lee, and Justin Johnson. Scalable 3d captioning with pretrained models. Advances in Neural Information Processing Systems (NeurIPS), 36, 2024.
[34] Kristen Grauman, Andrew Westbury, Eugene Byrne Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentrie video. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18995-19013 2022.
[35] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08777. 2023.
[36] Satanjeev Banerjee and Alon Lavie. Meteor:
An automatic metric for mt evaluation with improved correlation with human judgments. In ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65-72, 2005.
[37] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.0920 2023.
12

---

[38] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[39] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In JEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 652-660, 2017.
[40] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal Machine Learning Research, 9(11) 1), 2008
[41] Vimal Mollyn, Karan Ahuja, Dhruv Verma, Chris Harrison, and Mayank Goel. Samosa: Sensing activities with motion and subsampled audio. ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (ACM IMWUT), 2022.
[42] Maytus Piriyajitakonkij, Patchanon Warin, Payongkit Lakhan, Pitshaporn Leelaarporn, Nakorn Kumchaiseemak, Supasorn Suwajanakorn, Theerasarn Pianpanit, Nattee Niparnan, Sub- has Chandra Mukhopadhyay, and Theerawit Wilaiprasitpor.. Sleepposenet: Multi-view learning for sleep postural transition recognition using uwb. IEEE Journal of Biomedical and Health Informatics (IJBHI), 2020.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our paper focuses on developing an MLLM for seamless human perception and reasoning across common and sensing modalities, which matches with our experimental results Sec 4.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer:
[Yes]
Justification:
We discuss the limitation of the work Sec 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
13

---

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide detailed implementation details of our HoloLLM in Sec 4.1. Besides, we provide a detailed data curation pipeline Sec and complete details of the used datasets, established benchmarks, and the training strategy in Sec B. They are sufficient reproduce our experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may
14

---

be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide the code with sufficient instructions to faithfully reproduce the main experimental results in the supplemental materials. All data and code will be released after the paper accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //wpss.ccppubii.. cttpssccccppbccceecccccccccccc ges Coteeeubeeeeeeeeeeeeeeeeo ss iesiionoooiiiii cy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
15

---

Answer: [Yes]
Justification:
We provide all training and testing details in Sec 4.1, Sec B and our codes.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported because it would be too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-gimm error bar than state that they have 96% CI, if the hypothesis of Normality of eиrоrs is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the detailed information about the compute resources in Sec B.2, including the GPU and the training scheme for each experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
16

---

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpps///aorrr....pppbcc//////aa ottppppiiieiiiicccccuoooo.......
Answer: [Yes]
Justification: We provide our code supplemental material, which fully in accordance with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impacts of our paper in Sec. D.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), faimess considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out, For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: This paper does not pose such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
17

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite all original papers that produced the datasets and models, where the names the licenses are included.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: All datasets used in this paper are publicly available multimodal human sensing datasets, the detailed information can be found in the original papers [11, 10].
Guidelines:
18

---

The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: All datasets used in this paper are publicly available multimodal human sensing datasets, the detailed information can be found in the original papers [11, 10].
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The method development in this research does not involve LLMs as an important, original, or non-standard component.
Guidelines:
The answer NA means that the core method development in this research does not involve L.LMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: : https:///neurti //nuriiiss... hte/Conffreence/2222222 for what should or should not be described.
19

---

Appendix
A Technical Details on Data Curation
A.I Action QA generation
A.2 Action Caption generation
B Details on Experimental Settings
B.1 Datasets
B.2 Benchmarks
B.3 Training Details
C Additiooal Experiments
C.I Quantitative Results for Action Recognition
C.2 Complete Qualitative Results and Analysis
C.3 Multimodal Fusion for HoloLLM
C.4 Generalization for HoloLLM
D Broader Impacts
A Technical Details on Data Curation
In this section, we elaborate on more details about the data generation process for action question answering (QA) and captions.
A.1 Action QA generation
We formulate Action QA as question answering task with options. To ensure precision, two seed questions are first annotated by human expert, as shown Fig. 8. We then adopt GPT-4o [35] to
Human expert annotated seed questions
"What human's action according the input data? Answer question by selecting a word or phrase from the attached Action List."
"Plcase select word or phrase from the Action List which could represent the action of the human in the input data."'
Figure 8: Two seed questions for action QA annotated by human experts.
rewrite the seed questions to enhance the diversity, which extends them to a list of 15 questions. Here, the utilized prompt and 5 examples of rewritten questions are listed in Fig. 9.
Afterwards, for each data sample from certain modality, question is randomly selected from the list to construct the QA question ("Question') ). Furthermore, all action categories from the sensing dataset, along with the action label of the data sample, are appended to the question generate the Action List (options) and the corresponding Answer. As shown in Fig. 10, a typical Action QA data sample can be formulated as: [Question, Action List, Answer) along with the input videos from diverse modalities (The vision modality is shown in Fig. 10). During training and testing, the input videos combined with Question and Action List are passed to the Ho to predict the Answer.
A.2 Action Caption generation
The textual descriptions of human actions are not available in both the MM-Fi and XRF55 datasets. Therefore, the captions cannot be directly generated using in-context learning. To this end, we propose a human- collaborative pipeline to address caption generation. Specifically, we evenly select 108 110 samples from the MM-Fi / XRF55 dataset across various environments, subjects, and action categories to form seed data samples. These seed samples are captioned by human experts,
20

---

Prompting GPT- to rewrite questions
Prompts:
"You are professional writer. Now I want you to do sentence rewriting task. I will give you a sentence enclosed in parentheses, you give me len rewriling sentences. You camnot change the original meaning of the sentence. On this basis, you should make the rewritten sentenee structure as diverse as possible."
Rewritten Questions:
"Based on the input data, what action is the human taking? Select a word or phrase from the attached Action List,"
"What action does the input data indicate the human is performing? Choose term from the Action List to answer.
"Identify the human's action from the input data by selecting word or phrase from the Action List provided."
"A AAccording to the input data, what is the human action? Select an appropriate term from the Action List."
"Find word or phrase in the Action List that can indicate the human's 's action according to the input data."
Figure 9: Prompting GPT-4o [35] to rewrite the seed questions to enhance diversity. We provide 5 questions rewritten by GPT-4o as examples.
Data sample of Action QA
Question:
"What is the human's action according to the input data? Answer the question by selecting word phrase from the attached Action List."
Action List:
"Action List: 'stretching and relaxing', 'horizontal chest expansion', 'vertical chest expansion', "left twist', 'right twist', 'mark time', 'left limb extension', 'right limb extension', 'lunge toward left-front', 'lunge toward right-front', 'both limb extension', 'squat',
Input Video
'raising left hand', 'raising right hand', 'lunge toward left side', 'lunge toward right side', 'waving left hand', 'waving right hand', "picking up things', 'throwing toward left side', 'throwing toward right side', 'kicking toward left side', kickkngg toward right side', 'left body extension', 'right body extension', jjumping up', 'bowing"]"
Action Label: 'bowing'.
Figure 10: A typical data sample for the Action QA task.
and each caption is rewritten by GPT-4o as detailed in Appendix A. to enhance diversity, resulting in captions for each seed sample. We then transfer seed sample to an in-context caption sample: (Question, Video, Caption) by (1) Add a fixed question: Please give detailed descriptions of human't 's action., (2) Attach the video sequence of the seed sample, and (3) Randomly select a caption. A typical in in-context caption sample is shown in Fig. 11 (a).
Consequently, for the remaining data samples in both datasets, we adopt LLa Video [26] to automatically generate captions through in-context learning. As illustrated in Fig. 12, for a specific data sample s, we first build the prompting message by progressively integrating the 'system prompt' and an in-context caption sample of the same category as s. The in-context caption sample contains input sample (f 'video'] with sample [ qquestion') '] and output sample ('mppee[(rreer ['response") the query [ video') '] and query [ quesstion)] '] are appended to the message as the sample to be captioned. The complete prompting message is passed to the LLaVA- Video to obtain the Caption. As shown in Fig. 11 (b), a data sample for the Action Caption task can be formulated as: (Question, Caption), which is shared across all modalities. During training and testing, the input videos combined with Question are passed to the HoloLLM to generate the Caption.
Then,
21

---

In-context caption sample of Action Caption
Data sample of Action Caption
Question:
Question:
Please give detailed description human's action.
Please give detailed description human's action.
Caption:
Caption:
In the video, the boy places his hands on his thighs and bows toward the camera.
Standing in front of the table, the man bows to the camera, possibly to express gratitude.
Input Video:
Input Video:
(a)
(b)
Figure 11: (a) A aumannandottted in-context caption sample. (b) A typical data sample for the Action
Caption task.
Prompting I.L.VA Video to annotate caption via in-context learning
messages = [ ("role": sysstem" "content": ..* You are an Al visual assistant, and you are secing vidco. You are going to do human action caption task based on the vidco. You will get vidco and the Action Label for the human action in the video. The Action Label will be enclosed in pair of brackets. You need to give helpful, detailed, and accurate descriptions on the human action.
Here examples. You need generate description based your observation the given examples." """]
sample
in un__ontext___ cetttttcpptooooooeee caption_samples- samples:
dessages. (""ole""""usev"" "content": [sample[ 'video'), sample[ question }]})
messages. append (""ssse::"assittant"" "content": sample[ 'responses' }}-)
messages. ( "content": [query[ [video'], query[ question']]]
Prompting messages passed to LLaVA-Video [26] to automatically generate cap- tion for sensing data. Human-annotated in-context caption samples are included in the prompt, where each sample has input sample I' with sample (aleeliiqessssss and output sample [ response
Figure 12:
B Details on Experimental Settings
In this section, we provide more details the experimental settings. Specifically, we first present more details for the datasets Appendix B. Followed by detailed statistics for three experimental settings in Appendix B.2. Finally, the training details of HoloLLM are elaborated in Appendix B.3.
B.1 Datasets
MM-Fi. MM-Fi consists 27 action categories and 40 human subjects from different environ- ments. Each human subject contains synchronized sequences with aspects of 5 modalities: Video (V), Depth images (D), LiDAR point clouds (L), mm Wave point clouds (M), and WiFi-CSI (W). In total, there 16.448 multimodal sequences in the MM-Fi Fi datasets, and 5 frames are evenly sampled from each sequence to form a data sample.
XRF55. For XRF55, we only consider the human subjects with video modality, resulting in 19 human subjects from different environments with 55 action categories. Each human subject contains synchronized sequences with aspects of 5 modalities: Video (V), Depth images (D), Infrared images
22

---

(I), RFID signals (R), and WiFi-CSI (W). In total, there are 19,800 multimodal sequences in the XRF55 datasets, and 10 frames are evenly sampled from each sequence form data sample.
B.2 Benchmark
We design three experimental settings: (1) Random Split (Random), (2) Cross-Subjed Split (Cross- Sub), and (3) Cross-Environment Split (CrossEnv). Specifically, 'Random' involves a random split of all samples with ratio of 3:1, and 'CrossSub' "CrossEnv" selects samples from nonoverlapping human subjects/ environments for training and testing.
Scttings
MM-Fi
XRF55
Train Size
Test Size
Train Size
Test Size
Random
12,336
4,1T2
14,850
4,950
CrossSub
11,657
4,791
14,300
5,500
CrossEnv
12,565
3,883
16,500
3,300
Table 4: The detailed statistics of three experimental settings HoloLLM.
Detailed statistics for three experimental settings on the sizes of the training and testing sets are summarized in Tab.
B.3 Training Details
The AdamW optimizer with 81 0.9, B2 0.95, and weight decay of 0.1 is adopted in training. For stage one, we utilize the training set the corresponding experimental setting (Random, CrossSub, CrossEnv) pretrain the tailored encoders for 120 epochs learning rate is initialized to 0.1 with linear warmup strategy for 10 epochs, and then decayed at the 60-th and 100-th epochs with decay factor of 0.1. For stage two, we train the HoloLLM GPUs for epochs. We set the accumulated iterations to 4 and form an effective batch size of 64 / 48 for the MM-Fi / XRF55 datasets, respectively. Followed by OneLLM [30], the linear warmup strategy is utilized for the first 2K iterations with maximum learning rate of 2c-5.
single A100 GPU. The
A 100
Settings
Madels
Sources
Types
MM-Fi
XRPSS
L
W
Tekenpacker
arXiv'24 24
Proj
13.6
14.5
32.8
18.9
11.5
18.7
9.2
8.2
7.1
2.2
6.5
6.6
Random
Homeyhee
CVPR'24
OneLLM
CVPR 24
Proj
9.0
9.0
16.2
6.1
9.0
9.9
2.1
2.0
1.5
1.9
2.3
2.0
ImageBind
CVPR'25
99.5
96.8
63.7
13.2
11.7
$7.0
90.8
59.1
93.3
14.6
51.1
61.8
HoloL.LM
Proj
99.8
99.7
97.3
92.8
83.5
86.5
98.0
96.9
97.5
39.1
37.7
73.8
Tokenpacker
arXiv'24
CrossSub
Honeybee
OneLLM
CVPR'24
CVPR'24
Proj
11.6
8.3
9.3
27.2
13.9
6.6
8.3
14.1
3.0
4.2
4.3
1.8
2.4
4.9
2.6
3.8
ImageBind
CVPR
Enc
85.7
70.7
56.9
17.7
11.0
48.4
18.6
14.4
22.0
4.2
5.7
13.0
HoloLLM
98.4
76.6
75.4
Tokenpacker
arXiv'24 24
Proj
15.0
9.4
20.2
11.7
10.7
13.4
9.8
10.1
6.2
2.0
4.5
6.5
Honeyhee
CVPR'24
CrossEns
OneLLM
CVPR 24
Proj
9.5
9.5
5.6
6.8
2.4
6.8
1.5
3.5
1.8
1.9
2.3
2.2
ImageBind
19.7
21.1
HoloLLM
Proj
80.6
93.2
61.0
53.0
9.5
$9.5
28.9
12.4
28.3
1.7
2.5
14.8
Table 5: Evaluation of Human Action Recognition on MM-Fi [11] and XRE55 [10] across three settings. The Accuracy (%) is reported.
C Additional Experiments
In this section, we present the quantitative results for Action Recognition in Appendix C.I, along with complete qualitative results and a comprehensive analysis for all modalities Appendix C.2. Additionally, the results of naive multimodal fusion strategy provided in Appendix C.3.
23

---

Please word phrase to describe the tusan action.
Mhat 15 the humon action? UsD word to andwor.
Provi de one-word answer to describe the husan action.
Squat.
P
Horizontal chest expansion.
what is the huran's is action In the video?
khat person doingg in video?
kick soesone.
khat the tarsan dotnng in video?
The mon Is seen squacting dose and then standing again..
the man in the video is soen stretching his ares ouz to the sides, which is comnon excreise to oxpand the chest.
The video shows tsn bays, of whon kicks the other, irplying that they had some kind of argusent.
(a) Video
(b) Depth
(c) Infrared
Use only one word or phrase to describe the trman action.
What is the huran's action? Use only one word phrese to respend.
What i5 the human's action?
Provide one-word
freforance vidoo)
(roforence vidzo)
videe))
5
uung left
Lunge toward left wiche
Jurping
khat action human
What activity is the busan daing thin the video?
what is the action of the huran in the video?
penfornting
video?
The san in the video is sean throwang an object to the left ssde while standing in a room.
The the vide video is lunging to the leftt while raiwing his torizontally.
girl
jumps
(d) LIDAR
(e) mm Wave
i WiFi-CSI
Figure 13: Qualitative results on sensing modalities. All examples are from the testing set of MM-Fi
and XRF55 datasets under the "Cross-Environment" setting.
C.1 Quantitative Results for Action Recognition
We provide the quantitative results corresponding to Fig. 5 for Action Recognition in Tab. 5. As discussed in Sec 4.2, HoloLLM is superior to other MLLMs on action recognition tasks for almost all modalities. For certain sensing modalities, comparable performances are observed under the "CrossSub" and "CrossEnv" settings. In fact, some sensing modalities are sensitive to different subjects and scenarios. Enhancing the generalization capability of sensing modalities toward diverse subjects and environments raises an important topic for the future.
C.2 Complete Qualitative Results and Analysis
We show the complete qualitative results of HoloLLM under "CrossEnv" setting for both common and sensing modalities in Fig. 13. These results show that HoloLLM can perform Action QA and Action Caption tasks across multiple modalities in diverse environments. Moreover, the white dashed circles in Fig. 13 (b) and (c) illustrate that HoloLLM possesses the ability to capture fine- -nained sodality-speciffe action information and reason over it to identify the correct action category.
C.3 Multimodal Fusion for HoloLLM
We conduct a preliminary exploration of HoloLLM's multimodal fusion capability. Specifically, we first adopt naïve fusion strategy, in which aligned multimodal tokens generated by UMIP are directly concatenated. Multimodal reasoning is achieved by prepending the fused tokens to human instructions before feeding them into the LLM. For the MM-Fi and XRF55 datasets, we consider the Vision (V), mm (M), WiFi (W) and Vision (V), Infrared (I), WiFi (W) modalities, respectively. The results are summarized in Tab. 6.
24

---

Modality
MM-Fi
Modality
XRF55
Recognition
QA
Caption
Recognition
QA
Caption
V
80.6
79.5
25.7
V
28.9
25.9
19.8
M
61.0
61.4
24.5
28.3
22.1
17.1
W
9.5
8.2
15.9
2.5
4.5
13.7
V+M
84.6
66.4
25.0
V+I
34.3
21.5
19.2
V+W
80.9
52.1
23.7
V+W
28.3
13.9
18.7
M+W
61.0
57.6
22.9
I+W
27.2
9.2
16.1
V+M+W
86.1
69.1
24.4
V+I+W
29.7
19.2
18.9
Table Results naive multimodal fusion on MM-Fi and XRF55 under the "roos-Envvroommen" setting.
shows naive multimodal fusion can enhance action recognition performance for certain modalities (highlighted in bold). However, for action QA and captioning tasks, naive multimodal fusion fails to improve performance.
Furthermore, conduct experiments to explore more advanced adaptive fusion techniques, including weighted sum (WS) and max pooling (MP) across multimodal tokens. the weighted-sun technique, the weights assigned to Vision and mm Wave/Infrared modalities are 0.8 and 0.2, respectively.
Modality
MM-Fi
Modality
XRF55
HAR
QA
Caption
HAR
QA
Caption
V
80.6
79.5
25.7
V
28.9
25.9
19.8
M
61.0
61.4
24.5
I
28.3
22.1
17.1
V+M (Naīve)
84.6
66.4
25.0
V+I (Naīve)
34.3
21.5
19.2
V+M (WS)
82.6
81.4
26.9
V+I (WS)
30.9
26.2
20.6
V+M (MP)
80.6
80.4
26.7
V+I (MP)
28.9
20.5
18.9
Table 7: Comparison of different multimodal fusion strategies MM-Fi and XRF55 datasets.
As shown in Tab. 7, the weighted sum fusion strategy outperforms single-modality and naïve multimodal fusion on Action QA and captioning tasks. We believe that more advanced, learning- based multimodal fusion techniques could further improve performance. We expect HoloLLM to serve as challenging benchmark that motivates the research community to develop deeper insights into multimodal fusion methods.
C.4 Generalization for HoloLLM
HoloLLM generalizes to new modalities more efficiently than other MLLMs due to the novel UMIP architecture. Specifically, each new modality equipped with a lightweight, modality-seecific encoder that is extensively pretrained to capture fine-graines features. The new modality is then efficiently integrated into HoloLLM by UMIP with minor data and fine-tuning.
We demonstrate that HoloLLM achieves superior data efficiency over the 'Baseline' model on two novel modalities: Audio [41] and UWB [42]. The results are summarized in Tab.
HAR
Action QA Audio
Audio
UWB
UWB
Baseline
37.71
9.46
30.73 54.17
11.33
HoloLLM
63.94
16.77
16.45
Table 8: Results of HAR and Action QA tasks novel Audio and UWB modalities.
Compared to the BBaelliee model, HoloL.L can be efficiently generalized to novel modalities with minor data and fine-tuning.
25

---

D Broader Impacts
This work introduces HoloLLM, MLLM that achieves seamless human perception and reasoning by integrating sensing modalities. HoloLLM establishes a multisensory foundation model, which beneficial for developing embodied agents that are applicable in diverse real-world scenarios, including low-light environments, occlusions, and privacy- may suffer similar concerns with other MLLMs, such as hallucinating or meaningless outputs (especially for some sensing modalities under cross-subject or cross-environment settings), inherited biases from base models, and energy consumption due large-scale parameters. This raises important research topics such as enhancing the generalization capability of sensing modalities, aligning the base model with human intention, and efficiently pruning the foundation model. Despite these challenges, the release of HoloLLM would be beneficial, it would foster further development of embodied AI.
scenarios. However, HoloLLM
26


Reviews:
[{"review_id": "c2k0AposF4", "reviewer": "Reviewer_DvFs", "summary": "This paper presents HoloLLM, a large language model that understands human actions using not just visual data, but also other sensor types like LiDAR, infrared, WiFi, and mmWave radar. These sensors work better than cameras in dark, blocked, or privacy-sensitive places.\n\nSince these sensor types don’t have much paired text data and are very different from images, the authors design two solutions: (1) Tailored encoders to extract useful features for each sensor type. (2) UMIP, a module that combines those features with CLIP embeddings and gradually aligns them to the language model.\n\nThey also build a new dataset with human + AI collaboration for tasks like action recognition, question answering, and captioning. Experiments show that HoloLLM works much better than previous models (up to 30% better) on these tasks.", "questions": "1. Your human-VLM data curation pipeline is insteresting, but how do you ensure annotation quality—especially for captions generated by LLaVA-Video? Could you report: Inter-annotator agreement (if any human review was done); Human evaluation of generated captions (e.g., fluency, accuracy scores); Percentage of samples verified or discarded. Addressing this would increase trust in your benchmarks, which are key to evaluating your model. Without quality control, the evaluation may be biased or noisy.\n\n2. HoloLLM is framed as a general multisensory foundation model, yet it is only evaluated on MM-Fi and XRF55, which have similar indoor settings and overlapping sensor types. How well does HoloLLM generalize to new domains, sensors, or unseen environments? If feasible, please include zero-shot or few-shot results on other datasets, such as SenseFi or Widar.", "limitations": "The authors include a brief limitations section noting that HoloLLM is currently limited to human action understanding tasks (recognition, QA, captioning), and does not yet handle higher-level capabilities like task planning or real-world agent control. This is a good start.  Technical limitations are not fully addressed because the model is only evaluated on two similar indoor datasets.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Quality: (1) Strengths: The technical methodology is sound and well-motivated. The UMIP design is carefully engineered to address the unique challenges of data scarcity and sensor heterogeneity. The experiments are broad, covering five sensing modalities across two datasets and three settings (Random, CrossSub, CrossEnv). Ablation studies are helpful in validating both the tailored encoders and UMIP. (2) Weaknesses: The empirical rigor could be improved: Some accuracy (e.g., CrossEnv) are small and need stronger design support. The validation of the human-VLM collaborative pipeline is weak—there is no measure of annotation quality or consistency. No comparison with multimodal transformers trained end-to-end or with stronger adaptive fusion techniques beyond naive fusion.\n\nClarity: (1) Strengths: The paper is logically organized and generally easy to follow at a high level. Motivation is concrete (smart homes, privacy, occlusion), and the contribution is clearly stated. (2) Weaknesses: Key sections, especially Section 3.2 (UMIP), are too dense, please consider adding pseudocode or a simplified diagram of the iterative process. The link between CLIP embeddings and non-visual modality alignment needs clearer explanation.\n\nSignificance: (1) Strengths: The paper addresses an important and under-explored problem: language grounding with non-visual sensing. (2) Weaknesses: Broader generalization across datasets or unseen modalities is not demonstrated.\n\nOriginality: (1) Strengths:The two-stage training strategy and the human-VLM collaborative annotation pipeline are thoughtful and well-justified. (2) Weaknesses: The idea of cross-attention-based fusion is not new—more emphasis is needed on what differentiates UMIP from prior Q-former-style projectors (e.g., OneLLM, BLIP-2). (2) Need to better position UMIP versus recent work on universal modality encoders (e.g., ImageBind, OmniFusion, UnifiedIO). (3) The use of CLIP for non-visual modality embedding needs stronger justification, or at least empirical support.", "was_revised": true, "final_justification": "The authors have addressed my main concerns with new experiments on annotation quality, adaptive fusion, and generalization to new datasets and modalities.While cross-environment performance is still limited and the novelty is moderate, I maintain my Borderline accept recommendation.", "rebuttal": "We appreciate the reviewer DvFs for the constructive comments and the appreciation. Here we answer all the questions by extensive experiments and clarification.\n\n**Q1: Some accuracy (CrossEnv) is small and need stronger design.**\n\n**Answer:** We appreciate the suggestion and admit the issue. In fact, the “cross-subject” and “cross-env” generalization of some modalities (e.g., WiFi) is very challenging, as theoretically they are susceptible to variations in subjects and environments, and the data scarcity makes these tasks even harder. Though the tasks are hard, HoloLLM still outperforms other SOTA MLLMs in both settings by a large margin. In future work, we aim to enhance the generalization ability of HoloLLM by constructing a large-scale sensing dataset. Specifically, we plan to integrate existing public datasets [1, 2] and collect additional data from a broader range of environments (e.g., outdoor scenarios) and tasks (e.g., task planning and agent action generation). By fine-tuning HoloLLM with this new dataset, we believe that HoloLLMv2 will obtain strong generalization ability. This work focuses on aligning new yet powerful sensing modalities with LLM, given limited data and various modalities, and thus offers a novel technical solution with a comprehensive benchmark.\n\n**Q2: There is no validation of the human-VLM collaborative pipeline.**\n\n**Answer:** As suggested, we invite 10 volunteers and sample 1% caption data from MM-Fi (165 / 16,448) and XRF55 (198 / 19,800) for inter-annotator agreement evaluation (Fleiss’ Kappa score [6]). Specifically, the volunteers are asked to perform a three-class classification task: “Does the caption match (label 2), partially match (label 1), or not match (label 0) the video?” The matching score for each class is defined as: # of samples with label n / # of all samples. Results are summarized below:\n\n| |MM-Fi|XRF55|\n|-|-|-|\n|Matching scores (match / partially match / not match) |91.21% / 8.79% / 0%|85.76% / 14.24% / 0%|\n|Fleiss Kappa score|0.7354|0.6278|\n\nThe results indicate that annotations generated by our pipeline closely match the human action videos, with no samples labeled as ‘not match’ within the sampled data. Additionally, the two volunteers achieve substantial agreement, with Fleiss’ Kappa scores > 0.6 on both datasets. Furthermore, we will release all annotations along with the HoloLLM source code.\n\n**Q3: Comparison with methods trained end-to-end or with stronger adaptive fusion techniques.**\n\n**Answer:** We have already compared HoloLLM with end-to-end trained SOTA methods, including ImageBind and OneLLM. Specifically, ImageBind is trained end-to-end using an InfoNCE loss across modalities, while OneLLM is fine-tuned end-to-end with a task-specific instruction tuning loss. The results are presented in Tab. 1 and Tab. 2 in the original manuscript.\n\nIn addition, we agree with the suggestion that adopting naïve fusion (concatenating all multimodal tokens) alone is insufficient. Therefore, we add experiments to explore more advanced adaptive fusion techniques, including weighted sum (WS) and max pooling (MP) across multimodal tokens.\n\n||MM-Fi||||XRF55|||\n|-|-|-|-|-|-|-|-|\n| |HAR|QA|Caption| |HAR|QA|Caption|\n|V|80.6|79.5|25.7|V|28.9|25.9|19.8|\n|M|61.0|61.4|24.5|I|28.3|22.1|17.1|\n|V+M (Naive)|**84.6**|66.4|25.0|V+I (Naive)|**34.3**|21.5|19.2|\n|V+M (WS)|82.6|**81.38**|**26.9**|V+I (WS)|30.91|**26.24**|**20.6**|\n|V+M (MP)|80.6|80.43|26.7|V+I (MP)|28.9|20.48|18.9|\n\nThe results show that the weighted sum fusion technique outperforms single-modality and naïve multimodal fusion on Action QA and captioning tasks. We believe that more advanced, learning-based multimodal fusion techniques could further enhance performance. We hope that HoloLLM can serve as a challenging benchmark to inspire the research community toward more insightful contributions in multimodal fusion techniques.\n\n**Q4: Clarify Section 3.2 (UMIP) by adding pseudocode and the link between CLIP embeddings and multimodal alignment.**\n\n**Answer:** Due to the limited characters, we will **directly add** the pseudocode in the revised manuscript. Here, we further clarify the link between CLIP embeddings and non-visual modality alignment. As illustrated in Eq.(1), we leverage CLIP to generate pre-aligned embeddings $Y_{CLIP}^m$  for modality $m$, which are condensed using adaptive average pooling and used as queries (Eq.(3)). **Therefore, CLIP provides pre-aligned queries that contain coarse modality-specific features.** This enables the UMIP to adaptively identify aligned modality-specific features via cross-attention and progressively integrate them for enhancement (Eq.(4)).\n\n**Q5: Demonstrate generalization of HoloLLM to new domains (e.g., SenseFi), sensors, or unseen environments?**\n\n**Answer:** As suggested, we add experiments for new domains and sensors.\n\n(1) Generalization to New Datasets.\n\nWe report zero-shot results on SenseFi [8]. Specifically, SenseFi performs human sensing using WiFi-CSI signals and includes six novel action categories. We then apply the ‘Baseline’ model (as detailed in line 248) and HoloLLM trained on the MM-Fi dataset under the ‘random’ setting to perform zero-shot human action QA on the official SenseFi test set, which contains 264 samples.\n\n|Model|Action QA|\n|-|-|\n|Baseline|5.43|\n|HoloLLM|16.67|\n\nIn zero-shot settings, HoloLLM’s superior performance shows its strong generalization ability and discriminability.\n\n(2) Generalization to Unseen Sensing Modalities\n\nHoloLLM can integrate new modalities more efficiently than other MLLMs, owing to the novel architecture of UMIP. Specifically, we extensively pretrain a lightweight, modality-specific tailored encoder for each new modality to sufficiently capture fine-grained features. The new modality is then efficiently integrated into HoloLLM by UMIP with minor data and fine-tuning.\n\nWe add an experiment to demonstrate the superior data efficiency of HoloLLM compared to the ‘Baseline’ model on two novel modalities, audio [9] and UWB [10]. The results are summarized below.\n\n| |HAR| |Action QA| |\n|-|:-:|:-:|:-:|:-:|\n| |Audio|UWB|Audio|UWB|\n|Baseline|37.71|9.46|30.73|11.33|\n|HoloLLM|63.94|16.77|54.17|16.45|\n\nCompared to the ‘Baseline’ model, HoloLLM can be efficiently generalized to novel modalities with minor data and fine-tuning.\n\n(3) Generalization to Unseen Environments.\n\nWe already presented the results in Tab. 1 and Tab. 2 in the original manuscript under the ‘CrossEnv’ setting.\n\n**Q6: Position UMIP versus prior Q-former-style projectors and universal modality encoders.**\n\n**Answer:** The differences between HoloLLM and existing methods can be categorized into four key aspects. **(1) Task.** It deals with a more challenging problem on data-scarce yet powerful sensing modalities with heterogeneous features. **(2) Model Architecture.** It proposes a novel projector that incorporates fine-grained, modality-specific features for enhancing multimodal alignment. **(3) Training Strategy.** It presents a two-stage training strategy to sufficiently capture the modality-specific features and efficiently align multimodal inputs with the text space. **(4) Benchmark Results.** It establishes the first multisensory benchmark for human perception and reasoning.\n\nWe also present a detailed comparison of HoloLLM with Q-former-style projectors (OneLLM [7], BLIP-2 [11]) and universal modality encoders (ImageBind [12], OmniFusion [13], UnifiedIO [14]). Due to the limited characters, we only include OmniFusion and UnifiedIO here. Please refer to our response to **Reviewer 4VBc Q1** for all other methods.\n\n| |Task|Model Architecture|Training Strategy|Benchmark Results|\n|-|-|-|-|-|\n|HoloLLM (ours)|Align **data-scarce** but powerful **sensing modalities** (e.g., WiFi) with LLM using only limited data, while preserving their **heterogeneous features** for seamless human perception and reasoning.|A unified multimodal projector that considers **modality-specific features** by taking features from CLIP and **tailored encoders** as coarse queries and keys/values.|**Two-stage training.** (1) Pretrain tailored encoders to capture modality-specific features. (2) Fine-tune the UMIP to generate discriminative multimodal tokens aligned with text.|HoloLLM achieves SOTA performances on two **human sensing** datasets, establishing the first **multisensory benchmark** for human perception and reasoning.|\n|OmniFusion [13]|Fuse tangent images, RGB images, and 3D geometric features to **generate high-quality dense depth maps.**|A typical UNet architecture to generate depth map.|Train the **whole UNet** from scratch with the task-specific loss for depth image prediction.|Evaluate on depth image generation benchmarks.|\n|UnifiedIO [14]|Align vision-related modalities (e.g., images, bounding boxes) with text for sparse and dense vision tasks (e.g., VQA or segmentation).|Pure transformer encoder-decoder architecture.|Train the whole transformer from scratch via **large-scale vision-text pairs**.|Only evaluated on vision-language benchmarks for sparse and dense visual tasks, such as GRIT.|\n\n**Q7: Justify the use of CLIP for non-visual modality embedding.**\n**Answer:** Our use of CLIP for non-visual modality embeddings is inspired by both OneLLM [7] and Meta-Transformer [4], which suggest that **a well-pretrained transformer can serve as a universal cross-modal encoder.**\nConsequently, HoloLLM adopts the CLIP vision encoder as a unified encoder to generate pre-aligned initial embeddings for each modality.\n\nHowever, we find that the CLIP encoder is challenging to \"learn heterogeneous characteristics of rare sensing modalities\", thus we design tailored encoders to capture modality-specific features and integrate them via UMIP, denoted as one of the main novelties. As illustrated in Tab.3, the modality-specific knowledge is critical to enhancing the discriminability.\n\n**References**\n\nDue to limited characters, please refer to the responses of Reviewer 4VBc for references."}, {"review_id": "8OOH1BF36I", "reviewer": "Reviewer_78xg", "summary": "This paper introduces HoloLLM, a multimodal large language model (MLLM) designed to integrate a range of sensing modalities beyond vision, for typical home environments. One contribution is UMIP, a framework that unifies pre-aligned modalities with text modality. The authors also propose a human–VLM collaborative pipeline and use it to construct two new benchmarks that facilitate language grounded human sensing. Experimental results on these benchmarks demonstrate that HoloLLM outperforms previous state-of-the-art models.", "questions": "1) see weaknesses\n2) what message does Figure 6 right column intend to convey?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths. \n\n- Exploring MLLMs that extend beyond vision to incorporate additional modalities available from embodied agents (e.g., robots) is an important yet underexplored area.\n\n- This paper proposes UMIP, a simple yet effective architecture that integrates modality-specific encoders to address challenges in data-scarcity modality-to-text alignment.\n\n-  To support evaluation, the authors introduce two new benchmarks for language-grounded human sensing, built by extending existing datasets.\n\nWeaknesses\n\n-\tWhat are the key architectural or methodological differences between HoloLLM and the baseline approaches it was compared against? HoloLLM shows significantly improved performance, often by large margins. It would be helpful to clarify whether this improvement is primarily due to its architectural design (e.g., UMIP) or differences in training data. Specifically, were the modality-specific encoders in the baseline models trained using the same data as HoloLLM, or were there discrepancies in training conditions that could account for the performance gap?\n\n-\tIn Table 1 and Table 3, all different modalities are reported separately. What is the performance by integrating these modalities together? \n\n-\tWhat is the inference time latency?", "was_revised": true, "final_justification": "The rebuttal clarified my questions. The paper makes a nice contribution to an under-explored area (i.e., unconventional modalities).", "rebuttal": "We sincerely appreciate the reviewer 78xg for the insightful questions and detailed review. Here, we answer all the questions and add extensive experiments for clarification.\n\n**Q1: What are the key architectural or methodological differences between HoloLLM and the baseline approaches it was compared against? It would be helpful to clarify whether this improvement is primarily due to its architectural design (e.g., UMIP) or differences in training data. Specifically, were the modality-specific encoders in the baseline models trained using the same data as HoloLLM, or were there discrepancies in training conditions that could account for the performance gap?**\n\n**Answer:** We appreciate the suggestion to clarify the key differences between HoloLLM and the two baseline approaches we adopted in the ablation study (Tab.3 in the original manuscript). Here, we summarize the key differences among the three models.\n\n|Models|Encoders|Projectors|Training Data|\n|-|-|-|-|\n|Baseline|The unified CLIP encoder $E_{CLIP}(\\cdot)$|Q-former projector with learnable queries and CLIP features as keys and values.|The same training data (detailed in Appendix B.2) is used for both pretraining the tailored modality-specific encoders and fine-tuning the projectors. There are no differences in training data between HoloLLM and the other two baseline models.|\n|Baseline+Tailored Encoder|The tailored modality-specific encoder $E_T^m(\\cdot)$| Q-former projector with learnable queries and modality-specific features as keys and values.|Please see above.|\n|Baseline+Tailored Encoder+UMIP (HoloLLM)| Both the unified CLIP encoder $E_{CLIP}(\\cdot)$ and the tailored modality-specific encoder $E_T^m(\\cdot)$|UMIP with CLIP features as coarse queries and modality-specific features as keys and values.|Please see above.|\n\nIn summary, we ensure a fair comparison, and the difference between HoloLLM and the baseline models lies entirely in our architectural designs, including the tailored modality-specific encoders and the UMIP. Specifically, the tailored encoders effectively capture heterogeneous modality-specific features within sensing modalities through pretraining. Meanwhile, the UMIP achieves efficient multimodal alignment while preserving modality-specific discriminative information. Both components play critical roles in significantly improving performance, as demonstrated by the ablation.\n\nWe will incorporate the clarification regarding the differences between HoloLLM and the other baseline models in the revised manuscript.\n\n**Q2: In Table 1 and Table 3, all different modalities are reported separately. What is the performance of integrating these modalities together?**\n\n**Answer:** We appreciate the comment regarding the multimodal fusion capabilities of HoloLLM. We have provided experimental results of naïve multimodal fusion, simply concatenating all multimodal tokens, in Appendix C.3 in the manuscript. The results show that naïve multimodal fusion can improve action recognition performance. However, for action QA and captioning tasks, naïve multimodal fusion fails to improve performance. We conjecture that this is because naïve multimodal fusion does not fully leverage complementary information across modalities nor effectively reduce redundancy.\n\nTo make a stronger multimodal fusion baseline, we further supplement an experiment to explore stronger adaptive fusion techniques, including weighted sum (WS) and max pooling (MP) across multimodal tokens.\n\n||MM-Fi||||XRF55|||\n|-|-|-|-|-|-|-|-|\n| |HAR|QA|Caption| |HAR|QA|Caption|\n|V|80.6|79.5|25.7|V|28.9|25.9|19.8|\n|M|61.0|61.4|24.5|I|28.3|22.1|17.1|\n|V+M (Naive)|**84.6**|66.4|25.0|V+I (Naive)|**34.3**|21.5|19.2|\n|V+M (WS)|82.6|**81.38**|**26.9**|V+I (WS)|30.91|**26.24**|**20.6**|\n|V+M (MP)|80.6|80.43|26.7|V+I (MP)|28.9|20.48|18.9|\n\nThe results show that the weighted sum fusion techniques outperform single-modality and naïve multimodal fusion on Action QA and captioning tasks. We believe that more advanced, learning-based multimodal fusion techniques could further enhance performance. We hope that HoloLLM can serve as a challenging benchmark to inspire the research community toward more insightful contributions in this multimodal fusion technique.\n\n**Q3: What is the inference time latency?**\n\n**Answer:** We appreciate the suggestion to show inference latency, which is critical for real-world applications. Specifically, we select the video and WiFi as representative common and sensing modalities to conduct a fair comparison on the same GPU setting, i.e., one A100. We report the inference times for ImageBind (encoder-based method), OneLLM (projector-based method), and HoloLLM on the MM-Fi dataset across all three tasks under 'random' settings. Inference latency is calculated by averaging the time across all batches with a test batch size of 16. The inference latencies of HoloLLM and other state-of-the-art MLLMs are summarized in the following table.\n\n|Models| | | |Inference latency (seconds/batch)| | |\n|-|-|-|-|-|-|-|\n| |HAR| |Action QA| |Action Caption| |\n| |V|W|V|W|V|W|\n|ImageBind|21.1|15.2|23.3|18.0|31.0|27.4|\n|OneLLM|20.0|15.1|23.0|16.5|30.0|24.1|\n|HoloLLM|19.7|14.7|22.0|16.1|29.5|22.5|\n\nThe inference latencies of all MLLMs are nearly identical. We think this is reasonable since the main inference bottleneck occurs on the LLM side, and all models use the same LLaMA2-7B backbone.\n\n**Q4: What message does Figure 6 right column intend to convey?**\n\n**Answer:**  The message conveyed by the right column of Fig. 6 in the manuscript highlights the significance of our UMIP.\n\nAs discussed in **Q1**, HoloLLM w/o UMIP (i.e., Baseline + Tailored Encoder) uses the original Q-former as projector, whereas HoloLLM employs the proposed UMIP as projector. The right column of Fig. 6 shows that the multimodal tokens generated by UMIP in HoloLLM, along with text tokens from ground-truth captions, are correctly grouped by action category. Consequently, UMIP facilitates better multimodal alignment with the LLM compared to the original Q-former. The right column of Fig. 6 also provides an intuitive explanation for the performance improvement brought by UMIP on tasks that require a deeper understanding of language-based instructions and action categories, such as Action QA in Tab. 3.\n\nWe really appreciate the reviewer's detailed comments on our figure, which is truly a bit vague in this aspect. We will modify Section 4.4 for better clarification in the revised manuscript."}, {"review_id": "eT5GQLDCST", "reviewer": "Reviewer_mtFH", "summary": "The paper introduces HoloLLM Multimodal Large Language Model designed to understand human behavior by integrating unconventional sensing modalities with natural language. This model addresses the limitations of Vision-Language Models (VLMs), which struggle with poor lighting, occlusions, or privacy concerns.", "questions": "Also in Weaknesses. \nHow is the proposed method similar / different from LanguageBind[1]? How would the model perform by directly applying Languagebind [1] technique?\n\n[1] Zhu B, Lin B, Ning M, et al. Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment[J].", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength: \n* This paper tackles a novel task of integrating and working with unconventional sensors\n* Reasonable experiments demonstrate the effectiveness of the proposed method over existing works\n* The paper is clearly written and easy to follow\n\nWeakneses: \n* How is the proposed method similar / different from LanguageBind[1]? How would the model perform by directly applying Languagebind [1] technique?\n* The proposed method fails to improve performance at times.  For example, in the MM-Fi dataset, the single-modality vision model achieves 79.5% on Action QA, while the fused Vision+mmWave+WiFi model drops to 69.1% Which begs the usefulness of the proposed method and task. \n* The proposed method shows poor generalization and cross-subject testing performance. As the paper claims that the model enables \"seamless human perception and reasoning across heterogeneous environments\". This begs the feasibility of its real-world utility. \n\n\n[1] Zhu B, Lin B, Ning M, et al. Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment[J].", "was_revised": true, "final_justification": "Most of my concerns are addressed. I will keep my positive score", "rebuttal": "We sincerely appreciate the reviewer mtFH for the detailed review and suggestions. Here we add extensive experiments and answer all questions.\n\n**Q1 How is the proposed method similar/different from LanguageBind[16]? How would the model perform by directly applying Languagebind [16] technique?**\n\n**Answer:** We appreciate the suggestion to compare HoloLLM with the recent unified multimodal encoder, LanguageBind [16], and apologize for having overlooked this work during our literature review. Here, we illustrate the differences regarding the target task, model architecture, training strategy, and benchmark results among ImageBind [12], LanguageBind, and HoloLLM as follows.\n\n| |Task|Model Architecture|Training Strategy|Benchmark Results|\n|-|-|-|-|-|\n|HoloLLM (ours)|Align **data-scarce** but powerful **sensing modalities** (e.g., WiFi) with LLM using only limited data, while preserving their **heterogeneous features** for seamless human perception and reasoning.|A unified multimodal projector that considers **modality-specific features** by taking features from CLIP and **tailored encoders** as coarse queries and keys/values.|**Two-stage training.** (1) Pretrain tailored encoders to capture modality-specific features. (2) Fine-tune the UMIP to generate discriminative multimodal tokens aligned with text.|HoloLLM achieves SOTA performances on two **human sensing** datasets, establishing the first **multisensory benchmark** for human perception and reasoning.|\n|ImageBind [12]|Unify embeddings of **common modalities** (e.g., text, audio) by aligning them with images.|Transformer encoders for common modalities.|Fine-tune all transformer encoders via contrastive learning on **large-scale modality-image** pairs.|Only evaluated on **common modalities non-reasoning** benchmarks (e.g., classification, retrieval), such as MSR-VTT (video), NYU-D (depth).|\n|LanguageBind [16]|Unify embeddings of **common modalities** (e.g., image, audio) by aligning them with text.|Transformer encoders with LoRA fine-tuning for common modalities.|Fine-tune all transformer encoders via contrastive learning and LoRA on **large-scale modality-image** pairs.|Only evaluated on **common modalities non-reasoning**\n\nIn summary, the differences between our proposed HoloLLM and the LanguageBind are (1) deals with a more challenging problem on data-scarce yet powerful sensing modalities with heterogeneous features, (2) proposes a novel projector that incorporates fine-grained, modality-specific features for enhancing multimodal alignment, (3) presents a two-stage training strategy to sufficiently capture the modality-specific features and efficiently align multimodal inputs with the text space, and (4) establishes the first multisensory benchmark for human perception and reasoning.\n\nAs suggested, we evaluate LanguageBind. Similar to ImageBind, we treat it as a strong multimodal encoder, connect it to LLaMA2-7B, and fine-tune it on both MM-Fi and XRF55 same as HoloLLM. The results are summarized as follows (Top: MM-Fi, Bottom: XRF55).\n\n|Settings|Models|Sources|Types|Action QA| | | | | |Action Caption| | | | | |\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n| | | | |V|D|M|L|W|A|V|D|M|L|W|A|\n|Random|LanguageBind|ICLR’24|Enc|7.2|7.5|7.2|3.6|5.4|6.2|16.5|15.5|14.1|15.5|16.3|15.6|\n| |ImageBind|CVPR’23|Enc|89.3|76.7|45.8|11.1|8.0|46.2|28.4|21.2|18.8|16.2|14.8|19.9|\n| |HoloLLM|-|Proj|99.8|99.7|95.8|84.2|52.8|86.5|30.8|31.1|29.6|27.4|23.0|28.4|\n|Cross-Sub|LanguageBind|ICLR’24|Enc|6.8|6.1|6.5|3.3|4.9|5.5|14.9|15.2|11.1|15.1|16.1|14.5|\n| |ImageBind|CVPR’23|Enc|76.9|43.3|45.5|6.8|7.7|36.0|25.8|21.0|20.9|15.3|16.5|19.9|\n| |HoloLLM|-|Proj|98.0|98.9|88.0|66.5|8.0|71.9|30.6|30.5|29.5|24.9|16.7|26.4|\n|Cross-Env|LanguageBind|ICLR’24|Enc|6.7|5.9|4.1|6.3|5.2|5.6|16.2|15.4|5.7|5.7|12.3|11.1|\n| |ImageBind|CVPR’23|Enc|41.0|5.3|24.0|7.6|5.5|16.7|19.4|19.8|17.6|15.0|14.9|17.3|\n| |HoloLLM|-|Proj|79.5|91.6|61.4|41.4|8.2|56.4|25.7|27.5|24.5|19.6|15.9|22.6|\n\n\n|Settings|Models|Sources|Types|Action QA| | | | | |Action Caption| | | | | |\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n| | | | |V|D|I|R|W|A|V|D|I|R|W|A|\n|Random|LanguageBind|ICLR’24|Enc|27.7|10.9|38.1|2.5|5.7|17.0|13.8|15.2|12.3|13.6|14.7|13.9|\n| |ImageBind|CVPR’23|Enc|62.2|22.2|79.0|5.3|10.0|35.8|19.3|13.0|24.3|12.3|12.7|16.3|\n| |HoloLLM|-|Proj|94.5|92.3|92.6|27.1|11.2|63.5|34.2|34.8|34.7|15.5|14.0|26.6|\n|Cross-Sub|LanguageBind|ICLR’24|Enc|4.7|9.7|2.2|2.5|5.2|4.9|14.5|14.6|12.5|13.7|13.9|13.8|\n| |ImageBind|CVPR’23|Enc|13.3|11.1|20.4|3.8|4.9|10.7|15.8|14.6|18.3|12.7|14.5|15.2|\n| |HoloLLM|-|Proj|44.3|42.1|38.3|3.4|3.6|26.3|22.3|23.1|22.8|11.8|13.7|18.7|\n|Cross-Env|LanguageBind|ICLR’24|Enc|2.8|2.5|2.0|2.1|3.4|2.6|12.8|12.5|13.5|13.5|12.7|13.0|\n| |ImageBind|CVPR’23|Enc|4.7|4.9|16.9|2.8|2.6|6.4|13.1|14.3|16.9|12.7|12.0|13.8|\n| |HoloLLM|-|Proj|25.9|8.6|22.1|2.6|4.5|12.8|19.8|14.7|17.1|10.8|13.7|15.2|\n\nCompared to ImageBind, LanguageBind shows relatively lower performance. We deduce there are two main factors: (1) ImageBind employs a stronger vision encoder (CLIP ViT-H), and (2) it fully fine-tunes modality-specific encoders (e.g., depth), whereas LanguageBind only adopts CLIP ViT-L and applies LoRA for fine-tuning. We will incorporate the results into Tab 1 and Tab 2 in the final manuscript.\n\n**Q2: The proposed method fails to improve performance on Action QA with single/fusion setting. Illustrate the usefulness of the method and task.**\n\n**Answer:** We appreciate the comment regarding the multimodal fusion capabilities of HoloLLM. As mentioned, the results in Appendix C.3 show that naïve multimodal fusion (simply concatenating all multimodal tokens) fails to improve performance on action QA and captioning tasks. We conjecture that this is because naïve multimodal fusion does not fully leverage complementary information across modalities nor effectively reduce redundancy. Moreover, we add experiments to explore stronger adaptive fusion techniques, including weighted sum (WS) and max pooling (MP) across multimodal tokens.\n\n||MM-Fi||||XRF55|||\n|-|-|-|-|-|-|-|-|\n| |HAR|QA|Caption| |HAR|QA|Caption|\n|V|80.6|79.5|25.7|V|28.9|25.9|19.8|\n|M|61.0|61.4|24.5|I|28.3|22.1|17.1|\n|V+M (Naive)|**84.6**|66.4|25.0|V+I (Naive)|**34.3**|21.5|19.2|\n|V+M (WS)|82.6|**81.38**|**26.9**|V+I (WS)|30.91|**26.24**|**20.6**|\n|V+M (MP)|80.6|80.43|26.7|V+I (MP)|28.9|20.48|18.9|\n\nThe results show that the weighted sum fusion techniques outperform single-modality and naïve multimodal fusion on Action QA and captioning tasks. We believe that more advanced, learning-based multimodal fusion techniques could further enhance performance. We hope that HoloLLM can serve as a challenging benchmark to inspire the research community toward more insightful contributions in this multimodal fusion technique.\n\n**Q3: The proposed method shows poor generalization and cross-subject testing performance, which may hinder the real-world feasibility.**\n\n**Answer:** In fact, the cross-subject and cross-env generalization of these modalities (e.g., WiFi) is very challenging, as theoretically they are susceptible to variations in subjects and environments, and the data scarcity makes these tasks even harder. Though the tasks are hard, HoloLLM still outperforms other SOTA MLLMs in both settings by a large margin. We further conduct additional experiments to demonstrate the generalization capabilities of HoloLLM from two additional perspectives: new datasets and unseen sensing modalities:\n\n(1) Generalized to New Datasets.\n\nWe report zero-shot results on SenseFi [8]. Specifically, SenseFi performs human sensing using WiFi-CSI signals and includes six novel action categories. We then apply the ‘Baseline’ model (as detailed in line 248) and HoloLLM trained on the MM-Fi dataset under the ‘random’ setting to perform zero-shot human action QA on the official SenseFi test set, which contains 264 samples.\n\n|Model|Action QA|\n|-|-|\n|Baseline|5.43|\n|HoloLLM|16.67|\n\nIn zero-shot settings, HoloLLM’s superior performance demonstrates its strong generalization ability and discriminability.\n\n(2) Generalization to Unseen Sensing Modalities\n\nHoloLLM can integrate new modalities more efficiently than other MLLMs, owing to the novel architecture of UMIP. Specifically, we extensively pretrain a lightweight, modality-specific, tailored encoder for each new modality to sufficiently capture fine-grained features. The new modality is then efficiently integrated into HoloLLM by UMIP with minor data and fine-tuning.\n\nWe add an experiment to demonstrate the superior data efficiency of HoloLLM compared to the ‘Baseline’ model (as detailed in line 248). Specifically, we evaluate two novel modalities, audio [9] and UWB [10], with the results summarized below.\n\n| |HAR| |Action QA| |\n|-|:-:|:-:|:-:|:-:|\n| |Audio|UWB|Audio|UWB|\n|Baseline|37.71|9.46|30.73|11.33|\n|HoloLLM|63.94|16.77|54.17|16.45|\n\nCompared to the ‘Baseline’ model, HoloLLM can be efficiently generalized to novel modalities with minor data and fine-tuning.\n\nIn future work, we aim to enhance the generalization ability of HoloLLM by constructing a large-scale sensing dataset. Specifically,  we plan to integrate existing public datasets [1,2] and collect additional data from a broader range of environments (e.g., outdoor) and tasks (e.g., task planning and agent action generation). By fine-tuning HoloLLM with this new dataset, we believe that HoloLLMv2 will exhibit significantly improved cross-subject and cross-environment generalization. This work focuses on aligning new yet powerful sensing modalities with LLM, given limited data and various modalities, and thus offers a novel technical solution with a comprehensive benchmark. Regarding the real-world feasibility, we believe introducing complementary modalities is critical for embodied AI agents. HoloLLM denotes a baseline to engage the research community in further advancing MLLMs with data-scarce data modality. \n\n**References**\n\nDue to limited characters, please refer to the responses of Reviewer 4VBc for references."}, {"review_id": "guVA6grew6", "reviewer": "Reviewer_4VBc", "summary": "This paper presents HoloLLM, a MLLM that takes multiple modalities for human activity recognition (and QA, captioning). The key technical components of this paper are: (1) modality-tailored (implemented as resnet of pointnet) encoder, and (2) UMIP, which uses CLIP vision encoder to iteratively perform cross-attention to refine the embedding. The authors also curated two multi-modality datasets, MM-Fi and XRF55, for sensory modality-text alignment, by generating captions or QAs according to the vision modality contained in the dataset.", "questions": "Please see weaknesses above.", "limitations": "Please see weaknesses above.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. HoloLLM achieved significantly better performance on the two datasets the authors curated.\n2. The motivation is clearly stated. And the paper is well organized.\n3. The authors contributed two curated datasets for sensory modality-text alignment experiments.\n\nWeaknesses:\n1. The technical contribution seems incremental, exhibiting limited originality or theoretical soundness. The ablation study showed that UMIP plays a less important role in the system, while most of the performance improvements are achieved by the tailored encoder, which is a resnet or pointnet encoder trained for each modality.\n2. The contribution problem is amplified by the fact that the paper primarily focused on aligning single modality to text, with only preliminary multi-modal fusion results presented in appendix. As explained by the authors, since they only considered naive modality fusion methods, the multi-modal fusion results are only marginally better than single-model results. The authors may need to consider may more attention to multi-modal cases, given their motivation of \"enable seamless human perception and reasoning across heterogeneous environments\"", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer 4VBc for insightful questions and suggestions. We answer all the questions by extensive empirical results and elaborations.\n\n**Q1: The technical contribution seems incremental, exhibiting limited originality or theoretical soundness.**\n\n**Answer:** We understand the reviewer may evaluate our work from the perspective of VLMs research community. However, we argue that HoloLLM bridges an underexplored gap in MLLM: how to introduce new rare modalities to enhance MLLM's capability for physical agents. HoloLLM’s target task, model architecture, training strategy, and benchmark results differ substantially from existing SOTA MLLMs. To this end, we provide a detailed comparison in the following table. As discussed in section 4.2, the Encoder-based methods (ImageBind [12], LanguageBind [16]) aim to unify multimodal embeddings, while the Projector-based methods (BLIP-2 [11], OneLLM [7]) focus on designing effective projectors.\n\n| |Task|Model Architecture|Training Strategy|Benchmark Results|\n|-|-|-|-|-|\n|HoloLLM (ours)|Align **data-scarce** but powerful **sensing modalities** (e.g., WiFi) with LLM using only limited data, while preserving their **heterogeneous features** for seamless human perception and reasoning.|A unified multimodal projector that considers **modality-specific features** by taking features from CLIP and **tailored encoders** as coarse queries and keys/values.|**Two-stage training.** (1) Pretrain tailored encoders to capture modality-specific features. (2) Fine-tune the UMIP to generate discriminative multimodal tokens aligned with text.|HoloLLM achieves SOTA performances on two **human sensing** datasets, establishing the first **multisensory benchmark** for human perception and reasoning.|\n|BLIP-2 [11]|**Image-text** alignment for visual-language tasks (e.g., image caption).|Original Q-former with **learnable queries** to integrate features from the vision encoder.|Pretraining projectors via **large-scale image-text** pairs.|Only evaluated on **image-language** benchmarks, such as VQAv2|\n|OneLLM [7]|Align **common modalities** (e.g., image, audio) with text for multimodal caption and QA tasks.|Q-former-style projector with multimodal **learnable queries** to integrate features from a unified encoder.| Pretraining projectors and further fine-tuning the whole LLM via **large-scale modality-text** pairs.|Only evaluated on **common modalities** benchmarks, such as VQAv2 (image), NextQA (video).|\n|ImageBind [12]|Unify embeddings of **common modalities** (e.g., text, audio) by aligning them with images.|Transformer encoders for common modalities.|Fine-tune all transformer encoders via contrastive learning on **large-scale modality-image** pairs.|Only evaluated on **common modalities non-reasoning** benchmarks (e.g., classification, retrieval), such as MSR-VTT (video), NYU-D (depth).|\n|LanguageBind [16]|Unify embeddings of **common modalities** (e.g., image, audio) by aligning them with text.|Transformer encoders with LoRA fine-tuning for common modalities.|Fine-tune all transformer encoders via contrastive learning and LoRA on **large-scale modality-image** pairs.|Only evaluated on **common modalities non-reasoning** benchmarks (e.g., classification, retrieval), such as MSR-VTT (video), AS-A (audio).|\n\nIn summary, our proposed HoloLLM (1) deals with a more challenging and unexplored problem on data-scarce yet useful sensing modalities with heterogeneous features (**also mentioned by reviewers DvFs, 78xg, and mtFH**), (2) proposes a novel projector that incorporates modality-specific features for enhanced multimodal alignment (**also mentioned by reviewers DvFs and 78xg**), (3) presents a two-stage training strategy to sufficiently capture the modality-specific features and efficiently align multimodal inputs with the text space (**also mentioned by reviewer DvFs**), and (4) establishes the first multisensory benchmark for human perception and reasoning (**mentioned by all four reviewers**). \n\n**Q2: The ablation study showed that UMIP plays a less important role in the system, while most of the performance improvements are achieved by the tailored encoder.**\n\n**Answer:** We argue that UMIP is still critical given the two reasons:\n\n(1)   Significant improvement on the Action QA task (up to **9.8%** on MM-Fi).\n\nUMIP contributes to better aligning multimodal tokens with the text while sufficiently preserving discriminative, modality-specific features. This results in a substantial performance gain on Action QA tasks—up to 9.8% on MM-Fi, as shown in Tab. 3— which demand a deeper understanding of language-based human instructions and action categories. Additionally, Fig. 6 (b) provides an intuitive example of enhanced multimodal-text alignment, where multimodal tokens generated by UMIP and text tokens from ground-truth captions are correctly grouped by action category.\n\n(2) Additional experiments show that UMIP enhances the generalization capability of HoloLLM.\n\nUMIP adopts a **novel decoupled architecture**, where CLIP generates coarse queries and tailored encoders extract modality-specific features as keys and values. It enables HoloLLM to achieve improved zero-shot generalization and efficient adaptation to new modalities, as demonstrated by two additional experiments.\n\n(2a) Generalized to New Datasets.\n\nWe report zero-shot results on SenseFi [8]. Specifically, SenseFi performs human sensing using WiFi-CSI signals and includes six novel action categories. We then apply the ‘Baseline’ model (as detailed in line 248) and HoloLLM trained on the MM-Fi dataset under the ‘random’ setting to perform zero-shot human action QA on the official SenseFi test set, which contains 264 samples.\n\n|Model|Action QA|\n|-|-|\n|Baseline|5.43|\n|HoloLLM|16.67|\n\nIn zero-shot settings, HoloLLM can achieve better performance. The decoupled architecture of UMIP effectively leverages CLIP’s generalization capability—acquired through extensive pretraining—to generate coarse queries that generalize better than learnable queries.\n\n(2b) Generalization to Unseen Sensing Modalities\n\nHoloLLM can integrate new modalities more efficiently than other MLLMs, owing to the decoupled architecture of UMIP. Specifically, we extensively pretrain a lightweight, modality-specific, tailored encoder for each new modality to sufficiently capture fine-grained features. The new modality is then efficiently integrated into HoloLLM by UMIP with minor data and fine-tuning.\n\nWe add an experiment to demonstrate the superior data efficiency of HoloLLM compared to the ‘Baseline’ model (as detailed in line 248). Specifically, we evaluate two novel modalities, audio [9] and UWB [10], with the results summarized below.\n\n| |HAR| |Action QA| |\n|-|:-:|:-:|:-:|:-:|\n| |Audio|UWB|Audio|UWB|\n|Baseline|37.71|9.46|30.73|11.33|\n|HoloLLM|63.94|16.77|54.17|16.45|\n\nCompared to the ‘Baseline’ model, HoloLLM can be efficiently generalized to novel modalities with minor data and fine-tuning.\n\n**Q3: The contribution problem is amplified by the fact that the paper primarily focused on aligning single modality to text. The authors may need to consider may more attention to multi-modal cases.**\n\n**Answer:** We appreciate the suggestion and believe this should be further illustrated. We deduce that the marginal improvement of naïve multimodal fusion arises from its failure to fully exploit complementary cross-modal information and reduce redundancy. As suggested, we consider more multi-modal fusion cases other than naive fusion (simply concatenate all multimodal tokens), including weighted sum (WS) and max pooling (MP) across multimodal tokens. The results of additional experiments are listed as follows.\n\n||MM-Fi||||XRF55|||\n|-|-|-|-|-|-|-|-|\n| |HAR|QA|Caption| |HAR|QA|Caption|\n|V|80.6|79.5|25.7|V|28.9|25.9|19.8|\n|M|61.0|61.4|24.5|I|28.3|22.1|17.1|\n|V+M (Naive)|**84.6**|66.4|25.0|V+I (Naive)|**34.3**|21.5|19.2|\n|V+M (WS)|82.6|**81.38**|**26.9**|V+I (WS)|30.91|**26.24**|**20.6**|\n|V+M (MP)|80.6|80.43|26.7|V+I (MP)|28.9|20.48|18.9|\n\nThe results show that the weighted sum fusion techniques outperform single-modality and naïve multimodal fusion on Action QA and captioning tasks. We believe that more advanced, learning-based multimodal fusion techniques could further enhance performance. We hope that HoloLLM can serve as a challenging benchmark to inspire the research community toward more insightful contributions in this multimodal fusion technique.\n\n**References**\n\n[1] XRF V2: A Dataset for Action Summarization with Wi-Fi Signals, and IMUs in Phones, Watches, Earbuds, and Glasses. (ACM IMWUT 2025)\n\n[2] Mm-fi: Multi-modal non-intrusive 4d human dataset for versatile wireless sensing. (NeurIPS 2023)\n\n[3] Domain‑Adversarial Training of Neural Networks. (JMLR 2016)\n\n[4] Meta-Transformer: A Unified Framework for Multimodal Learning. (arXiv 2023)\n\n[5] Chain‑of‑Thought Prompting Elicits Reasoning in Large Language Models. (NeurIPS 2022)\n\n[6] Measuring Nominal Scale Agreement among Many Raters. (Psychological Bulletin 1971)\n\n[7] OneLLM: One Framework to Align All Modalities with Language (CVPR 2024)\n\n[8] SenseFi: A library and benchmark on deep-learning-empowered WiFi human sensing (Patterns 2023)\n\n[9] SAMoSA: Sensing Activities with Motion and Subsampled Audio (ACM IMWUT 2022)\n\n[10] SleepPoseNet: Multi-view learning for sleep postural transition recognition using UWB (IJBHI 2020)\n\n[11] BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models (ICML 2023)\n\n[12] Imagebind: One embedding space to bind them all (CVPR 2023)\n\n[13] Omnifusion: 360 monocular depth estimation via geometry-aware fusion (CVPR 2022)\n\n[14] UNIFIED-IO: A Unified Model for Vision, Language, and Multi-modal Tasks (ICLR 2023)\n\n[15] XRF55: A Radio Frequency Dataset for Human Indoor Action Analysis. (ACM IMWUT 2024)\n\n[16] LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment (ICLR 2024)"}]

Author rebuttal:

Meta-review: Summary:
The paper presents HoloLLM, an MLLM that integrates rare sensing modalities to enable seamless human perception and reasoning across heterogeneous real-world scenarios. To efficiently align sensing modalities with text using minimal fine-tuning, the paper proposes the Universal Modality Injection Projector (UMIP) along with modality-specific discriminative features.

Key strengths highlighted by the reviewers:
(i) The paper addresses an important and under-explored problem (i.e., unconventional modalities).
(ii) To formulate the new task, the authors curated new datasets and proposed a comprehensive benchmark, which will hopefully support future research.
(iii) Experiments demonstrate the effectiveness of the proposed method.
(iv) The motivation is clearly stated, and the paper is well organized.
(v) The proposed UMIP is a simple yet effective architecture.

Key weaknesses remaining after the rebuttal:
(i) The model’s cross-environment performance remains limited.
(ii) Novelty is moderate.
(iii) The distinction between the proposed contributions and existing approaches such as LanguageBind needs to be clarified in the final draft.
(iv) The paper primarily focuses on aligning a single modality to text; additional experiments on multi-modal fusion (as presented in the rebuttal) should be included in the final draft.

Decision summary:
The paper received one Accept, two Borderline Accepts, and one Borderline Reject. The authors provided additional results and explanations in the rebuttal, and reviewers agreed that most of their concerns were addressed. Although the architectural novelty of the method is limited, the area chair believes that the paper tackles an important and under-explored problem. The proposed dataset and models are expected to be a valuable contribution to the community; hence, the recommendation is to accept the paper. The authors are expected to address the remaining weaknesses in the final draft, include additional explanations and results reported in the rebuttal to the final draft or appendix,  and open-source their code and data as stated in the original submission.
Decision: Accept (poster)