Title: HoloLLM: Multisensory Foundation Model for Language-Grounded Human Sensing and Reasoning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multimodal Large Language Model

Human Sensing and Reasoning

IoT Sensors
Abstract: Embodied agents operating in smart homes must understand human behavior through diverse sensory inputs and communicate via natural language. While Vision-Language Models (VLMs) have enabled impressive language-grounded perception, their reliance on visual data limits robustness in real-world scenarios with occlusions, poor lighting, or privacy constraints. In this paper, we introduce HoloLLM, a Multimodal Large Language Model (MLLM) that integrates uncommon but powerful sensing modalities, such as LiDAR, infrared, mmWave radar, and WiFi, to enable seamless human perception and reasoning across heterogeneous environments. We address two key challenges: (1) the scarcity of aligned modality-text data for rare sensors, and (2) the heterogeneity of their physical signal representations. To overcome these, we design a Universal Modality-Injection Projector (UMIP) that enhances pre-aligned modality embeddings with fine-grained, text-aligned features from tailored encoders via coarse-to-fine cross-attention without introducing significant alignment overhead. We further introduce a human-VLM collaborative data curation pipeline to generate paired textual annotations for sensing datasets. Extensive experiments on two newly constructed benchmarks show that HoloLLM significantly outperforms existing MLLMs, improving language-grounded human sensing accuracy by up to 30\%. This work establishes a new foundation for real-world, language-informed multisensory embodied intelligence.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents HoloLLM, a MLLM that takes multiple modalities for human activity recognition (and QA, captioning). The key technical components of this paper are: (1) modality-tailored (implemented as resnet of pointnet) encoder, and (2) UMIP, which uses CLIP vision encoder to iteratively perform cross-attention to refine the embedding. The authors also curated two multi-modality datasets, MM-Fi and XRF55, for sensory modality-text alignment, by generating captions or QAs according to the vision modality contained in the dataset.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. HoloLLM achieved significantly better performance on the two datasets the authors curated.\n2. The motivation is clearly stated. And the paper is well organized.\n3. The authors contributed two curated datasets for sensory modality-text alignment experiments.\n\nWeaknesses:\n1. The technical contribution seems incremental, exhibiting limited originality or theoretical soundness. The ablation study showed that UMIP plays a less important role in the system, while most of the performance improvements are achieved by the tailored encoder, which is a resnet or pointnet encoder trained for each modality.\n2. The contribution problem is amplified by the fact that the paper primarily focused on aligning single modality to text, with only preliminary multi-modal fusion results presented in appendix. As explained by the authors, since they only considered naive modality fusion methods, the multi-modal fusion results are only marginally better than single-model results. The authors may need to consider may more attention to multi-modal cases, given their motivation of \"enable seamless human perception and reasoning across heterogeneous environments\"\n\nQUESTIONS:\nPlease see weaknesses above.\n\nLIMITATIONS:\nPlease see weaknesses above.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-09"}

comments: SUMMARY:
The paper introduces HoloLLM Multimodal Large Language Model designed to understand human behavior by integrating unconventional sensing modalities with natural language. This model addresses the limitations of Vision-Language Models (VLMs), which struggle with poor lighting, occlusions, or privacy concerns.

STRENGTHS AND WEAKNESSES:
Strength: 
* This paper tackles a novel task of integrating and working with unconventional sensors
* Reasonable experiments demonstrate the effectiveness of the proposed method over existing works
* The paper is clearly written and easy to follow

Weakneses: 
* How is the proposed method similar / different from LanguageBind[1]? How would the model perform by directly applying Languagebind [1] technique?
* The proposed method fails to improve performance at times.  For example, in the MM-Fi dataset, the single-modality vision model achieves 79.5% on Action QA, while the fused Vision+mmWave+WiFi model drops to 69.1% Which begs the usefulness of the proposed method and task. 
* The proposed method shows poor generalization and cross-subject testing performance. As the paper claims that the model enables "seamless human perception and reasoning across heterogeneous environments". This begs the feasibility of its real-world utility. 


[1] Zhu B, Lin B, Ning M, et al. Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment[J].

QUESTIONS:
Also in Weaknesses. 
How is the proposed method similar / different from LanguageBind[1]? How would the model perform by directly applying Languagebind [1] technique?

[1] Zhu B, Lin B, Ning M, et al. Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment[J].

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces HoloLLM, a multimodal large language model (MLLM) designed to integrate a range of sensing modalities beyond vision, for typical home environments. One contribution is UMIP, a framework that unifies pre-aligned modalities with text modality. The authors also propose a human–VLM collaborative pipeline and use it to construct two new benchmarks that facilitate language grounded human sensing. Experimental results on these benchmarks demonstrate that HoloLLM outperforms previous state-of-the-art models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths. \n\n- Exploring MLLMs that extend beyond vision to incorporate additional modalities available from embodied agents (e.g., robots) is an important yet underexplored area.\n\n- This paper proposes UMIP, a simple yet effective architecture that integrates modality-specific encoders to address challenges in data-scarcity modality-to-text alignment.\n\n-  To support evaluation, the authors introduce two new benchmarks for language-grounded human sensing, built by extending existing datasets.\n\nWeaknesses\n\n-\tWhat are the key architectural or methodological differences between HoloLLM and the baseline approaches it was compared against? HoloLLM shows significantly improved performance, often by large margins. It would be helpful to clarify whether this improvement is primarily due to its architectural design (e.g., UMIP) or differences in training data. Specifically, were the modality-specific encoders in the baseline models trained using the same data as HoloLLM, or were there discrepancies in training conditions that could account for the performance gap?\n\n-\tIn Table 1 and Table 3, all different modalities are reported separately. What is the performance by integrating these modalities together? \n\n-\tWhat is the inference time latency?\n\nQUESTIONS:\n1) see weaknesses\n2) what message does Figure 6 right column intend to convey?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents HoloLLM, a large language model that understands human actions using not just visual data, but also other sensor types like LiDAR, infrared, WiFi, and mmWave radar. These sensors work better than cameras in dark, blocked, or privacy-sensitive places.\n\nSince these sensor types don’t have much paired text data and are very different from images, the authors design two solutions: (1) Tailored encoders to extract useful features for each sensor type. (2) UMIP, a module that combines those features with CLIP embeddings and gradually aligns them to the language model.\n\nThey also build a new dataset with human + AI collaboration for tasks like action recognition, question answering, and captioning. Experiments show that HoloLLM works much better than previous models (up to 30% better) on these tasks.\n\nSTRENGTHS AND WEAKNESSES:\nQuality: (1) Strengths: The technical methodology is sound and well-motivated. The UMIP design is carefully engineered to address the unique challenges of data scarcity and sensor heterogeneity. The experiments are broad, covering five sensing modalities across two datasets and three settings (Random, CrossSub, CrossEnv). Ablation studies are helpful in validating both the tailored encoders and UMIP. (2) Weaknesses: The empirical rigor could be improved: Some accuracy (e.g., CrossEnv) are small and need stronger design support. The validation of the human-VLM collaborative pipeline is weak—there is no measure of annotation quality or consistency. No comparison with multimodal transformers trained end-to-end or with stronger adaptive fusion techniques beyond naive fusion.\n\nClarity: (1) Strengths: The paper is logically organized and generally easy to follow at a high level. Motivation is concrete (smart homes, privacy, occlusion), and the contribution is clearly stated. (2) Weaknesses: Key sections, especially Section 3.2 (UMIP), are too dense, please consider adding pseudocode or a simplified diagram of the iterative process. The link between CLIP embeddings and non-visual modality alignment needs clearer explanation.\n\nSignificance: (1) Strengths: The paper addresses an important and under-explored problem: language grounding with non-visual sensing. (2) Weaknesses: Broader generalization across datasets or unseen modalities is not demonstrated.\n\nOriginality: (1) Strengths:The two-stage training strategy and the human-VLM collaborative annotation pipeline are thoughtful and well-justified. (2) Weaknesses: The idea of cross-attention-based fusion is not new—more emphasis is needed on what differentiates UMIP from prior Q-former-style projectors (e.g., OneLLM, BLIP-2). (2) Need to better position UMIP versus recent work on universal modality encoders (e.g., ImageBind, OmniFusion, UnifiedIO). (3) The use of CLIP for non-visual modality embedding needs stronger justification, or at least empirical support.\n\nQUESTIONS:\n1. Your human-VLM data curation pipeline is insteresting, but how do you ensure annotation quality—especially for captions generated by LLaVA-Video? Could you report: Inter-annotator agreement (if any human review was done); Human evaluation of generated captions (e.g., fluency, accuracy scores); Percentage of samples verified or discarded. Addressing this would increase trust in your benchmarks, which are key to evaluating your model. Without quality control, the evaluation may be biased or noisy.\n\n2. HoloLLM is framed as a general multisensory foundation model, yet it is only evaluated on MM-Fi and XRF55, which have similar indoor settings and overlapping sensor types. How well does HoloLLM generalize to new domains, sensors, or unseen environments? If feasible, please include zero-shot or few-shot results on other datasets, such as SenseFi or Widar.\n\nLIMITATIONS:\nThe authors include a brief limitations section noting that HoloLLM is currently limited to human action understanding tasks (recognition, QA, captioning), and does not yet handle higher-level capabilities like task planning or real-world agent control. This is a good start.  Technical limitations are not fully addressed because the model is only evaluated on two similar indoor datasets.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}