Title: SensorLM: Learning the Language of Wearable Sensors
Venue: NeurIPS 2025
Accepted: yes
Keywords: Sensor-Language Models

Wearable Sensors

Foundation Models

Multimodal Learning

Healthcare
Abstract: We present SensorLM, a family of sensor-language foundation models that enable wearable sensor data understanding with natural language. Despite its pervasive nature, aligning and interpreting sensor data with language remains challenging due to the lack of paired, richly annotated sensor-text descriptions in uncurated, real-world wearable data. We introduce a hierarchical caption generation pipeline designed to capture statistical, structural, and semantic information from sensor data. This approach enabled the curation of the largest sensor-language dataset to date, comprising over 59.7 million hours of data from more than 103,000 people. Furthermore, SensorLM extends prominent multimodal pretraining architectures (e.g., CLIP, CoCa) and recovers them as specific variants within a generic architecture. Extensive experiments on real-world tasks in human activity analysis and healthcare verify the superior performance of SensorLM over state-of-the-art in zero-shot recognition, few-shot learning, and cross-modal retrieval. SensorLM also demonstrates intriguing capabilities including scaling behaviors, label efficiency, sensor captioning, and zero-shot generalization to unseen tasks. Code is available at https://github.com/Google-Health/consumer-health-research/tree/main/sensorlm.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces SensorLM, a family of sensor-language foundation models that learn from large-scale wearable sensor data paired with automatically generated textual descriptions. The authors propose a hierarchical captioning pipeline (statistical, structural, semantic) to describe over 59 million hours of multimodal data collected from more than 100,000 individuals, enabling alignment with natural language for downstream tasks. The model integrates contrastive and generative objectives, extending prior multimodal architectures (e.g., CLIP, CoCa) to the sensor-text domain. Extensive evaluations demonstrate SensorLM's superiority in zero-shot recognition, few-shot learning, and cross-modal retrieval, making it a compelling step toward interpretable and generalizable sensor understanding.\n\nSTRENGTHS AND WEAKNESSES:\n### [Strengths]\n1. The hierarchical captioning pipeline—comprising statistical, structural, and semantic levels—offers a unique way to supervise time-series signals with interpretable textual labels.\n2. SensorLM demonstrates strong empirical results across a wide range of datasets, tasks, and architectures, showing robustness and generalization beyond conventional sensor models.\n3. The method supports test-time inference without requiring fine-tuning or task-specific labels, which is particularly valuable for real-world deployment in personalized or low-resource settings.\n4. The authors provide a comprehensive evaluation that includes classification, retrieval, and metabolic prediction tasks, highlighting the model's versatility across modalities and objectives.\n\n### [Weaknesses]\n1. The quality of the pretraining data relies on heuristically generated captions, which may introduce noise or bias that affects the downstream alignment performance.\n2. The model is currently evaluated only on wearable sensor data, and its applicability to other time-series domains—such as industrial or medical sensors—remains untested.\n\nQUESTIONS:\n1. Have the authors evaluated SensorLM’s generalization ability on non-wearable time-series data, such as medical monitoring signals or industrial sensors?\n2. What is the computational overhead of SensorLM at inference time compared to conventional sensor models, especially in edge or real-time environments?\n3. Can the hierarchical captioning pipeline be adapted or extended to support human-written descriptions or other forms of supervision beyond rule-based heuristics?\n4. How robust is the model when some sensor modalities are missing, noisy, or partially corrupted at inference time?\n\nLIMITATIONS:\nPlease see above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new sensor data captioning strategy, which has three hierarchies: statistical, structural, and semantic captions. Then, the paper presents SensorLM, aiming to generate captions based on sensor data. SensorLM is a hybrid method combining contrastive alignment and text generation. Experiments demonstrate that SensorLM is superior to baselines methods on diverse downstream tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper is overall well written and organized. It is easy to follow the idea.\n2. Building large datasets is a fundamental part of the community and would largely inspire the following research.\n3. The idea of hierarchical captioning is interesting and meaningful. \n\nWeaknesses:\n1. The main contribution of this work is the sensor-text dataset with hierarchical captions. However, the authors do not publish the dataset. Although the data collection details are presented. Datasets are different from models, which are hard and time-consuming to reproduce. Thus, it is critical to publish the dataset.\n2. The quality of the dataset is not studied. It would be better if the authors could show how existing studies work on the new dataset and how the hierarchical captions impact the performance. \n3. The novelty of the SensorLM model is limited. It combines generative and contrastive learning, which is not new. In addition, the authors use $\\lambda_{con}=1$ and $\\lambda_{cap}=1$ as the default setting, which is CoCa actually. The paper fails to show the benefits of tuning the loss weights $\\lambda$.\n4. In Tables 2 and 3, the results of the baselines are nearly random. It is quite unfair to make such a comparison. The two baselines should be fine-tuned using the proposed dataset first, then make inferences and comparisons.\n5. The paper would be more suitable for the dataset track instead of the research track.\n\nQUESTIONS:\nWhat does it mean by \"zero-shot\" in your context? Does it mean the test classes are unseen, or the test datasets are collected separately from your training set?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a way to teach foundation models the “language” of wrist-worn sensors. The authors build a hierarchical captioning pipeline that turns each 24-hour, 26-feature sensor window into statistical, structural and semantic sentences. They train transformer-based dual-encoder + decoder models with combined contrastive-and-generative objectives, unifying CLIP, Cap and CoCa styles in their framework. In zero-shot activity recognition, cross-modal retrieval and few-shot health prediction, SensorLM beats large LLM-tabular baselines and sensor encoders (e.g., DINO, MSN, etc) by sizable margins.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n+ The paper delivers a substantial engineering effort for wrist-sensor data. Combined contrastive + generative training unifies CLIP-, caption- and CoCa-style objectives and is backed by ablation and scaling-law studies. Experiments cover zero-shot, few-shot and linear-probe settings and show clear gains over several baselines.\n+ The pipeline is illustrated step-by-step; model architecture and objectives are mathematically specified; Other details are moved to a well-structured appendix.\n\nWeaknesses:\n\n- All downstream tasks rely on a fixed 26 features. The framework is hard-coded to those 26 features from a single wrist device. It is unclear how it adapts when tasks demand additional statistics or when multiple sensors each supply their own 26-D vectors.\n- Zero-shot evaluation is compared only with Gemini/Gemma (LLM-on-tabular). Strong raw-data time-series foundation models capable of zero-shot representation learning (e.g., Mantis, Moment, Chronos, UniMTS, etc) are omitted, making claims of superiority less convincing.\n- Training and test sets come from the same user cohort and device (Section 5), yet the paper emphasises “generalisation”. A leave-person-out (cross-device) split is required to demonstrate genuine robustness.\n- The architecture largely reuses established image-text paradigms (ITC, CoCa) with minimal conceptual change.\n\nQUESTIONS:\nPlease refer to the weaknesses part.\n\nLIMITATIONS:\nI think some further discussions about fixed feature dependency would be good.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work constructs a large-scale multimodal sensory dataset and presents a well-structured sensor-data–caption pretraining pipeline that generalizes effectively across multiple downstream tasks. It directly addresses the challenge of data scarcity, particularly the lack of aligned sensor data, in this domain. The pretrained model demonstrates strong performance across a range of evaluation settings.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n+ The scale of this work is large, making a good contribution to the sensory data learning field, which suffers from data scarcity and lack of aligned data.\n\n+ The paper addresses the lack of interpretability of sensory data and forms the captions into structured and interpretable formats. The hierarchical structure is interesting.\n\n+ The evaluation is extensive and convincing compared to many advanced pipelines.\n\nWeaknesses:\n\n- My primary concern is whether the proposed learning pipeline and pretrained model can generalize to out-of-domain data. While the evaluation is extensive, all experiments are conducted on the authors’ collected datasets. In practice, users may employ different device types or encounter variations in data formats. Given the inherent heterogeneity in devices and environments in sensory data applications, it is unclear whether the framework can effectively interpret preprocessed or out-of-distribution data without performance degradation.\n\n- The alignment strategy combining contrastive and generative losses is effective for joint sensor-language modeling. However, enforcing all modalities into a shared latent space may compromise the fidelity of modality-specific representations, especially for time-series sensor data. A more fine-grained approach that applies alignment at different levels based on context could improve performance. In the current design, modality alignment appears uniform and context-agnostic.\n\n- It is unclear whether SensorLM can handle missing modality scenarios or adapt to varying modality combinations. In real-world applications, users may choose not to wear certain sensors due to personal preference, battery constraints, or hardware availability. The model’s flexibility in such cases should be discussed or evaluated.\n\n- The paper briefly mentions a hierarchical design for captions but lacks sufficient detail on its structure. It would be helpful to elaborate on how the statistical, structural, and semantic levels are defined and how the model captures both shared and modality-specific information across these layers.\n\n- Given the large number of participants involved in data collection, more details on the labeling process, both for hard labels and soft labels, would be valuable. For smaller research groups looking to adopt this approach, practical guidance on data annotation would improve reproducibility and usability.\n\n- In many real-world scenarios, fully aligned multimodal sensory data is not available. It would strengthen the work to discuss whether and how the proposed approach can be extended to handle unaligned or weakly aligned sensory inputs.\n\nQUESTIONS:\n1. Can the proposed learning pipeline or pretrained model generalize to out-of-domain data?\n\n2. Does the modality alignment strategy risk weakening modality-specific representations?\n\n3. Can SensorLM handle scenarios with missing modalities or adapt to varying modality combinations?\n\n4. Could the authors provide more details on the design and implementation of the “hierarchical” caption structure?\n\n5. Is the proposed approach extendable to unaligned multimodal data?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-10"}