SensorLM: Learning the Language of Wearable Sensors
Authors: Yuwei Zhang
Kumar Ayush
Siyuan Qiao
A. Ali Heydari
Girish Narayanswamy
Maxwell A Xu
Ahmed Metwally
Jinhua Xu
Jake Garrison
Xuhai Xu
Tim Althoff
Yun Liu
Pushmeet Kohli
Jiening Zhan
Mark Malhotra
Shwetak Patel
Cecilia Mascolo
Xin Liu
Daniel McDuff
Yuzhe Yang
Venue: NeurIPS 2025 poster ()
Keywords: Sensor-Language Models
Wearable Sensors
Foundation Models
Multimodal Learning
Healthcare
Primary area: applications

Abstract: We present SensorLM, a family of sensor-language foundation models that enable wearable sensor data understanding with natural language. Despite its pervasive nature, aligning and interpreting sensor data with language remains challenging due to the lack of paired, richly annotated sensor-text descriptions in uncurated, real-world wearable data. We introduce a hierarchical caption generation pipeline designed to capture statistical, structural, and semantic information from sensor data. This approach enabled the curation of the largest sensor-language dataset to date, comprising over 59.7 million hours of data from more than 103,000 people. Furthermore, SensorLM extends prominent multimodal pretraining architectures (e.g., CLIP, CoCa) and recovers them as specific variants within a generic architecture. Extensive experiments on real-world tasks in human activity analysis and healthcare verify the superior performance of SensorLM over state-of-the-art in zero-shot recognition, few-shot learning, and cross-modal retrieval. SensorLM also demonstrates intriguing capabilities including scaling behaviors, label efficiency, sensor captioning, and zero-shot generalization to unseen tasks. Code is available at https://github.com/Google-Health/consumer-health-research/tree/main/sensorlm.

SensorLM: Learning the Language of Wearable Sensors
Yuwei Zhang 13:t. Kumar Ayush '. Siyuan Qiao² A. Ali Heydari'
Girish Narayanswamy Xuhai Xu' Tim Althoff'
Maxwell A. Xul Ahmed A. Metwally'
Shawn Xu
Jake Garrison'
Yun Liu?
Pushmeet Kohli²
Jiening Zhan'
Mark Malhotra' Shwetak Patel
Cecilia Mascolo Xin Liu' Daniel McDuff Yuzhe Yang!.41
Google Research "Gogle DeepMind 3 University of Cambridge *University of California, Los Angeles
Abstract
We present SensorLM, family of sensor-language foundation models that enable wearable sensor data understanding with natural language. Despite its pervasive nature, aligning and interpreting sensor data with language remains challenging due to the lack of paired, richly annotated sensor-text descriptions in uncurated, real-world wearable data. We introduce hierarchical caption generation pipeline designed to capture statistical, structural, and semantic information from sensor data. This approach enabled the curation of the largest sensor-language dataset date, comprising over 59.7 million hours of data from more than 103,000 people. Furthermore, SensorLM extends prominent multimodal pretraining architectures (e.g., CLIP, CoCa) and recovers them as specific variants within generic archi- tecture. Extensive experiments on real-world tasks in human activity analysis and healthcare verify the superior performance of SensorLM over state-of-the-ard in zero-shot recognition, few-shot learning, and cross-modal retrieval. SensorLM also demonstrates intriguing capabilities including scaling behaviors, label efficiency, sensor captioning, and zero-shot generalization to unseen tasks. Code is available https://giiubb.com//ooog wtt.e////ooooooooaa/oo///ooaaaea secaltt-rsccarcc/rre/mmii////s/ htnps//c/mmmii//ccoooommmm
1 Introduction
The human experience unfolds as a continuous dialogue between sensory perception and language articulation. A perceived change in raw physiological readings like "heart rate can be seamlessly translated from low-level statistical descriptions (e.g.. "heart rate spikes 65 to 90") to high-level semantic abstractions (e.g.. "periods of strength training noted"). This intrinsic interplay between raw sensation and linguistic abstraction forms human comprehension of internal states, health conditions, behavioral changes, and more [7, 16]. Wearable sensors now record these narratives at unprecedented resolution, producing minute-lvel data that captures dense insights into individual's physiological and behavioral states [24, 41]. Aligning and interpreting this rich, continuous stream of fine-grained sensor data with intuitive and actionable language descriptions is critical for user engagement [40], clinical decision-making [38], personalized insights [7], and behavioral interventions [26].
However, directly interpreting continuous raw sensor data poses significant challenges. While large language models (LLMs) excel at processing textual sequences, they inherently struggle with the high-dimensional, continuous, and temporally extensive nature of wearable data [21]. For example, single day of minute-level multimodal sensor recordings (see Fig. (a)) would expand to over 200,000 tokens [24], far exceeding the practical context length limits of most contemporary LLMs [30,32] Interestingly, even with minimal subsampling to fit within context constraints, LLMs fail to perform
"Co-first authors, Work done while Google Research.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

A
Dally Multimodal Sensor (24 hours)
max
Paroo Text Generation
deviationn of 6. and voku range 22 2013
Tump Son
Structural
Incressing teecil
Shece
spple step munde anreewaee dep
Semantic tacdvity,
Asnabic Workous durig monuus noir sur
Doroonen
LOCar
Semsor Encoder
Test Enooder
Samsor Erooder
Tedt Enooder
Sensor Encoder
Text Encoder
Den
2uro shoe
sencor-deed
Contrastive
Cross RESNTOON
Semsor Encoder
Multimodal Text Decoder
Encoder
Encnder
Mi Aiiimoddil
Captioning Loss
Text Decoder
Tee-shotiinnnnng
Capabbiliie
SensorLM
satt spoct : fioo luiing
WITECT cuption genoration
Sensor-language foundation models (SensorLM) for wearable data, Aligning and interpreting sensor data with natural language remains challenging. (A) We present a comprehensive study using over 59.7 million hours of multimodal wearable data from 103,000 individuals. (B) We design hierarchical pipeline covers statistical, structural, and semantic sensor information.
Figure I:
for automatic paired text generation SensorLM pretraining framework and its use cases for diverse downstream tasks. (D) Radar plot comparing the performance of SensorLM and baselines across various tasks and settings (details in Sec. 5).
(C) The
well: Fig. I(d) illustrates the poor zero-shot performance of Gemma-3-22 (-mma [30] and Gemini 2.0 [29] (details in Sec. 5). As such, standard LLM methods either become overwhelmed by data volume or sacrifice the granularity needed for aligning sensor events with meaningful linguistic descriptions.
Furthermore, despite preliminary efforts in constructing paired sensor-text datasets facilitate effective cross-modal aligmment, the absence of large-scale, high-quality sensor- lansor-language corpora remains significant bottleneck [6, 10, 15,40) 40J. As illustrated in Table current sensor-text datasets are significantly limited in scale and primarily structured in restrictive question-answeriig formats [6, 15, 40). This highlights fundamental gap the lack of principled and scalable methods for generating truly large-scale and diverse sensor-language lansor-language datasets. Consequently, existing approaches yield sparse and narrowly scoped sensor-text pairs, insufficient for training generalizable foundation models capable of comprehensive multimodal understanding across diverse, open-ended tasks.
To fill the gap, we introduce SensorLM, family of sensor-language foundation models that unlock meaningful interpretation of raw wearable data and enable novel sensor applications through natural language. The effectiveness of SensorLM stems from three innovations: (1) hierarchical, automated caption-generatie pipeline that systematically captures multi-level and semantic-fron fine- enn-igai sensor streams; (2) building upon this pipeline, the curation of the largest-scale sensor-lane langua- dataset to date, comprising over 59 million hours of wearable data from more than 103,000 individuals; and (3) a generic pretraining framework that integrates diverse multimodal architectures (e.g., CLIP [25], Cap [33], CoCa [39]) for scalable and robust learning.
structural,
To rigorously evaluate SensorLM, we benchmark its performance against state-of-the-art (SOTA) methods across a diverse range of real-world tasks in domains such as human activity analysis and metabolic health. Extensive experiments verify the efficacy of SensorLM on multimodal understand- ing and its ability to enable new sensor-driven applications. Our contributions are as follows:
We introduce a hierarchical captioning pipeline for raw sensor data, enabling the curation of the largest gensor-language study to date with over 59 million hours of data from over 103,000 people. We design SensorLM, a family of sensor- language foundation models that enable diverse sensor capabilities through natural language.
We conduct extensive experiments across various tasks in human activity analysis and metabolic health, verifying the superior performance of SensorLM against SOTA methods.
Further analyses reveal intriguing properties of SensorLM on its scaling behaviors, data efficiency, multimodal understanding & generation, and zero-shot generalization to unseen tasks and concepts.

---

Table 1: Comparisons of studies on sensor-text modeling.
Sensors'
Text
Study
People
Hours (000s)
PPG
ACC
EDA
TEMP
ALT
statistical
structural
scmantic
Xing [36]
<1
X
X
X
X
Moon [23]
X
X
Yu et [40]
X
X
Li et al. [15]
214
3.7
X
X
X
X
/
SensorLN (ours)
103,731
59,749
/
/
PPC: Phowoplehyysmugraphyy
Accelerometer, Electrodemnal Activity, Temperature. Altitude.
2
Related Work
Aligning Sensor with Language. Despite growing interests integrating sensor data with natural language, large-scale paired corpora derived from uncurated wearable data remain scarce, yet are crucial for training effective cross-modal models [25, 35, 39]. Prior work typically addresses this integration by using pre-derived textual summaries of sensor features or ingesting raw sensor values as tabular inputs LLMs downstream prediction tasks [7, 13, 16, 20]. Other multimodal methods incorporate specialized sensor encoders with alignment modules to enhance an LLM's 's capability to interpret sensor data [6, 10, 14]. Recent efforts initiate the creation of sensor-text data primarily through employing sensor
frameworks [15, 40], explicitly tailored towards human activity recognition. However, these approaches often rely on manual summarization and are inherently limited by sparse and narrowly-scoped sensor-text pairs, restricting generalization broader scenarios (see Table 1). In contrast, we focus on learning aligned sensor- lansur- representations directly from large-scale, hierarchical captions, enabling strong zero-shot generalization across diverse tasks.
Multimodal Sensor Foundation Models. Recent advances in sensor data modeling have demon- strated improved accuracy, robustness, and generalization by leveraging self-supervised pretraining on large-scale physiological and wearable sensor datasets [24, 41 Existing sensor foundation models primarily focus single-modal but multi-channel sensor data, utilizing either contrastive- based [1. 11,3177444 or reconstruction- based pretraining objectives [24]. More recent efforts have explored multimodal sensor foundation models by aligning individual physiological signals (e.g., ECG, EEG) or IMU sensor data with other modalities, including language or vision, through joint representation learning [11, 23, 45]. Our work extends sensor foundation models towards comprehen- sive multimodal lensor-languaa understanding, enabling novel sensor applications (see Fig. 1) and multimodal capabilities via joint modeling of diverse sensor types and natural language.
Vision-Languuae Pretraining. Vision-language pretraining (VLP) has significantly advanced multimodal AI, enabling models to effectively learn representations that integrate visual and textual data [25.39]. Leveraging large-scale image-text datasets, VLP methods have produced multimodal foundation models capable of strong zero-shot image classification, captioning, visual question answering, and cross-modal retrieval 22,35,33]]. Key VLP methods include contrastive learning (c.g.. CLIP [25]), which aligns image and text embeddings while separating dissimilar pairs, and generative pretraining (e.g.. Sim VLM [35]). which employs prefix language modeling objective. More recent models like CoCa [39] combine both contrastive and generative objectives for enhanced multimodal performance. The success of VLP has inspired similar pretraining strategies in other modalities such video, audio, time series (8,23,4
24,33.4) 44,, as well specialized domains such medicine [9, 19,46). Our work extends VL.P paradigms to the sensor domain, implementing generic pretraining framework that incorporates prominent multimodal architectures for scalable sensor-language modeling.
3 Sensor-Language Dataset Construction
Building lensor-language foundation model requires essential components: (1) large-scale, diverse collection of sensor data, and (2) corresponding captions that capture useful properties of the data. We summarize prior efforts on sensor-text modeling in Table 1, and detail our study below.

---

Sleeping
Ellitiical
Statistical
3 The teart rais readings show innrri of e5.35.3 a slanzied devsation 14.7..
100.3
3 Acceleromen log anergy features fiear of 49.2, fmariiumi 194.0.
72.8
TEMP ACCEL
126.3 62.7
Structural
3 A peak heart rote is detected at minuse 940
3 From minute 840 to 1180, acceliig log energy exhibits decreasing ng trend...
slope indicates
trand sparning to
Semantic
240
480
720
960
1200
Elptical ocourred between minute 830 and B6G Sleeg took place during the marutes 5 thrrughh A08
(minutes)
Hierarchical sensor caption generation pipeline. Given multimodal wearable data, we generate thrce levels of captions: (1) statistical, capturing basic quantitative metrics; (2) structural, describing temporal patterns, trends, and signal dynamics; and (3) semantic, providing contextual information such as behavioral states or physical activity. When applicable, each caption annotated with time frame (marked in grey).
Figure 2:
3.1 Sensor Data and Processing
We collected wrist-worn wearable sensor data from Fitbit and Pixel Watch devices. The dataset includes multimodal time series signals from photoplethysmoorrppyy (PPG), 3-axis accelerometer (ACC), altimeter (ALT), skin temperature (TEMP), and electrodermal activity (EDA) sensors. From these time series, we extracted 26 features: twelve metrics related to heart rate and heart rate variability (HRV) derived from the PPG signal, ten accelerometer-busse features (c.g.. jerk, steps), as well as the mean and slope of skin temperature, mean tonic electrodermal activity, and mean altimeter pressure (full details are in Appendix A. The input to the model consists of one-day windows of sensor data at minutely resolution, resulting in input matrices of shape [26 features 1440 minutes].
3.2 Hierarchical Sensor Caption Generation
significant challenge in sensor-language pretraining is the general absence of paired textual descriptions for wearable sensor data. Unlike domains such as vision or clinical imaging, where data often comes with associated text (e.g.. image captions or clinical reports) [9, 29,25, 25, 39], sensor data collected in the wild typically lacks such naturally occurring pairings. Existing annotations, when available, are often sparse and limited to coarse labels such as discrete activity categories (Table 1).
To bridge this gap and enable robust alignment between natural language and multimodal sensor streams, we propose a hierarchical caption generation strategy. This approach creates captions at three distinct levels of abstraction-ssaaiistiaall, structural, and semantic. By doing so, we aim train a model capable of representing not only basic numerical summaries and dynamic temporal patterns, but also high-level events and contextual states observed in daily sensor data. An illustrative example of the overall process and all three level of captions is shown in Fig. 2.
Statistical Captions. Statistical captions provide a quantitative summary of the sensor data. For each distinct sensor feature channel (e.g., heart rate, temperature, step count), basic statistical measures- including mean, maximum, minimum, and standard deviation are computed. To enhance linguistic diversity, these values are embedded into a variety of sentence structures using a set of templates (details in Appendix A.2). This offers concise, data-driven overview of individual sensor streams.
Structural Captions. Structural captions focus on encoding the dynamic characteristics and patterns within time-series sensor data. This includes identifying and describing notable trends,fluctuulie fluctuations, and other temporal features across sensor channels. Methodologically,, we apply sliding windows to each time series to detect significant increasing, decreasing, stable trends, as well as identifying sharp spikes and drops. Similar to statistical captions, these identified patterns are then verbalized using a diverse set of templates, from which a random subset is selected for each data sample. This enables the model to learn the temporal shape and behavioral dynamics of the sensor signals.
Semantic Captions. Semantic captions aim to capture the high-level meaning and context embedded the sensor data, reflecting what the individual might be doing or experiencing. This layer incorpo- rates information about recognized activities and sleep periods, specifying start and end times of each event-e.g.. "Observed Outdoor Bike activity from minute 550 to 561". Additionally, usgr-logged mood data with corresponding timestamps is integrated, such as "The person logged their mood as

---

sensor
text
Lio only (CUPP)
Sensor Encoder
Text Encoder
heart rete nereusea between minnste and 140 Duning renuitee Aarobi e Worl evant.
Sensor Encoder
Text Encoder
ncrcc
Reep only
(Cap)
Multimodal Text Decader
Sensor Encoder
Text Encoder
podling projection
Contrastive Loss
pocing preection
Leom
Sensor Encoder
Text Encoder
Infention
Multimodal
Cos + Leo (CoCa)
Multimodal Decoder
Captioning Loss
Leap
Figure 3: The SensorLM architecture, pretraining objectives, and representative variants. Our framework integrates both contrastive (Leoa) and generative (Cep) objectives, extending prior approaches (c.g.. CLIP [25], Cap [33], CoCa (39]) to sensor data and recovering them specific configurations within single architecture.
Frustrated at minute 1110." " Integrating this contextual information enables the model associate real-world events and subjective states with underlying sensor patterns.
3.3 Large-Scale Pretraining Sensor-Test Dataset
Building upon the collected sensor data and caption generation pipeline, we construct the largest sensor-language paired pretraining dataset to date orders of magnitude larger than prior studies (Table 1). All data de-identified and used with participant consent and IRB review (Advarra). The dataset comprises 2,489,570 darson-days of data from 103,643 people across 127 countries, collected between March 1st and May 1s 2024. (Further details are provided in Appendix A.1.)
We generate multiple caption combinations and evaluate their effectiveness in Sec. Unless otherwise specified, the combination of structural and semantic captions in main experiments.
4 SensorLM
SensorLM presents generic framework for training sensor-language foundation models (Fig. 3). Inspired by prominent VI.P architectures with contrastive (e-g.. CLIP [25], SigLIP [42]). generative (e.g., Cap [33]), and hybrid (c.g., CoCa [39]) objectives, SensorLM extends these approaches to sensor data and recovers them as specific configurations within a single architccture.
From Unimodal to Multimodal Paradigms. We first position SensorLM within the broader family of foundation models that leverage language supervision, adapting these paradigms to the sensor domain. Traditional discriminative sinle-encoder models perform well on lxxe-vocabulary tasks like activity recognition but lack flexibility for open- open-ended language grounding. Contrastive dual-encoder approaches [25] enable zero-shot generalization and efficient retrieval by aligning modalities in shared space. Encoder-decoder architectures, in contrast, generate rich natural language conditioned on the input (e.g., sensor data), offering fine-grained interpretability via language. While expressive, they may struggle with robust cross- strategies into single framework for censor-language pretraining.
cross-modal modal alignment. SensorLM integrates these complementary
Architecture. SensorLM comprises sensor encoder, a text encoder, and multimodal text decoder (Fig. 3). The sensor encoder transforms time-series sensor data into a compact latent representations. We adapt Vision Transformer (ViT) architecture to handle sensor data by segmenting the sequence into patches, applying linear embeddings, and processing them through transformer blocks to capture both local and long-range temporal dependencies. The text encoder follows a similar process to encode input text into unimodal representations. Finally, the multimodal text decoder is causally masked transformer that integrates both sensor features (via aross-attention) and text features to produce multimodal text representations.
5

---

Pretraining Objectives. SensorLM is pretrained with composite loss function that combines both contrastive and generative objectives. Given a batch of N sensor-text pairs {(z,,, Mn) [e the contrastive loss Loon is applied normalized unimodal text embeddings vs (from the text encoder) and sensor embeddings 8; (from the sensor encoder) via symmetric cross-modal objective [5]:
1 N
N
log exp(sim(s;,
exp(sim(p;, log
)
Loon
2=1 م jei exp(sim 8;, 25 T/)
j= jot exp(sim((r v. 8,) 8,)/T)
sensor-to-tex.
sensor
where sim(..- -) is the similarity measure between embeddings, and T denotes the temperature param- eter. The captioning loss Cep is standard cross-entropy loss applied to the outputs of multimodal text decoder. It maximizes the conditional likelihood of the paired text y given the sensor input æ under forward autoregressive factorization:
Ceap Σ
r-1 log (o y<1,
Combined Objective and Representative Variants. The final objective a weighted combination:
LsensorLM Avea Lcon Acap Leap. where Acon and Aeup control the balance between contrastive and generative learning. Under this unified formulation, SensorLM seamlessly recovers representative variants within a single framework (Fig. 3): (1) CLIP (i.e., Aaa 0), (2) Cap (i.e., Acce = 0), and (3) Xcap = 1). We investigate these model variants in Sec. 5.3.
CoCa (i.e., Avon
The SensorLM Family. We train four variants of SensorLM with increasing sizes: SensorLM-S SensorLM-B, Se and SensorLM-XX -XL These variants employ sensor encoders with 3M, 114M, 404M, and 1. 27B parameters (details in Appendix B.I). We study the effects of model scaling in Sec. 5. Unless otherwise specified, we report results using SensorLM-B.
Experiments & Results
Datasets. We utilize three evaluation datasets for activity recognition, health-related tasks and cross- modal retrieval. Detailed information and demographic breakdowns are provided in Appendix A.3.
Activity dataset. The Activity dataset comprises 22,289 person-day from 10,013 individuals. We randomly sampled ~~ 1,000 test examples for each activity for zero-shot activity recognition (AR) and few-shot adaptation. This dataset is from the same population as our pretraining sensor data. Metabolic dataset. The metabolic dataset, used for "Hypertension" and "Anxiety" prediction tasks, were sourced from held-out, IRB-approved observational study of adults in the US [22]. The data comprises 241,532 examples (151,346 training and 90,186 testing) from 1,979 individuals. It includes self-reported information covering demographics (e.g., age, sex, weight) and medical conditions (e.g., "Hypertension" and " "Anxiety"),, providing rich labels for health analyses.
Sensor-text retrieval dataset. We construct retrieval dataset of 39,766 examples sourced from separate set of 975 individuals who were not included in the pretraining set. The same selection criteria and caption generation method used in pretraining were applied.
Baselines. For zero-shot classification and cross-modal retrieval, there are currently no existing multimodal baselines with these capabilities. We therefore compare against two LLMs, Gemini 2.0 Flash [29] and Gemma-3-27B [30], by formatting sensor data as tabular input. In addition, we finetune Gemini 2.0 with paired sensor and activity data through supervised fine-tuning (SFT). For few-shot learning and linear probing, we compare with SOTA SSL methods including SimCLR [5], DINO [3], and Masked Siamese Network (MSN) [2]. Implementation details can be found in Appendix B.
Metrics. For zero-shot classification, we report area under the ROC curve (AUROC), Macro F1- score, and Balanced Accuracy (BAce). For few-shot learning, we report AUROC across 5, 10, 20, 50 samples per class. Cross-moda retrieval is evaluated using Recall@ I and Recall@5 (R @ K).
5.1 Main Results
Zero-Shot Learning. One of the key capabilities enabled by SensorLM is zero-shot classification, which we evaluate on range of AR tasks. We use prompt ensembling [25] from the text encoder

---

Table 2: Zero-shot activity and concept classification. We compare SensorLK with representative LLM baselines across (a) main activity classification, (b-d) activity-related concept classification, and (e-f) fine-grained recognition. Gemini 2.0 with supervised fine-tuning (SFT) does not output class- probbbbiiy scores; therefore AUROC is not reported ("-"). Detailed task definitions and experimental setups provided in Appendix B.5,
(a) Activity recognition (20-class)
(b) Activity by environmental context
Metries
AUROCT
FIT
BAce
Metries
AUROCT
F11
BAcc
Gemma-3-27B [30]
0.50
0.01
0.05
Gemma-3-27B [30]
0.51
0.22
0.25
Gemini [29]
0.51
0.03
0.07
Gemini 2.0 (29)
0.50
0.19
0,25
Gemini 2.0 (SFT) (29)
0.06
0.10
Gemini 2.0 (SFT) [29]
0.25
0.28
SansorIM
(438%)
0.29 (+23%) (+21%)
SensorLM
0.64 (+13%) (-08%) 0.38 (+10%)
(c) Cardio vs. strength training
(d)
Locomotion vs. stationary
Metries
AUROCT
FI1
BAcc
Metries
AUROCT
F11
BAcc
Gemma-3 [30]
0.53
0.42
Gemma-3-271 [30]
0.51
0.51
Gemini 2.0 [29]
0.50
0.39
0.50
Gemini 2.0 [29]
0.55
0.52
0.55
Gemini 2.0 (SFT) [29]
0.44
0.53
Gemini (SFT [29]
0.53
0,55
Sensor1M
0.71 (+18%)
0.63 (-19%) 0.66 (+13%)
SensorIN
0.61 (+06%)
0.58 (-05%)
0.58 (+12%)
(c) Fine-grained recognition (gym cardio)
(f) Fine-grained recognition (outdoor sports)
Metries
AUROCT
FI
BAcc"
Metrics
AUROCT
F1
BAcc
Gemma- 3-27B [30]
0.49
0.16
0.25
Gemma 3 3-27B [30]
0.50
0.18
0.25
Gemini 2.0 [29]
0.52
0.20
Gemini [29]
0.54
0.29
Gemini 2.0 (SFT) [29)
0.28
0.35
Gemini 2.0 (SFT) [29]
0.26
0.32
SenaorLM
0.76 (+34%)
0,50 (+22%)
0.51 (+16%)
SensorLM
0.83 (+29%)
0.52 (+26%)
0.53 (+21%)
with diverse prompts to compute average label embeddings. and classify by selecting the label whose embedding is closest to the sensor embedding. We include 6 tasks in total, including 20-class activity recognition, activity concept classification (2-4 classes by "environmental" context or "physiological" type), and fine- grainer recognition (4 classes within "gym cardio" or "outdoor sports"), As shown in Table 2, LLM baselines perform near random on sensor-based activity classification. In contrast, SensorLM demonstrates strong zero-shot capabilities across all metrics and tasks. For abstract concepts such as "environmental" or "physiological" type, the superior performance suggests that the model captures both explicit activity categories and higher-level conceptual understanding.
Zero-Shot Cross-Modal Retrieval. We evaluate the zero- shot cross-modal retrieval performance of SensorLM by assessing its ability to retrieve relevant text descriptions given sensor queries (Sensor + Text) and vice versa (Text - Sensor). Sensor-to-tex retrieval enables querying de- scription based on sensor input, while text-to-sensor re- trieval supports use cases such as expert-driven querying for specific sensor patterns using natural language. Table 3 confirms SensorLM's exceptionally strong performance across all sample sizes, significantly outperforming LLM baselines, which largely failed at this task. SensorLM achieves perfect retrieval on 100-sample benchmark and maintains high accuracy even at larger scales (5k and 40k samples; full results in Appendix C.2).
Table 3: Zero-shot cross-modal retrieval. SensorLM achieves consistently strong re- trieval performance Baseline L.MSs struggle, with most tasks infeasible due to context limits (marked as "-"). Detailed experimental setups and com- plete results are in Appendix B.7 and C.2.
a held-out dataset.
100 samples 5,000 samples
Metries
R@
R@5
R@1
R@5
Sensor + Text:
Gemma-3-22 3-27B [30]
1.0
5.0
Gemini 2.0 [29]
5.0
9.0
Qualitatively, Fig. 4 verifies SensorLM's ability retrieve accurate descriptions for unseen sensor data involving multiple activities. In addition to correctly retrieving the ground truth, the top similar captions are also semantically relevant - capturing either similar activities occurring at the same time (e.g.. "Walk" vs. "Run"" or the same activity at different times. This reflects its understanding of both activity semantics and temporal alignment.
SensorLM
100.0
100.0
98.2
99.4
Text Sensor:
Gemma-3-2778 [30]
Gemini 2.0 2.0[29]
SensorLM
100.0
100.0
96.7
99.2
7

---

Steeping
Treadmil
Strength Training
Ground
Run recorded minute 452 and Treadmill recorded within minune range Peried Strength throusht 356.
Y 65.2
85.1
from
Similar Caption
TEMP ACCEL
90.7
SC: 0:34
Walk was detected between minutes 432 and 437, from minute 481 to 606, the user had a period of Walk Sleep spieade ocourred between minute and 21B
65.3
Similar Caption
SC: 0.32
An instance Run was identiil from minure to'l Sod Period of Trendmill noted from minute 574 636.
B68 Instance Strength training was
30.3
Isentified from
0
120
240
360
480
600
Caption
(minutes)
SC:-0.05
Bike event was detected between minutes 516 and
Figure 4: Zero-shot sensor-to-text retrieval qualitative example. We show similarity scores (SC) for the correctly retrieved ground truth, lop similar captions, and dissimilar caption. Green highlights correct events and time frames; Yellow indicates partial matches; and Red denotes incorrect events and time frames. In addition to correctly retrieving the ground truth, SensorLM also demonstrates semantic understanding by assigning high similarity scores to partially correct candidates.
Hypertension
Anwiety
Activity
0.54
DINO
0.58
AUROC
0.03
SensorLM
SimCLR
AUROC
0.66
AUROC
0.52
0.54
32
0.51
0.52
.050
K0
50
0.50
60
0.50
26
50
20 50
Training Labels Per Class
Training Labels Per Class
Treining Labels Per Cless
Figure 5: Few-shot adaptation to downstream tasks. We evaluate the label efficiency different pretrained sensor encoders with varying numbers of training labels per class. Across diverse tasks, SensorLM achieves comparable or superior performance than SOTA methods. Additional results are provided in Appendix C.I.
Few-Shot Transfer Learning. We evaluate the quality of the learned sensor representations through few-shot learning on three tasks: 20-class AR using the Activity dataset, and two binary prediction tasks on clinical condition ("Hypertension") and mental health ("Anxiety")) using the Metabolic dataset. Fig. 5 shows the few-shot performance of SensorLM compared to baseline SSL methods as the number of training labels per class increases from 5 to 50. For AR, SensorLM significantly out- performs baselines across all few-shot settings, achieving AUROC of 0.88 with only 50 labels per class, highlighting its effective representations in discriminating activities under limited supervision. For "Hypertension" and "Anxiety" prediction, SensorLM shows competitive performance. Linear probing results on the full training set support similar conclusions (Appendix C.I).
5.2 Analyses
Scaling Laws. We investigate the scaling behavior of SensorLM when trained on large-scale sensor- language dataset by varying training compute, dataset size, and model seale [12]. As observed in Fig. 6, the performance
zero-shot activity recognition improves consistently with increased training steps and data size. For example, when scaling up the compute, SensorLM-B increases from an AUROC of 0.66 at 17.4 TPU hours to 0.75 at 174 TPU hours. Expanding training data also yields consistent gains, especially for larger models. However, gains beyond 12 million hours diminish. showing saturation trends as reported in prior work [12, 24]. Larger models such as SensorLM-L and SensorLM-X achieve higher performance when sufficiently trained, though under low compute they may underperform due to undertraining (a phenomenon consistent with scaling law observations). Overall, SensorLM demonstrates consistent and predictable scaling behaviors, highlighting the applicability of scaling laws to sensor- language modeling.
and Incomplete Data (Appendix C.7). We evaluate SensorLM robustness to out-of-domain (OOD) and missing data. With our metabolic downstream dataset containing additional device types unseen during pretraining, we compare performance on in-domain and OOD devices. SensorLM demonstrates strong generalization with only degradation in some of
Robustness to
8

---

0.05M
60M
Sensort
Sersor
Sensort
0.8
Paro-Shott Activity AUROC
0.6-
0.6
0.7
0.5
10
10
10P
10'
101
10² Model Size (Millon of Parama)
10
Compute
Data Size (Hours)
(TPUV8 vb cone nours)
Figure 6: Scaling behaviors SensorLM. We show the zero-shot downstream performance SensorLM as a function compute (left), data size (middle), and model size (right). Results demonstrate that increasing compute, data, and model size each leads to consistent performance gains.
the tasks. We also assess missing-date -ata scenarios, finding that performance degrades only marginally, indicating that SensorLM remains robust and flexible with incomplete sensor inputs.
Zero-Shot Generalization Unseen Classes. To evaluate SensorLM's gen- eralizability to novel activities, we con- ducted case study by pre-training data comprising 20 activity classes and tested on previously unseen ones (de- tails in Appendix B.8). The results of one-versus-all all AUROC and 95% CIs for these classes are shown in Fig. 7(a). The impressive zero-shot performance achieved on these unseen activities indi- cates that SensorLM learns the broader concept of activities rather than memo- rizing the training set. Specifically, ac- tivities that are conceptually similar to those the training set exhibit higher performance (c.g., "Snowboarding" is similar to "Skiing"). This is further sup- ported by the visualization of label and sensor embeddings using t-SNE [34] (Fig. 7(b)), where semantically related activities form coherent clusters, indicat- ing that SensorLM infers unseen classes based on their proximity to known con- cepts in the learned embedding space.
(e)
(b)
Sensor Embeddings
Snowboarding
Seowirarting
Sainn
Racquetbal
Poddieboardig
Text Prompt Embeddings
Budménnonn
Snowboarding (unsecn)
Dending
Sking
Calsthenics
SensorL
Plttes
Dasketbal
Gemmi
Gemma
Zero-shot generalization analysis of SensorLM, (a) Overall performance on unseen activities. (b) Learned rep- resentations in both the sensor and text embedding spaces. We visualize conceptually similar but previously unseen activities as a case study. The semantic alignment across modalities allows SensorLM to infer the nature of unseen activities based on their conceptual proximity known ones.
Figure 7:
Sensor Caption Generation. A key strength SensorLM stems from its encoder-decoder archi- tecture, trained with generative objective. This design positions SensorLM to effectively process multimodal sensor embeddings, yielding sensor captioning capabilities. Beyond its inherent clas- sification and retrieval capabilities, SensorLM demonstrates strong generative performance. On dedicated evaluation set of 200 sensor-text pairs (sampled from the sensor-caption retrieval dataset), SensorLM directly generates captions from sensor input (and [START] token). As shown in Fig. 8, SensorLM (pre-trained on semantic captions) outperforms powerful L.MM baselines, including Gemini 2.0 Flash [29] and Gemma-3-27B [30]. These results highlight SensorLM's capability sensor-language foundation model that can both interpret sensor data and generate meaningful natural language descriptions.
5.3 Ablation Studies
We provide main ablation studies on SensorLM, with complete results in Appendix C.4 and C.5.

---

Captioning Performance
Correct Example
Predicted
Ground Truth
Cemma
SemaorLM
Gemni
Elliptical was detected between minutes 750 and 765. Strength training was detected ted between minutes 765 and 796.
Elliptical recorded within the 750-763 minute range. Strength training event was detected between minutes 764 and 797.
Partially Correct Example
Ellipticel was recorded between minute 591 and 599, Peried of walk noted from minute 540 until Workout detected between minutes and 622.
Elliptical state observed from minute 591 up 599, Spinning period between minutes 603 and 625. Walk recorded within the 540-566 minute range.
Precsion
Rocal
Figure 8: Sensor caption generation results. (A) Captioning performance of SensorLM and baselines. For SensorLM, only the sensor data and a [START] token are provided lo generate captions. (B) A correctly generated example. (C) A partially correct example with inaccurate components. SensorLM produces meaningful and semantically accurate captions. Detailed experimental setups and additional results are Appendix B.9 and C.3,
Understanding the impact of pretraining caption variants. We investigate how different levels of sensor descriptions affect downstream tasks. Table shows that semantic captions crucial for zero-shot AR, whereas models trained solely on statistical or structural captions underperform due to lack of activity-level concepts. Combining semantic and structural captions improves zero-shot AR over semantic-only models, in- dicating that local signal patterns and temporal struc- ture contribute complementary information. However, adding statistical captions to semantic ones reduces zero-shot AR performance, likely because daily-level statistics are less relevant to fine-grained activities. Yet, statistical captions benefit "Anxiety" and "Hyperten- sion" predictions, either alone or when combined with other caption types. Overall, the "semantic + structural" combination provides the best trade-off across tasks.
Table 4: Comparisons of SensorLM caption variants. We evaluate different combinations of caption types used during pretraining. AUROCT is used as the metric. Best results of each column are in bold and the second best are underlined. Default setting used in the main experiments marked in gray
Caption Variant
Zen--Shot
Linear Probing
statistical
stractural semantic
Activity
Activity
Ansiety
0.76
0.67
d
x
0.50
0.78
0.63
0.95
x
0.66
0.84
0,68
0.84
0.94
0.65
x
0.49
0.79
0.67
/
0.66
0.86
0,68
Pretraining architectural variants. We study differ- ent architectural variants trained with contrastive and generative objectives. As Table 5 reports, SensorLM (CoCa) consistently outperforms the single-objective variants across key metrics for both zero-shot classi- fication and linear probing. The generative objective enhances the contrastive alignment by providing fine- grained text supervision, leading to improved cross- modal understanding. Notably, the SensorLM (Cap) model performs poorly in zero-shot classification due the absence of trained projection head.
Table 5: Comparisons of SensorLM architec-
tural variants. We compare different choices used during pretraining. Default setting used the main experiments is marked in gray...
Zero-Shot
Linear Probing
Arch Variant
AUROC"
FI"
AUROC!
Fi
SensortM (CL
0.53
SeneorLN (SigLIP)
0.78
0.17
0.38
SensorLM (Cap)
0.55
0.01
0.90
0.32
SensorLM (CoCa)
0.84
0.29
0.94
0.57
Discussion
Limitations. It is crucial to understand that SensorLM is not a clinically validated diagnostic tool and is not intended for clinical diagnosis, treatment, medical becision-making; deployment for such uses would require further analysis of applicable healthcare regulations. Additionally, our evaluation is limited to specific wearable devices and sensor data; while the method is general, further work is needed to explore its generalizability on other types of sensor data.
Conclusion. We present SensorLM, family of sensor-language foundation models that unlock the understanding of wearable sensor data through natural language, enabled by a novel hierarchical captioning pipeline and the largest sensor-language dataset to date. SensorLM achieves superior per- formance zero-shot, few-shot, and cross-modal retrieval tasks, as well as demonstrating intriguing properties such as scaling, generative, and zero-shot generalization capabilities.
10

---

References
[1] Salar Abbaspourazad, Oussama Elachqar, Andrew Miller, Saba Emrani, Udhyakumar Nallasamy, and lan Shapiro. Large-scale training of foundation models for wearable biosignals. In The Twelfth International Conference Learning Representations, 2023.
[2] Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Mike Rabbat, and Nicolas Ballas. Masked siamese networks for label-efficient learning. In European Conference on Computer Vision, pages 456-473 Springer, 2022.
[3] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 9650-9660, 2021.
[4] James Carpenter and John Bithell. Bootstrap confidence intervals: when, which, what? practical guide for medical statisticians. Statistics medicine, 19(9): 141-1164, 2000.
[5] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[6] Wenqiang Chen, Jiaxuan Cheng. Leyao Wang, Wei Zhao, and Wojciech Matusik. Sensor2text: Enabling natural language interactions for daily activity tracking using wearable sensors. Pro- ceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 8(4):1-26, 2024.
[7] Justin Cosentino, Anastasiya Belyaeva, Xin Liu, Nicholas A Furlotte, Zhun Yang. Chace Lee, Erik Schenck, Yojan Patel, Jian Cui, Logan Douglas Schneider, et al. Towards a personal health large language model. arXiv preprint urXiv:2406.06474. 2024.
[8] Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. Clap learning audio concepts from natural language supervision. In ICASSP 2023-2023 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEFE, 2023.
[9] Zhi Huang, Federico Bianchi, Mert Yuksekgonul, Thomas J Montine, and James Zou. A visual-language foundation model for pathology image analysis using medical twitter. Nature medicine, 290922307-, 2023.
[10] Sheikh Asif Imran, Mohammad Nur Hossain Khan, Subrata Biswas, and Bashima Islam. Llasa: A multimodal 1lm for human activity analysis through wearable and smartphone sensors. arXiv preprint arXiv:2406. 14498, 2024.
[11] Weibang Jiang, Yansen Wang, Bao-liang Lu, and Dongsheng Li. Neurolm: universal multi- task foundation model for bridging the gap between language and eeg signals. In The Thirteenth International Conference on Learning Representations, 2025.
[12] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXi::2001.0836/ 2020.
[13] Yubin Kim, Xuhai Xu, Daniel McDuff, Cynthia Breazeal, and Hae Won Park. Health- Ilm: Large language models for health prediction via wearable sensor data, arXiv preprint arXiv:2401.06866 2024.
[14) Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742. PMLR, 2023.
[15] Zechen Ii, Shohreh Deldari, Linyao Chen, Hao Xue, and Flora D. Salim. Sensorl.LM: Aligning large language models with motion sensors for human activity recognition, 2025.
[16] Xin Liu, Daniel McDuff, Geza Kovacs, Isaac Galatzer-Levv, Jacob Sunshine, Jiening Zhan, Ming-Zhe Zher Poh, Shun Liao, Paolo Di Achille, and Shwetak Patel. Large language models are few-shot health learners. arXiv preprint arXiv:2305.15525, 2023.
[17] Ziyu Liu, Azadeh Alavi, Minyi Li, and Xiang Zhang. Guidelines for augmentation selection in contrastive learning for time series classification. arXiv preprint 0rXiv:2407.09336. 2024.
[18] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. International Conference Learning Representations, 2017. 2017.

---

[19] Ming Y Lu, Bowen Chen, Drew FK Williamson, Richard J Chen, Ivy Liang, Tong Ding, Guil- laume Jaume, Igor Odintsov, Long Phi Le, Georg Gerber, al. A langulee foundation model for computational pathology. Nature Medicine, 30(3):863-877) 2024.
[20] Mike A Merrill, Akshay Paruchuri, Naghmeh Rezaei, Geza Kovacs, Javier Perez, Yun Liu, Erik Schenck, Nova Hammerquist, Jake Sunshine, Shyam Tailor, et al. Transforming wearable data into health insights using large language model agents. arXiv preprint arXiv:2406.0m 064664 2024.
[21] Mike A. Merrill, Mingtian Tan, Vinayak Gupta, Thomas Hartvigsen, and Tim Althoff. Language models still struggle to zero-shot reason about time series. In EMNLP (Findings), pages 3512- 3533, 2024.
[22] Ahmed A Metwally, A Ali Heydari, Daniel McDuff, Alexandru Solot, Zeinab Esmaeilpour, Anthony Z Faranesh, Menglian Zhou, David B Savage, Conor Heneghan, Shwetak Patel, et al. Insulin resistance prediction from wearables and routine blood biomarkers. arXiv preprini arXiv:2505.03784, 2025.
[23] Seungwhan Moon, Andrea Madotto, Zhaojiang Lin, Aparajita Saraf, Amy Bearman, and Babak Damavandi. Imu2clip: language- grounded motion sensor translation with multimodal contrastive learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 13246-13253, 2023.
[24] Girish Narayanswamy, Xin Liu, Kumar Ayush, Yuzhe Yang. Xuhai Xu, shun liao, Jake Garrison, Shyam A. Tailor, Jacob Sunshine, Yun Liu, Tim Althoff, Shrikanth Narayanan, Pushmeet Kohli, Jiening Zhan, Mark Malhotra, Shwetak Patel, Samy GhaffaG, and Daniel McDuff. Scaling wearable foundation models. In The Thirteenth International Conference on Learning Representations, 2025.
[25] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 7748-87632 PmLR, 2021.
(26) Mickael Ringeval, Gerit Wagner, James Denford, Guy Paré, and Spyros Kitsiou. Fithit-based interventions for healthy lifestyle outcomes: systematic review and meta-analysis. Journal of medical Internet research, 22 22110))+2394, 2020.
[27] Cédric Rommel, Joseph Paillard, Thomas Moreau, and Alexandre Gramfort. Data augmentation for learning predictive models on eeg: a systematic comparison. Journal of Neural Engineering, 1966020 2022.
[28] Chi Ian Tang. Ignacio Perez-Pozuelo Dimitris Spathis, and Cecilia Mascolo. Exploring con- trastive learning in human activity recognition for healthcare. arXiv preprint arXiv:2011.11544 2020.
[29] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Saan-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. /805, 2023.
[30] Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint (tt.:::5503.198888 2025.
[31] Rahul Thapa, Bryan He, Magnus Ruud Kjaer, Hyatt Moore IV, Gauri Ganjoo, Emmanuel Mignot, and James Zou. SleepFM: Multi-modal representation learning for sleep across brain activity, ECG and respiratory signals. In Forty-first International Conference on Machine Learning, 2024.
[32] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.0m 09288,9888888 2023.
[33] Michael Tschannen, Manoj Kumar, Andreas Steiner, Xiaohua Zhai, Neil Houlsby, and Lucas Beyer. Image captioners are scalable vision leamers too. Advances in Neural Information Processing Systems, 36:46830-46855, 2023.
[34] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne, Journal of machine learning research, 9(11), 2008.
12

---

[35]
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai. Yulia Tsvetkov, and Yuan Cao. Simvlm: Sim- ple visual language model pretraining with weak supervision. arXiv preprint wrXiv:210..10900 2021.
[36] Tianwei Xing, Luis Garcia, Federico Cerutti, Lance Kaplan, Alun Preece, and Mani Srivastava. Deepsqa: Understanding sensor data via question answering. In Proceedings of the International Conference on Internet-of-Tiine Things Design and Implementation, pages 106-118, 2021.
[37] Yuzhe Yang, Xin Liu, Jiang Wu, Silviu Borac, Dina Katabi, Ming- Shegr Poh, and Daniel McDuff. Simper: Simple self-supervised learning of periodic targets. In The Eleventh International Conference on Learning Representations, 2023.
[38] Yuzhe Yang. Yuan Yuan, Guo Zhang, Hao Wang. Ying-Cong Chen, Yingeheng Liu, Christo- pher G Tarolli, Daniel Crepeau, Jan Bukartyk, Mithri R Junna, et al. Artificial intelligence- enabled detection and assessment of parkinson's 's disease using nocturnal breathing signals. Nature Medicine, 28 28(10):2007-2215, 2022.
[39] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022.
[40] Xiaofan Yu, Lanxiang Hu, Benjamin Reichman, Dylan Chu, Rushil Chandrupatla, Xiyuan Zhang, Larry Heck, and Tajana Rosing. Sensorchat: Answering qualitative and quantitative questions during long-term multimodal sensor interactions. arXiv preprint arXiv:2502.0288; 2025.
[41] Hang Yuan, Shing Chan, Andrew P Creagh, Catherine Tong, Aidan Acquah, David A Clifton, and Aiden Doherty. Self-supervised learning for human activity recognition using 700,000 person-days of wearable data. NPJ digital medicine, 7(1):91, 2024.
[42] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. Proceedings of the IEEE/CVF international conference on computer vision, pages 11955-11986, 2023.
[43] Xiang Zhang, Ziyuan Zhao, Theodoros Tsiligkaridis, and Marinka Zitnik. Self-supervised contrastive pre-training for time series via frequency consistency. Advances neural information processing systems, 35:3988-444 4003, 2022.
[44] Xiyuan Zhang, Diyan Teng, Ranak Roy Chowdhury, Shuheng Li, Dezhi Hong, Rajesh Gupta, and Jingho Shang. Unimts: Unified pre-training for motion time series. Advances in Neural Information Processing Systems, 37: 97:107469-107499 2024.
[45] Yubao Zhao, Tian Zhang. Xu Wang, Puyu Han, Tong Chen, Linlin Huang, Youzhu Jin, and Jiaju Kang. Ecg-chat: A large ecg- languag model for cardiac disease diagnosis. arXiv preprint arXiv:2408.. 08849, 2024.
[46] Yukun Zhou, Mark Chia, Siegfried Wagner, Murat S Ayhan, Dominic J Williamson, Robbert R Struyven, Timing Liu, Moucheng Xu, Mateo G Lozano, Peter Woodward-Courd al. A foundation model for generalizable disease detection from retinal images. Nature, 622(7981) (117798)):166116 2023.
13

---

Table Definitions sensor features. detail the names, units, and definitions the 26 features derived from PPG, accelerometer, skin conductance, skin temperature, and altitude signals.
Feature
Unit
Definition
Photoplehyysmogrupyy (PPG)
Heart Rate
Beats/Min
Mean instantaneous heart rate.
Shannon Ent. RR
Nats
Shannon entropy the RR intervals.
Shannon Ent. RR Diffs
Nats
Shannon entropy of the RR interval differences.
RMSSD
Msec
Root mean squared st. dev. of RR intervals.
SDNN
Msec
Standard deviation of RR intervals.
RR Percent Valid
% of 5-minute window with valid RR intervals.
RR 80th Percentile
Msec
80 percentile of 5-minute window of RR intervals.
RR 20th Percentile
Msec
20th percentile of RR intervals.
RR Median
Msec
Median RR interval.
Heart Rate at Rest
Beats/Min
Mean heart rate rest.
Accelerometer (ACC)
Step Count
Steps
Number of steps.
Jerk Autocorclation Ratio
Ratio of lag=1 autocorrelation energy in 1st 3-axis principal component.
Log Energy
Log of sum of 3-axis root mean squared magnitude.
Covariance Condition
Estimate of condition number for 3-axis covariance matrix. Log of ratio of sum of energy 1t 3-axis principal component over energy of 3-axis root mean squared magnitude. Standard deviation of time between zero crossing of 1st 3-axis
Log Energy Ratio
a.u.
Zero Crossing St.
Seconds
principal component.
Zero Crossing Average
Seconds
Mcan of time hetween zero crossing of 1st 3-axis principal component.
Axis Mean
a.u.
Mean of 3-axis.
Kurfosis
Kurtosis of 3-axis root mean squared magnitude.
Sleep Coefficient
Sum 3-axis max-min range, binned into log-scaled bins.
Skin Conductance (EDA)
Skin Conductance Value
SSemmens
Center of linear tonic SCL value fit.
Skin Conductance Slope
S/MMnn
Intraminute slope of SCL values.
Lead Contact Counts
Counts
Number of times leads of the sensor contacting wrist in minute.
Skin Temperature (TEMP)
Skin Temperature Value
C
Value skin temperature.
Skin Temperature Slope
C/Min
Slope of skin temperature.
Altimeter (ALT)
Altitude Dev. Norm
Hactopasculs
Standard deviation of altimeter readings.
A Dataset Overview
A.1 Pretraining Sensor Dataset
Our wearable devices incorporate five distinct sensors: Photoplethysmoorraphy (PPG), Accelerometer (ACC), Skin Conductance (electrodermal activity, EDA), Skin Temperature (TEMP), and Altitude (ALT). While these sensors collect raw waveform signals at high frequencies (i.c., 100Hz, 25Hz, 200Hz, 6Hz, and 10Hz, respectively), we do not directly utilize these high-resolution signals. This decision is driven by practical constraints, including prohibitive storage requirements and battery consumption, which make raw data storage infeasible at our scale. Moreover, learning from full-day waveforms is computationally impractical; for example, a 200Hz signal over 24 hours results in approximately 17 million time points per instance.
Instead, we curate minutely aggregated features from the raw signals and use them as model inputs. These features are grounded in established domain literature, with prior work demonstrating their clinical utility [24]. For example, heart rate variability metrics like RMSSD and the Shannon Entropy of RR Intervals are recognized markers of cardiovascular health, while accelerometer-derinn features
14

---

Table Detailed statistics and demographic information for the pretraining and downstream datasets.
We report participant counts for the pretraining. Activity (downstream), and Metabolic (downstream) datasets, along with demographic breakdowns by sex, age, and BMI.
Activity (downstream)
Metabolic (downstream)
Category
Pretraining
train
test
train
test
Sex:
Male Female Not Specified
64,194 39,376
27.653
6,092
551
258
10,145
2,248
670
455
24
Age:
18 39
52,004
19,340
4,492
415
223
40 59
41,296
15,309
3,172
633
384
60 79
9,340
2,875
618
198
121
2 80
676
120
31
0
Not Specified
327
BMI:
Healthy (< 25)
38,582
15,942
3,685
319
188
Overwcight (25 30)
34,188
14,154
3,017
343
206
Obese (2 30)
25,969
6,131
1,316
481
274
Not Specified
324
81
18
49
28
Total
103,643
37,822
8,343
1,250
729
such as Jerk Ratio capture movement quality. Descriptions of the derived features and their associated sensor modalities are provided Table 6. Our pretraining set included smartwatch and fitness- tracker models (Google Pixel Watch 1-3, Fitbit MobileTrack, Charge 4-6, Sense 1-2, Versa 2-4, Inspire 2-3, Luxe, Versa, Charge 2, and Alta HR).
Table 7 summarizes the demographic composition of our pretraining dataset. The only inclusion criterion was having valid sensor data for least 20% of one day and least one logged event. The cohort is 38% female with mean age of 41.7 years (range: 18 100). BMI distributions indicate 37% healthy (< 25), 33% overweight (25 30), and 25% obese (2 30), with the remaining 5% unspecified.
All data de-identified and used with participant consent and IRB review (Advarra). The of data for pretraining in this manner was approved as exempt under 45 CFR $ 46. (6.10(4))(4) "because the research involves the use of identifiable private infornation/biospccimnns and information, which may include information about biospecimens, is recorded by the investigator in such a manner that the identity of the human subjects cannot readily be ascertained directly or through identifiers linked to the subjects, the investigator does not contact the subjects, and the investigator will not re-identify subjects."
A.2 Sensor Caption Generation
To encourage linguistic diversity and prevent overfitting to single text pattern, employ variety of sentence templates across the three levels of sensor captions. This section details the design of these templates.
Table provides a selection of statistical caption templates, illustrated using "Heart rate" as the example feature with average of 88.7, standard deviation of 9.3, minimum of 70.8. and maximum of 134.9. We employ 20 rewrite templates in total to generate varied statistical descriptions.
Table 9 shows examples of structural caption templates, including descriptions of trends (e.g., "a decreasing trend in heart rate between minute 680 and 960") and spike events (e.g., "a spike step count minute 720"). We use 15 rewrite templates in total for structural captions.
15

---

Table Example of prompt templates used statistical captions.
Statistical Caption Templates
1. The average Heart rate value 88.7, with extremes at 134. 9 (max) and 70.8 (min), and a std of 9.3.
2. The Heart rate data exhibits mean of 88.7, standard deviation
9.3, and its extreme values are 70.8 and 134.9.
3. Heart rate average 88.7, reaching a maximum 134.
minimum 4. Heart rate exhibits a mean 88.7, with peak and minimal values reaching 134.9 and 70.8, and standard deviation of 9.3.
70.8, with a standard deviation of 9.3.
5. For the Heart rate measurements, the mean is 88.7, the standard deviation is 9.3, and the data lies between 70.8 and 134.9.
Table 9: Example of prompt templates used in structural captions.
Structural Caption Templates
Trends:
1. An decreasing trend in Heart rate data recorded between minute 680 and 960.
2. Heart rate exhibits decreasing trend during minute 680-960 interval.
3. An decreasing trend in Heart rate data recorded between minute 680 and 960.
4. The Heart rate trend from minute 680
decreasing. 5. From minute 680 to 960, Heart rate exhibits an decreasing trend.
Spikes:
1. Spike event recorded for steps at minute 720.
2. Data indicates peak for steps at the 720-minute mark.
3. Minute 720 shows a spike for the steps.
4. peak is detected for steps at minute 720.
5. The steps experienced a spike at minute 720.
Table 10 presents templates for semantic captions, which describe high-level activities and their associated timeframes, such "Ouidoor Bike recorded between minute 1121 and 1133". We use 20 rewrite templates in total for semantic captions.
A.3 Downstream Datasets
Table 7 summarizes the demographic distributions for the downstream linear probing and few-shot evaluation datasets: the Activity dataset and the Metabolic dataset. For zero-shot classification tasks, we resample the test data to create a more balanced evaluation set. Additional details, including the number of samples per class, provided in Appendix B.5.
The activity downstream dataset used these same devices as our pretraining set, while the metabolic downstream dataset incorporated an additional 17 device types (Fithit Flex, One, Zip, Charge, Charge HR, Charge 3, Flex 2, Inspire HR, Blaze, Alta, Fibitt Ultra, Versa Lite, Surge, lonic, Fibit Classic, Ace 2, Inspire).
16

---

Table 10: Example of prompt templates used in semantic captions.
Semantic Caption Templates
1. From minute 1121 to 1133, the user had period of Outdoor Bike.
2. Outdoor Bike recorded within the 1121-1133 minute range.
3. Outdoor Bike episode occurred between minute 1121 and 1133.
4.
Outdoor Bike was recorded between minute 1121 and 1133. Identified Outdoor Bike across the timeframe of minute 1121 1133.
5.
Table 1I:
The SensorLM family. SensorLM consists of four variants with increasing model sizes. Architectural details for the sensor encoder, text encoder, and multimodal text decoder provided.
Sensor Encoder
Text Encoder & Multimodal Decoder
Sensor Text
Model
layers
MI.P
params
layers
layers
ML.P
# params
hidden
heads
SensorLM-S
12
512
512
12M
128
SensorLM-B
12
3072
114M
12
3072
191M
768
12
SensorLM-L
24
4096
404M
24
4096
519M
1024
16
SensorLM-XX
40
5632
1.27B
40
5632
1,45B
1408
16
B Implementation Details
SensorLM Model Architecture
As described earlier, SensorLM consists of decoder (Fig. 3). The input to the sensor encoder is a matrix of shape [26 features x 1440 minutes], representing one-day windows of sensor data minutely resolution. The sensor encoder is built using ViT-2D backbone with a 2D patch size of 10), producing 1872 tokens.
encoder, text encoder, and multimodal text
The text encoder and multimodal text decoder follow standard transformer architectures. The multimodal text decoder attends to the output tokens of the sensor encoder via cross-attention [39], enabling encoder-decoder setup for caption generation. For contrastive learning, representations from the sensor encoder and unimodal text encoder. Both representations are average- pooled, passed through projection heads, normalized, and then used to compute the contrastive loss Leon between sensor embeddings 8; and text embeddings v;,
The SensorLM family consists of four variants with different (increasing) model sizes: SensorLM-S, SensorLM- B, SensorLM-L, and SensorLM- XL. Architectural specifications for each variant are provided in Table 11.
B.2 Pretraining Details
All models are trained using the SensorLM pretraining objective (Lsemsorix) on Google v6 TPUs for 50k steps. We use a batch size of 1024 sensor-text pairs, with Ason = Aeap for main experiments. The temperature 7 for the contrastive loss is set to 0.01. Optimization performed using Adam optimizer with 3 = 0.9 and B2 = 0.95. A cosine warm-up schedule is applied for the first 10% of training steps, followed by linear decay of the learning rate zero.
B.3 Self-Supervised Learning Baselines
We compare SensorLM against the following sulf-super learning (SSL) baselines:
SimCLR [5] is widely adopted contrastive learning framework that aligns representations from augmented views of the same input while repelling those of different inputs. It relies solely
17

---

Table 12: Hyperparameters self-supervised learning baselines. We provide settings used MSN [2].
DINO [3], and SimCLR [5]. A single row value indicates that the setting was applied to all methods.
Configuration
MSN [2]
DINO [3]
SimCLR [5]
training steps
50,000
warmup steps optimizer
2,500
AdamW [18]
opt. momentum [,
[0.9, 0.99)
base learning rate
0.001
0.004
0.001
batch size
1,024
weight decay
0.0001
gradient clipping
3.0
learning rate schedule
linear warmup & cosine decay 26 (sensor) 1440 (minutes)
data resolution
on contrastive loss and unlabeled data, and has proven effective for both visual and text-based representation learning.
MSN [2] integrates contrastive-basee pretraining with masked image modeling, MSN aligns the representation of a masked view with that of the unmasked input by processing only visible patches. This approach enhances scalability with Vision Transformers and yields semantically meaningful features, achieving strong performance low-shot classification tasks.
DINO [3] employs self-distillation strategy using teacher-student framework. The student network is trained to match the teacher's representations across different augmentations. DINO has demonstrated robust and generalizable feature learning across various domains.
All three baselines follow sugmentation-driven training paradigms. We apply standardized set of time-series augmentations (e.g., j jittering. scaling. time flipping) based on prior work [17,27, 28, 43]. Each augmentation is applied independently with probability of 0.5. Jittering adds Gaussian noise with zero mean and standard deviation uniformly sampled from [0, 0.5]. Scaling multiplies the input by random factor sampled from 11.1, Notably, we omit scaling for DINO due to convergence issues observed during training.
All baseline models are pretrained from scratch under training settings listed in Table 12. For consistency, all methods use the same ViT-2D backbone as the sensor encoder in SensorLM, with 2D patch size of (2, 10).
B.4 LLM Baselines
For zero-shot classification, cross-modal retrieval, and sensor captioning, compare SensorLM against two LLMs: Gemini 2.0 Flash [29] and Gemma-3-27B [30], by formatting sensor data as tabular input.
Gemma-3-2777 [30] is a state-of-the-art open-weight large language model developed by Google DeepMind. Trained high-quality, multilingual corpus, it is optimized for instruction following and reasoning tasks. We use the instruction- centiidn-tunedddde variant, which exhibits strong performance in both zero-shot and few-shot scenarios.
Gemini 2.0 Flash [29] is lightweight, high-performance member of Google's Gemini multimodal model family, designed for attency-sensitie applications. supports vision-language reasoning and offers strong cross-modal capabilities with low inference latency.
For both baselines, we subsample the sensor data fit within the context length limitations of these LLMs.
18

---

Table 13:
Definition the activity concept classification tasks.
Class
Activities included
Prompt example
Task: Environmental context
Elliptical, Treadmill, Spianing, Weightlifting Yoga. Core training, Pilates, Stairelimher, Dancing, Indoor climbing. Kickboxing
Indoor Sports
User perforned Indoor Sportz.
Bike, Run, Hike, Tennis, Mountain Bike, Rollerblad- ing. Golf
Outdoor Sports
User perforned Dutdoor Sports.
Water Sports
Swim, Kayaking, Surfing, Paddleboarding
User perforned Water Sports.
Winter Sports
Skiing, Snowhoarding
User parformed Winter Sports.
Task: Locomotion vs. stationary
Locomotion Exercise
Walk, Run, Hike, Stairelimber, Treadmill, Elliptical
User perforned Locomotion exercise.
Stationary Exercise
Weightlifting, Yoga, Core training, Pilates
User parforned Stationary axercise.
Task: Cardio vs. strength training
Cardio Exercise
Elliptical, Treadmill, Spinning, Staizelimher, Run
User performed Cardio exercise.
Strength Exereise
Weightifting, Core training. Indoor climbing
User perforned Strength exercise.
B.5
Zero-Shot Classification
B.5.1 Task definition for zero-shot classification
We formulate six zero-shot classification tasks related to activity recognition. We compare SensorLM with representative LLM baselines across main activity classification, activity-related concept classi- fication, and fine- traiigd recognition, aiming to evaluate the effectiveness of SensorLM in diverse human activity analysis scenarios.
Main activity classification encompasses 20 activities. For each activity, the number of samples is indicated in parentheses. These include: Walking (881), Bike (860), Playing Sports (905), Running (793), Aerobics (910), Elliptical (887), Spinning (866), Weightlifting (842), Swimming (882). Hiking (848), Playing Tennis (821), CrossFit (902), Pilates (855), Stairelimber (863), Dancing (852), Indoor climbing (854), Golf (710), Skiing (420), Snowboarding (167), and Kayaking (212).
For aclivity-related concept classification, we define three sub-tasks. The first one, "Activity by envirommental context", comprises four classes: Indoor Sports, Outdoor Sports, Water Sports, and Winter Sports. The second task, "Cardio vs. strength training", distinguishes between classes: Cardio exercise and Strength exercise. Similarly, for the third task, "Locomotion vs. stationary". have two classes: Locomotion exercise and Stationary exercise. The activities included in each concept category are summarized in Table 13.
Finally, we introduce two fine-grained recognition tasks. The first task, "Fine-grained recognition (Gym cardio)", differentiates among Elliptical (887), Treadmill (884), Spinning (866), and Stair- climber (863). The second task, "Fine-graines recognition (outdoor sports)", involves classifying among Hike (848), Mountain Bike (673), Skiing (420), and Snowboarding (167).
B.5.2 Implementation for zero-shot classification
Implementation details for SensorLM. We employ prompt ensembling [25] using a diverse set of rewritten prompts that mirror the structure of pretraining captions. For each label class, we compute the average embedding from the text encoder across all prompts. Zero-shot classification is performed by selecting the label whose average text embedding is most similar to the sensor embedding from the sensor encoder. Examples of the prompts are provided in Table 14; we use a total of 30 prompts.
Implementation details for LLM baselines. For zero-shot classification using LLM baselines (i.e., Gemini 2.0 Flash [29] and Gemma-3-277 [30]), we employ a prompt format illustrated in Table 15. The prompt instructs the model to predict a class label based on the provided sensor data, which is formatted as tabular input. Additionally, we also fine-tune Gemini 2.0 Flash [29] on our dataset using supervised activity labels, following standard supervised fine-tuning (SFT) procedure.
19

---

Table 14: Example templates in zero-shot prompt ensemble.
Zero-Shot Templates
1. period of Run was observed during the session.
2. Detected a phase of Run.
3. Data shows Run took place
4. The main action was Run Run was detected during the observed period.
5.
Table 15: Zero-shot classification prompt used for LLM baselines.
Sero-Shot Classification Prompt
*48 Dvarall innrrcction: Your joh is to idantify user activity class by analycing a given day of Pithit data. Available Activity Classes: (clawu_latt
Sansor Description
$5808 Heart:
RR: Mean of the instantameoux heart beata per sinute. Calenlated minute vindow. Unit: Bexta/Nin br_mh__ratt
per minute during periods Unit:
arv_re__0th__pp perclpparenttleeeeeee
percencile BR
ailliseconda S-ainute
intervals. linit: Henc:
hrv_rr_ pttrrr_0tt_ppp perlspperentileeeeee The 20th parcentile of the R8 intervala silliseconda for Seximute vindows with valid
hre_rr_redian: nedian interval millizeconds
S-nimbe vindows valid intervala. Meed
rr:: Shannos antropy of the 88 intervala for 5-ninste windows with valid 88 intervala. Guit: Nata arv_shannon,co entropy-rred Shanzon entropy the Ml interval differences for 5-mimate vindows with valid IiI
intervula nilliscosda E-ninute vindova pemn_paccnntill. 0506: Standard deviation of BR intervala is nillineconds for S-nimote windows with valid BR
twan aquared standard
valid IIl intervals. Unit: Keec
Dait:
essve Activity:
staps: Number of stepa calculated noer mimnts vindow. Dnit: Stago
jerk_auto: Batio of
autocorrolation squared magei
3-axie principal
Unit:
log_energy: log
alog..mergy
covariance: Eatirate of condit munber 3-arix naveriance astrin. acovariance
jeg.onergy ratio: Log of ratio of eun of energy in 1st 3-axis principal component over energy of 3-axis root nean aquared ragnitude. Unit: pera_croxxing.atd Standard deviation
ratio
principel corposent, Seconda zero, www.errr..ggrre..... arg: Hean of the time betseen zero crossings of 1st 3-aris principal corponent in seconds, Unit: Seconde aris_rean: log of the mann aquare root of the aquared X 2 azen of the accelarsmeter. Unit: a.t.
crozsings
aitis,etd. Standard deviation aitineter readings liectopascale.. Unit:
equared zagnitude,
assuu Sleep:
coofpcccef::t::: Stn of 3 3-aris tAx nin range, binned into 16 log scaled bine. Uhit: aSloop
e*eve EDA:
pSierana
jeads_contact, contact_counte: Number of tixes the skin concuctance senser electrode leads make contact (li2oly related to signal quality). Unit: Counte
coda, slope_rra__
StL ekin, www.erettrreeecccooooo clope: Chango temperature degrees por
Intraniaute
pSiemens/Min
_ rrtt_terperrtture: Hean skin temperature in degrees Celetus calculated over ninute vindow. Unst: *c
vriet,
Here
the predicted
bebind choice:
ppredicted_claso". spredictod_clases *clane, "rabe__proaiilttteee""
"class gprobabllity.of.oo of _flssss__
"clace 3": sprobability, of ot. ot_sssss_3,
"class 4": of
)). "ressniaa"; creascoing?
118 Nos, it is your turn. Bore day Fiebit data for user in cav fornat. Each you is vindow of 10-minute values for corsespoading festure Analyse Fitbit thoroughly make prediction. (sensor_dats data)
tht Your output in JS0N forzat containing the predicted class and class probabslities for all classes. The class predicted chould be Araslable Activity classes 11st used. Also give reason your prediction:
probabilitios chould
earlier.
other classes apart
20

---

Table 16: Zero-shot cross-modal retrieval prompt for LLM baselines.
Zero-Shot Cross-Modal Retrieval Prompt
ttf Overall instruction: You vil1 be given a day of user Fitbit data and a set of 100 captions. Your job 1s to analyze the Fitbit data and colect top 10 captions that descrabe the Fitbis data the best.
Description
tiree Heart::
isk: Noan of the instantaneous heart rate in beate per minute. Calculated over 1 minute vindow. Unat: Beate/Nin beate por during
Rean
rest. Voit: Beata/Kin
arv_re_00th, pwrcettecentt.....ee.. intervals Unit: Heer
percentile
brv.rr 2r__0tt__pee potapprrenttl.. aeann The 20th perconcile of the BB intervale in aillicecond for minute vindows with valid
intervale. Unit: Hisc
arv_rr_ :
ailliscouds
B-nimute vindous valid intervals.
brv, channcn, entropy rr: Shannon entropy of the Ba intervals for 5 minute windows with valld BR intervals, bait: Nats bry,
rrd: Shanzon entropy of the BB interval differences for b t-ntaate vindous with valid RR intervals. Unit: Nats
unesd_percestile. percestile. 0583: Root nean equared standard
ninute vindows
volid Hill intervale. Unit: Keec
sdmn, perceatileeeeeee.. 0E96: Standard deviation RR intervals is nilliceconds for 6-ninuce vindous with valid BR intervale. Unit: Kees
ssoce Activity:
steps: Nusbor of steps calculated over a 1 minute vindow. tait: Stops
jerk_auto: Batio of lag=1 autocorrelation to energy in 1st axis principal companent, Unit: alog, energy of covariance: Ratizate of condition
Whit: energ
covariance natzix. acovariance
energy retio: Log of ratio of sun of energy in 1st 3-axis principal component over enorgy of d-aris root nean aquared nagaitude. Dait: alog
ratio
zoro_croccing std: Standard doviation sero_crossing ing wvg: Hean
principal corpenent, Seconds betvwen crossings of Beazis principul corposent seconda. Beconds axia zean: Log of the nean aquare root the aquared axea of the accelervnster: Uait; a.u.
let
aitin etd: Standard deviation of altineter readings in Hectopascals. Dnit: Rectopascale
kurtosia: Kurtosia
aquared sagsitude. Vait:
sssee fleep:
sleep_ coefficient: Sun of 3 3-axis nax nin range, bizned into 16 log scaled bins. Unit: aBloop
sssed EDA:
leadi_contact contectttttnttsss.......tttt Number conductance value
windov. Wait:
the conductance awnaur electrode leads make contact (likaly related
signal quality). thit: Counta
coda_elope_raal xiceo_ xicro_piesons slesons: Intraniaute clope of StL values. Unit: pSiezens/Min sinute
tesperature weist, Cespertrrrr Houn tesperature degrows Calaius calculated over vindow.
akin_terparature.
Change
ss60 Harw is exanple
structure with the predicted 10 captions:
"dirat:: coaption_100
"second":
cosptios_10 ID>,
"third": <caption, ID>,
*dourth"; <raption_1>>, Iosp
"sixch"; <eaption_ID>,
"swventh:: cosption_
"nighth": <paptios_TD>,
"ninch" <oapile_ID>>,
"tanth": cnaption_ID> 10>
is your turn. a day data user cav format.. rou is vindow 10-ninute average valusa for the correapooding feature zutric. Aualyse the Fitbit data thoroughly and choose captions from list which best describe the Pisbit data. _data)
Here is the set of 100 captions:
(captimn
outgut in J8OM fornat containing the predicted top 10 caption ids:
21

---

Table 17: Caption generation prompt used LLM baselines.
Caption Generation Prompt
ttt Overall instruction: You vill be given a day of user Fitbit data. Your jeb is to analyze the Fithis data and generate caption containing comantic information. Sezantic information aime capture the high level neaning and context embedded in pensor data, cssmbtially interpreting shat the individual night be doing experiencing-
tte Sencor Description
evove Heaart:
BS: Mean instantaneous
beate per zinute. Calculated beate por minute during persods rest. Unit: Beate/Mia
vindow. Beate/Nin
wt.ttrret__nean Mean
brv.rr_ wwwth.prrenttll..m....
The 60th percmnile of the IR intervale in aillieecode for 5-xinute vindows with valid B8. intervals. Unit: Mooc
The 20th percentile the BR intervale milliceemde for 5-minute vindoue with valid incervals. Unit: Moec
arv,rt modian: The pedian Ra interval in millicconds for nimure vindous with volid Ra intervale. Unit: Keec
ary chammoa.cntropo xr: Shannon entropy of the ill intervale for b-nimate Mindows with valsd HR intervale. Unit: Nate vindous with
wea.hhaannsscc cutropyy Shanzon entrepy the interval differences for
antervale. Unit: Nate
mesd_percenttle www.peeettee.eSee0 Root nean squared standard deviation of Bil intervale in millireconds for 6-mimube windows with valid AB intervals. thit: Ksec
Standard deviation intervalc is millieccads for 6 vindows with valid BR intervala. Dait: Maec
percentiie
stree Activity:
stops: Number of steps calculated over 1 minute vindow. Unit: Stepe
in 3-axis zoot nean equared nagaitude. covariance: Rosixatee of condition number for 3 axis covariance natrix. Unit: acovariance
jark_autor Batie of lage1 autocorrolation
principal companent, Dait: alog_ energy
log. energy: Log of
alog. energy
2og, energg__ration ratio: Log of ratio of sun of energy in 1st 3-axia principal conpesent ovez energy of 3-axia root mean equared nagnitude.
iard_crossing atd: Standard sog_ecoory__atta
betowen saro crossings fat B-azia principal corporent, Unit: Saconda co
zoro croo_croiingg.ee avg: Bean of the tise betooen zero crossings of ist 3-axis principal cosposent in seconda. Wait: Secouds axis_ zean: Log of the mean equare root of the equared X & axes of the accelervnster, Unit: a.u.
aitia_atd: Standard deviation of altiseter realings in Hactopazcala.
Rectopaacals
kurtosia: Kurtosis of B-axis accolervaster root nean squared nagnitudo. thit: a.u.
tiree Sleep:
ssre8 coufficiont: Sas of B-azia sax-nin
bisned into 16 log-acaled bina. Unit: dEleup
eds, level_real: Yean tonic akin conductancs value in micro Sissana over ninute windou. Unit: pBierana leada cauds_cotact,.... counts:: Yusber of tises the skin conductance sunsor electrode leads make contact (likaly related to signal quality). Whit: Counto
cada www.ppeeeell.irroooooo reel__iico_aiieeae akin, conperature, alope: Change
Intraninute slaps of tesperature in degrees Celaius pur minate ovez minute window. Unit: *C/Kia grist, resporrtttssres Moan skin teaperature in dogreos Coleius calculated over a 1 minute vindow. Unit: *C
joinsans/Min
Here
ezusple captions:
Essmpla i: Parind of Outdoor Bike noted from sinute 912 until 917, Outdoor Bike occurred fros sinuts 619 to 628.
Outdoor Bize recorded within the 661-650 ninute User engaged in Outdoer Bike berveen minute 923 and $37- Outdoor
480 through Daer engaged Dutdoor 730. Fron ninute to 456, the user had pariod Dutdoor Bike. Outdoor Bike tonk place during sinutea 572 through E64.
during
Esmaple Obaerved Valk spanning simutes 402 408. lestance of Valk identified fros sinute 663. paried
contimuus Valk phase from sizate 692 bwtseenn ninutes 851 and 888. Valk was recorded betveen rinsta 785 and 822, à Slaap pariod betseen minubaa 1546 and 1440.
Detection of Valk activity besseen minute 1021
Rearpla An instance of Kilk vaa identified fron minute to 289. instance Kal's was idantiffied from minute 799 to 828, A Halk period between simutes 279 and 309. Bleep occuzee frux sinute 1803 to 1440.
Reample Malk state abserved from ninute up 505. Sleup occurred from simute 400. The persos logged their sood Exclted at minute 1204. The person registared a noad of Prostrated at minute 773. At nimata 773, the individunl 'n feeling in Frustrated.
observed
up
Bike
mimutes 1051 1064, Butdoor
446 Nox, it in your turn, Hare is a day of Ficbit data for user in cav foreat. Each rou is window of 10-misute valies corresponding festure metric. Analyse the Pitbst data thoroughly and generate caption which Pithit (owmaor data]
sSu Your outgut:
22

---

B.6 Few-Shot Learning
For both few-shot learning and linear probing tasks, the sensor encoder of SensorLM is frozen. A single linear layer" is trained using a multinomial loss with balanced class weights (weights inversely proportional to class fre- quencies in the training set). This same approach is applied all SSL baselines for consistency.
Table 18: Class distribution for the Anxi- cty and Hypertension prediction tasks.
Task / Label
Train
Test
Anxiety
positive
55,030
34,749
For linear probing, we use the full training set (statistics provided in Table 7). For few-shot learning. we randomly sample 5-50 training examples per class and repeat each experiment five times with different random seeds. Table 18 summarizes the class distribution for the "Anxiety" and "Hypertension" tasks.
negative
96,316
55,437
Hypertension
positive
36,349
23,353
negative
114,997
66,833
Total
151,346
90,186
B.7 Cross-Modal Retrieval
Implementation details for SensorLM. To evaluate zero-shot cross-modal retrieval, we use top-k recall (R@k) @k) for & {1, 10). This metric measures the proportion of queries for which the ground-truth pairing appears within the top k retrieved results. Retrieval is conducted in both Sensor - Text and Text + Sensor directions. Cosine similarity between sensor and text embeddings is used as the scoring function to rank candidates. For each query, recall is computed by locating the ground-truth index within the ranked similarity list.
Implementation details for LLM baselines. For zero-shot cross modal retrieval using the LLM baselines (i.e., Gemini 2.0 Flash [29] and Gemma-3-27B [30]), use prompt format illustrated in Table 16. The prompt instructs the model to retrieve the top- 10 most relevant captions from a set of 100 candidates, based the provided sensor data formatted tabular input. We evaluate only in the Sensor - Text direction, as Text Sensor retrieval is not feasible due to the context length limitations of the LLMs.
B.8 Zero-Shot Generalization to Unseen Classes
To assess SensorLM's generalization to novel activities, we conduct a case study involving pretraining on a dataset comprising data only containing 20 activities, followed by testing exclusively set of 9 previously unseen classes.
The 20 activities used in the pretraining set include: Bike, Playing Sports, Running, Aerobics, Ellipti- cal, Weightlifting, Swimming, Hiking, Playing Tennis, CrossFit, Core training, Pilates, Bootcamp, Indoor Climbing, Golf, Kickboxing, Skiing, Rollerblading, and Kayaking.
The 9 unseen activities evaluated are: Snowboarding (167), Mountain Bike (673), Racquetball (19), Yoga (879), Paddleboarding (51), Badminton (109), Dancing (852), Calisthenics (61) and Basketball (127).
For each unseen activity, we evaluate SensorLM's zero-shot performance using a one-versus-all classification setup. The primary evaluation metric is the Area Under the Receiver Operating Characteristic Curve (AURÔC). with 95% bootstrap confidence intervals (CIs) computed using 1000 resampling iterations with replacement [4].
B.9 Caption Generation
Implementation details for SensorLM. For sensor captioning, we observe that increasing the number of layers in the multimodal text decoder improves captioning performance. Accordingly, train a SensorLM-B model with 12 decoder layers (compared to the default 3) using semantic captions during pretraining. During inference, captions are generated autoregressively from sensor data using a [START] token.
"nttps://sciktt-leet wetprteeeearrrrortrrrrrrreeeee wtt.sttable/oouule........eee hat.ll/ssggennrttte/kleeeeee monear_modell. Logisticheggression..i html.
23

---

19
18
19
84
89
17
19
13
88
18
199 15
12
89
F4
13
19
83
12
10.
14
19
19
19A
18
Aardiica
12
32
52
24
12
12
Aanobica
13
122
12
Sph-hing
12
18
Sphriigg
12
14
53
13
19
18
12
12
12
52
150
17
72
17
17
17
19
12
10
R
R
Intbrr Streiingg
12
12
22
2eing
12
4.5 2
(a) SimCLR [5]
(b) SensorLM
Figure 9: Confusion matrices for linear probing activity classification. Comparison of SimCLR [5] and SensorLM-trainee sensor encoders the downstream Activity datasct for 20-class classification. The y-axis represents ground truth labels, and the x-axis shows predicted labels.
Table 19:
Linear probing results. We use all data for task-specific lincar probing, AUROCT is used as the metrie. Best results of each column are in bold and the second best are underlined.
Task
Hypertension
Anxiety
Activity
SimCLR [5]
0.56
0.64
0.68
DINO [3]
0.56
0.62
0.66
MSN [2]
0.58
0.66
0.72
SensorLM
0.60
0.65
0.94
Implementation details for LLM baselines. For caption generation using LLM baselines (i.e., Gemini 2.0 Flash [29] and Gemma-3-27B [30]), we use prompt templates shown in Table 17. These prompts instruct the model to generate semantic caption based on the provided sensor data, formatted as tabular input.
C Additional Results
C.1 Few-Shot Learning& & Linear Probing
Table 19 presents task-specific linear probing results for SensorLM compared to several SSL baselines: SimCLR, DINO, and MSN. All evaluations use the full available training data, and performance is measured by AUROC. SensorLM achieves competitive or superior performance across all tasks, with the highest AUROC for " "Hypertension" (0.60) and "Activiry" (0.94). For "Anxiety", SensorLM (0.65) performs comparably to MSN (0.66), the top- nop-performing baseline for that task. These results highlight SensorLM's s robust representation learning for downstream applications.
Fig. 9 shows confusion matrices for SensorLM and SimCLR, illustrating the improved discriminative capability of SensorLM embeddings across diverse activity classes.
We also include results on two regression tasks: predicting Age and BMI from one day of sensor data. Using a frozen sensor encoder and a linear regression probe", we report Mean Absolute Error
learn, caaeenn.rr///tae https/aallleeooleeeeeeeeeeeetete
linear_moder model.
LinearRegreession.htll html
24

---

Table 20: Performance SensorLM and baselines on regression tasks.
Age
BMI
Metrics
MAE
MAPE
MAE
MAPE
SimCLR
9.18
21.03
5.85
19.57
DINO
9.63
22.03
5.97
19.97
MSN
9.36
21.39
5.84
19.53
SensorLM
9.18
20.81
5.75
19.17
Table 21: Zero-shot cross-modal retrieval. We evaluate moos-modal retrieval held-out dataset containing 40,000 examples, testing performance across varying retrieval set sizes. Se achieves consistently strong retrieval performance across settings. For baseline LLMs, most retrieval tasks are infcasible due to context longth limitations (marked as "-").
samples
5,000 samples
40,000 samples
Metrics
R@I
R@5
R@10
R@5
R@10
R@1
R@5
R@
Sensor Text:
Gemma-3-27B [30]
1.0
5.0
10.0
Gemini 2.0 [29]
13.0
SensorLM
100.0
100.0
100.0
98.2
99.4
99.6
96.1
98.7
99.0
Text Sensor:
Gemma-3-277 [30]
Gemini 2.0 [29]
SensorLM
100.0
100.0
100.0
96.7
99.2
99.4
90.0
96.9
98.1
(MAE) and Mean Absolute Percentage Error (MAPE). As shown in Table 20, SensorLM improves performance compared to SSL baselines.
C.2 Cross-Modal Retrieval
Table 21 presents the complete results zero-shot cross-modal retrieval, comparing SensorLM to LLM baselines. Eyaluations are conducted on query-target sets of size 100, 5k, and 40k, using Recall@1 1, @5 and Recall@) 10 (R K). These metrics capture the proportion of queries where the correct target appears among the top K retrieved results.
SensorLM consistently demonstrates strong retrieval performance across all settings and directions. Notably, it achieves perfect 100% recall at R@) I, R@5, 5, and R @ 10 on the 100-sample benchmark for both Sensor - Text and Text - Sensor retrieval. Even at the 40k scale, SensorLM maintains high accuracy, with R @ scores of 96.1% for Sensor + Text and 90.0% for Text + Sensor.
In contrast, the LLM baselines largely unable to perform most retrieval tasks due context length limitations, marked as in the table. Where results are available (mainly for the 100-sample set), their performance is significantly lower than SensorLM. These findings highlight SensorLM superior capability cross-modal retrieval, where general-purpose L.MS struggle without domain- specific architectural adaptations or training.
In addition to the semantic caption example in Fig. 4, Fig. 10 provides qualitative evidence of SensorLM's s ability to retrieve accurate statistical summaries from raw sensor inputs. SensorLM re- trieves the ground-truth caption as well as similar alternatives that match sensor statistics (highlighted in yellow), such means, standard deviations, and extreme values. In contrast, unrelated captions show low similarity scores, reflecting SensorLM's understanding of the statistical structure of the data.
Furthermore, Fig. illustrates SensorLM's capacity for retrieving structural descriptions from sensor data. It accurately identifies the ground-truth structural caption and retrieves similar captions
25

---

Ground Truth
Heart rate meen, max, min, std ane 70.5, 1435, 447, wrist
temperaturre atures ditla achibiiss of sld 2.0, and ecthnerm
Wakass are 24.5 35.2.
TEMP ACCE)
Similar Caption
48.3
mentiss moan maximum minimum 47.0,
age temperatures extromee (max) 19.0 (min),
22.7 30.3
Dissimilar Caption
240
480
720
960
1200
5C:-0.05
heart rate presents mean of 98.7 maxiuum of 152. minimum of 63.0) and a sid of 20.5, wrist temperatures bits mean of 20.8. with peak and minimal values reaching 30.8 and 8088 and std D.8.
Time (minutes)
Zero-shot sensor-to-text retrieval qualitative example for statistical captions. We show similarity scores (SC) for the correctly retrieved ground truth, top similar caption, and dissimilar caption. Green highlights correct statistics; Yellow indicates close matches; and Red denotes distinctly different stasisties. addition correctly retrieving the ground truth, SensorLM also demonstrates semantic understanding by assigning high similarity scores to numerically closer candidates.
Figure 10:
Heart rate indicates incressing trend spanning minute 200 450. Heart rate is abserved to be decreasing between minute 1240 and
0.66
TEMP ACCEL HR
66.2
1440 wrist temperature increases spanning minute 1240 1440.
Similar Caption
96.7 48.3
0.45
Heart revesls decreasing trend during minute
1640. Wrist temperature incressing from minute 860 minute 1120. wrist temperatures increasing from minute 12.0 to minute 1660.
Dissimilar Caption
Time (minutes)
720
SC2-0.26
decr eising temperature indicates Increasing trend spanning ute
writ temperature trand
Over minute
heart rabe wys eecreasing
Zero-shot sensor-to-text retrieval qualitative example for structural captions. similarity scores (SC) for the correctly retrieved ground truth, a top similar caption, and a dissimilar caption. Green highlights correct patterns and time frames; Yellow indicates partial matches; and Red denotes incorrect patterns and time frames. In addition correctly retrieving the ground truth, SensorLX also demonstrates semantic understanding by assigning high similarity scores partially correct candidates.
Figure 11:
show
Table 22: Sensor caption generation results. We compare SensorLM with LLM baselines using commonly adopted metrics (e.g., BERTScore, METEOR, ROUGE) for natural language generation.
BERT Score
Metrics
Precision"
Recall
FI
METEOR
ROUGE
Gemma-3-27B [30]
0.82
0.85
0.83
0.19
0.11
Gemini 2.0 [29]
0.86
0.87
0.86
0.26
0.20
SensorLM
0.93
0.91
0.92
0.33
0.40
that describe temporal dynamics, such as trends and spike events. This demonstrates SensorLM 's ability interpret and express dynamic temporal features within sensor signals.
C.3
Caption Generation
Table 22 evaluates sensor caption generation performance, comparing SensorLM against LLM baselines. We report standard Natural Language Generation (NLG) metrics: BERTScore (Precision, Recall, FI), METEOR, and ROUGE, which collectively assess semantic similarity, fluency, and content overlap. SensorLM consistently outperforms both LLM baselines across all metrics.
We also present three qualitative examples in Fig. 12, which compare sensor caption outputs from Gemini 2.0 Flash and SensorLM against ground truth references. The examples focus on semantic caption generation from wearable sensor inputs. Compared to the baseline, SensorLM demonstrates stronger semantic understanding by generating coherent event descriptions with accurately localized timeframes.
26

---

Ground Truth
Gomini 2.0 Fissh
SensorLM
Outdoor Bike was detected between minutes 886 and 890. A Outdoor Bike event was detected between minutes 673 and 876, A Sleep event detected between minutes 1436 and 1440.
Observed periods of Walk between minutes 430 and 500, as well as from minute 690 to 710, and again from 940 Sleep detected from minute until 1400.
Outdoor bike delected between minutes 873 end 878, Walk detected between minutes 876 and 689. Sleep between minutes
A Sport event was detected between minutes 793 and 897. Sleep recorded within the 1268-1440 minute
A Walk period between minutes 370 and 380, 420 and 470, 530 and 640, and 770 and 830. Sleep period between
Sport
Sleep
Spinning indicated between the and 923 minute marks. Sleep took during the minutes through 642.
Observed periods of high heart and activity. potentially indicating exercise exartion. between minutes and 1050. sleep period between minutes and was recorded between minute 650 and 670.
spinning event was detected between minutes 899 sleep period between minutes 42 656.
Qualitative comparison of sensor caption generation between Gemini 2.0 Flash and SensorLM against ground truth. We show various examples of semantic caption generation from wearable sensor inputs. Compared baseline, SensorLM demonstrates stronger semantic understanding by generating coherent events with near-correct localized timeframes.
Figure 12:
Table 23: Comparisons of SensorLM caption versions. We evaluate different combinations of caption types used during pretraining. AUROC* is used the metric. Best results of cach column are in bold and the second best underlined. Default setting used in the main experiments marked in gray.
Zero-Shot
Linear Probing
Retrieval (40k)
Caption Variant
Activity
Activity
Anxiety
Hypertension
Recall @
statistical
0.51
0.76
0.67
0.63
1.00
structural
0.50
0.78
0.63
0.59
1.00
semantic
0.71
0.95
0.65
0.60
0.89
statistical semantic
0.66
0.84
0.68
0.62
1.00
structural semantic
0.84
0.94
0.65
0.60
statistical structural
0.49
0.79
0.67
0.63
0.64
statistical structural semantic
0.66
0.86
0.68
0.63
0.90
C.4 Complete Ablation Results
Here we present the complete results for the ablation studies introduced Sec. 5.3. Table 23 examines how diffferent combinations of sensor caption types (statistical, structural, and semantic) affect SensorLM's performance across downstream tasks. Semantic captions are critical for zero-shot activity recognition, while combining them with structural captions further enhances performance across tasks, including cross-modal retrieval. The results also reveal trade-offs: statistical captions improve performance "Anxiery" and "Hypertension", but slightly reduce accuracy on "Activiry".
Table evaluates different architectural variants of SensorLM during pretraining. SensorLM (CoCa) consistently outperforms single-objective variants (SensorLM (CLIP) and SensorLM (Cap)) across key metrics for both zero-shot classification and linear probing, validating the benefits of integrating contrastive and generative objectives.
27

---

Table 24: Comparisons of SensorLM architectural variants. compare different choices used during pretraining. Default setting used in the main experiments is marked gray...
Zero-Shot
Linear Probing
Metrics
AUROCT
F11
Balanced Acc.
AUROC!
F11
Balanced Acc.
SensorLK (CLIP)
0.83
0.29
0.31
0,93
0.53
0.55
SensorLM (PigLIP)
0.78
0.24
0.26
0.87
0.38
0.40
SensorLX (Cap)
0.55
0.01
0.05
0.90
0.32
0.52
SensorLX (CoCa)
0.84
0.29
0.32
0.94
0.57
0.60
Table 25: Comparisons between UniMTS and SensorLM. Performance on zero-shot and linear probing (LP)
settings across activity recognition and metabolic health tasks.
Zero-Shot
Linear Probing (LP)
Model
Activity (20-class)
Gym Cardio
Outdoor Sports
Activity
Anxiety
Hypertension
UniMTS [44]
0.71
0.73
0.65
0.94
0.65
0.58
SensorLK
0.84
0.76
0.86
0,94
0.65
0.60
C,5
Comparisons to Existing Sensor Foundation Models
UniMTS [44] proposes a unified pre-training method for motion time series that synthesizes sensor data from human skeletons and uses contrastive learning to align it with natural language. To compare with UniMTS, we implement their captioning and pretraining method on our sensor-text data zero-shot baseline. Specifically, it employs CLIP [25] objective and treats activity descriptions as language imput, corresponding the semantic caption our framework. We adapt and train UniMTS under our experimental setup and compare with our results in Table 25.
The table confirms that SensorLM outperforms UniMTTS across all zero-shot evaluation tasks. To further understand the effect of hierarchical captions on this existing work, we observe from the table that: (1) Adding structural captions improves zero-shot AR performance, aligning with our caption ablation findings.
(2) Including statistical captions improves metabolic prediction tasks, though with slight trade-off in AR performance, again consistent with trends discussed previously. These results support both the quality of the dataset and the value of the hierarchical captioning strategy in improving generalization across diverse tasks.
C.6 Exploration of Loss Weights
As our work introduces a modular and generic pretraining framework, it straightforward to attach, remove, reweight objectives. This modularity allows systematic ablations, rapid integration of new components, and reveals design tradeoffs unexplored prior image-text studies. Here we sweep the relative weights of the contrastive (Aco)) and captioning (Aeup) losses beyond fixed settings by CLIP (Aem = 0) and CoCa (Agon Acp = 1). We find clear task-specific trade-offs: a 2 balance improves zero-shot activity recognition, whereas 5 yields the best metabolic-health prediction. These results indicate that careful tuning, enabled by our framework, yields measurable gains.
C.7 Robustness to OOD Data and Missingness
Here stratify our testing set to study the performance of SensorLM on data from OOD devices and with missingness and missing modalities. All reported results in the main paper already include these OOD and missing-data cases.
C.7.1 Inttrf-DDstrrbbution Devices
Our pretraining set contains 18 smartwatch and fitness-tracker models, while the metabolic down- stream dataset has additional 17 device types. As result, we able demonstrate generalizabil-
28

---

Table 26: Exploration loss weights in pretraining. We sweep the relative weights the contrastive (Dem) and captioning (Aau) losses beyond the fixed CLIP (Acon establishing different tradcoffs.
0) and CoCa (Auen
Aeap = 1) settings,
Weights
Zero-Shot
Linear Probing
Acon
Aeap
Activity (20-class)
Gym Cardio
Outdoor Sports
Activity
Anxiety
Hypertension
0.2
0.82
0.77
0.82
0.92
0.66
0.62
0.84
0.76
0.86
0.94
0.65
0.60
0.85
0.78
0.85
0.93
0.65
0.60
Table 27: Metabolic health prediction results on in-distribution (ID) and out-o--distributttinn (OOD)
devices. AUROCT is used for classification tasks (Anxicty, Hypertension) and MAE for regression tasks (Age, BMI)
Split
Anxiety (AUROCI)
Hypertension (AUROC))
Age (MAE-)
BMI (MAE*)
ID
0.65
0.61
9.24
5.74
OOD
0.72
0.55
8.55
5.79
Table 28:
Effect of missingness on SensorLM performance. We report AUROCT across different missingness levels and sensor modality settings.
Varying missing percentage
Missing %
Activity
Anxiety
Hypertension
<40%
0.94
0.68
0.62
40-60%
0.94
0.65
0.59
60-80%
0.91
0.65
0.65
Specific sensor modality missing
Modality Missing
Activity
Anxiety
Hypertension
w/o missing
0.94
0.67
0.62
w/ missing
0.89
0.65
0.60
ity on the metabolic test set by comparing performance on in-domain (N=83,243) and out-of-domain (N=6,943) devices. We follow the setting in the main paper (Sec. 5.1) and report classification (AUROC) and regression (MAE) results. As the table confirms, the model exhibits sound OOD generalization: while "BMI" and "Hypertension" predictions show slight variation, "Anxiety" and "Age" predictions may improve under OOD conditions.
C.7.2 Missingness
In both our pretraining and downstream datasets, sensor missingness arises naturally (e.g., devices like Fitbit Versa or Charge lack EDA sensors). We perform thorough preprocessing for data missingness following established practices: When entire sensor modality is absent, impute values with the population mean computed from the pretraining data. For temporal gaps within a sensor stream (e.g., dropouts), apply linear interpolation across gaps and forward/backward filling sequence boundaries. Consequently, SensorLM is trained on data with up to 80% missingness. It learns to model missingness patterns and shows robustness across scenarios.
To further investigate the performance under missingness, we conducted additional experiments on (1) subgroup analysis stratified by percentage of missingness, and (2) comparisons when certain sensor modalities are entirely absent. First, we observe only mild degradation as missingness increases from <40% to 60-80% Second, when specific sensor modalities (c.g., EDA HRV) are entirely absent,
29

---

we note moderate AUROC decrease (0.02-0.09) relative to the full coll-modality setting. These findings confirm that, although performance degrades slightly under missingness, SensorLM remains robust and flexible with incomplete sensor data.
D Societal Impact
Broader Impacts. Ubiquitous health technologies, particularly wearables, offer tremendous poten- tial to transform healtheare through continuous, longitudinal personal monitoring. However, the complexity of raw sensor data often restricts insights to low-level metrics. SensorLM addresses this challenge by translating multimodal sensor signals into natural language, enabling more intuitive and accessible interface for both consumers and domain experts. This capability promotes clearer, more actionable insights, potentially fostering proactive, personalized, and preventative health management.
Limitations and Ethical Considerations. While consumer health research holds great promise, it must be guided by careful attention to safety, fairness, and privacy. The possibility of misuse by malicious actors underscores the importance of responsible development. Although we advocate for open science, health data-by its requress delicate balance between research reproducibility and participant confidentiality.
Importantly, SensorLM is research prototype and is not intended for clinical use. has not been validated as diagnostic tool and should not be used for medical Decision-making without formal regulatory approval. Clinical deployment would necessitate rigorous validation and compliance with relevant healthcare regulations.
Finally, while SensorLM's methodology is designed to be generalizable, current evaluation is limited to specific wearable devices and sensor modalities. Further research is needed to assess its performance across broader device ecosystems, data modalities, and population groups.
30

---

NeurIPS Paper Checklist
Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly outline SensorLM 's main contributions, in- cluding the caption generation pipeline, curated sensor-langual paired dataset, and SensorLM pre-training framework.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contribu- tions made in the paper and important assumptions and limitations. A No or NA answer this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Limitations are discussed Sec. regarding the evaluation data and device general- izability.
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those
discussed in the paper.
The authors are encouraged create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to vi- olations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets with few runs. In general. empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or specch-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by review- ers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
31

---

Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
formal proofs provided appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experi- mental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We detail caption generation pipeline. model training, and evaluation setup in See. 3.2, Sec. 4, Sec. and Appendix for reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution a dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code. the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature the contribution. For example
(a) If the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is be a way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
model (e.g., large language model), then there should either
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-soure models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
32

---

Justification: Under our IRB IRB-approved protocol, we have obtained informed consent to release de-identified version of the downstream dataset, including hypertension, anxiety, age, and BMI labels. We are unable to release the pretraining data due to privacy concerns, but the main paper and Appendix provide sufficient details for reproducibility. Access will be limited researchers with verified academic affiliations.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https: httpppppppcppppppp/cccpppc quides/CodeSuue
ssi onPoliccy ilyoyyyy for more details.
While we encourage the release of code and data, we understand that this might not possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: www.miisssssoiioooiooooooo for more details.
/ //nips.com waablic/ggides/CCCe
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
They are detailed in Sec. 5 and Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report eтror bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We report CI for Fig. 7. While we do not have CIs reported for main zero-shot results (Table 2), we include extensive metrics and comparisons with baselines showing substantial enough performance gains.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main
claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap. etc.)
The assumptions made should be given (c.g., Normally distributed errors).
33

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1 1-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information the computer resources (type of compute workers, memory, time execution) needed to reproduce the experi- ments?
Answer: [Yes]
Justification:
All the details regarding compute resources provided in Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experi- mental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https://neurip: cc/ wt/ppulic//tticcuuuuiiicc
Answer: [Yes]
Justification: All our user data from wearable devices are de-identified and used with participant consent and IRB review. We also don't 't release the data publicly to avoid chances of potential misuse.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethies.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts the work performed?
Answer: [Yes]
Justification: Yes, this is discussed in Appendix D, as our work has clear applications in healthcare.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
34

---

'Tee conference expects that many papers will be foundational research and not tied to particular applications. let alone deployments. However, if there direct path any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not release data models, so it poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The use of the baseline L.M models and SSL methods is credited, with details in Appendix B.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode www.cooaaaaoooo dataattsssss has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
35

---

Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well as details about compensation (if any)?
Question:
Answer: [Yes]
Justification: Sec. 3 and Sec. 5 describes the IRB-approved observational studies, with more details the Appendix.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should paid least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification:
All data de-identified and used with participant consent and IRB review.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non- standadd component of the core methods in this research? Note that if the LIM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
36

---

Justification:
SensorLM involves a transformer based text encoder and multimodal decoder to
model unimodal and multimodal representations. We also compare against some LLM baselines like Gemini and Gemma.
Guidelines:
The answer NA mcans that the core method development in this research does not involve LLMs as any important, original, non-standard components. Please refer LLM policy htttp:://eeurppsccc/foooff should or should not be described.
for what
37


Reviews:
[{"review_id": "Kd41wnjlmR", "reviewer": "Reviewer_yhqJ", "summary": "This work constructs a large-scale multimodal sensory dataset and presents a well-structured sensor-data–caption pretraining pipeline that generalizes effectively across multiple downstream tasks. It directly addresses the challenge of data scarcity, particularly the lack of aligned sensor data, in this domain. The pretrained model demonstrates strong performance across a range of evaluation settings.", "questions": "1. Can the proposed learning pipeline or pretrained model generalize to out-of-domain data?\n\n2. Does the modality alignment strategy risk weakening modality-specific representations?\n\n3. Can SensorLM handle scenarios with missing modalities or adapt to varying modality combinations?\n\n4. Could the authors provide more details on the design and implementation of the “hierarchical” caption structure?\n\n5. Is the proposed approach extendable to unaligned multimodal data?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n+ The scale of this work is large, making a good contribution to the sensory data learning field, which suffers from data scarcity and lack of aligned data.\n\n+ The paper addresses the lack of interpretability of sensory data and forms the captions into structured and interpretable formats. The hierarchical structure is interesting.\n\n+ The evaluation is extensive and convincing compared to many advanced pipelines.\n\nWeaknesses:\n\n- My primary concern is whether the proposed learning pipeline and pretrained model can generalize to out-of-domain data. While the evaluation is extensive, all experiments are conducted on the authors’ collected datasets. In practice, users may employ different device types or encounter variations in data formats. Given the inherent heterogeneity in devices and environments in sensory data applications, it is unclear whether the framework can effectively interpret preprocessed or out-of-distribution data without performance degradation.\n\n- The alignment strategy combining contrastive and generative losses is effective for joint sensor-language modeling. However, enforcing all modalities into a shared latent space may compromise the fidelity of modality-specific representations, especially for time-series sensor data. A more fine-grained approach that applies alignment at different levels based on context could improve performance. In the current design, modality alignment appears uniform and context-agnostic.\n\n- It is unclear whether SensorLM can handle missing modality scenarios or adapt to varying modality combinations. In real-world applications, users may choose not to wear certain sensors due to personal preference, battery constraints, or hardware availability. The model’s flexibility in such cases should be discussed or evaluated.\n\n- The paper briefly mentions a hierarchical design for captions but lacks sufficient detail on its structure. It would be helpful to elaborate on how the statistical, structural, and semantic levels are defined and how the model captures both shared and modality-specific information across these layers.\n\n- Given the large number of participants involved in data collection, more details on the labeling process, both for hard labels and soft labels, would be valuable. For smaller research groups looking to adopt this approach, practical guidance on data annotation would improve reproducibility and usability.\n\n- In many real-world scenarios, fully aligned multimodal sensory data is not available. It would strengthen the work to discuss whether and how the proposed approach can be extended to handle unaligned or weakly aligned sensory inputs.", "was_revised": true, "final_justification": "The authors’ rebuttal has addressed my main concerns, and I will maintain my positive rating.", "rebuttal": "Thank you for your supportive remarks! We are delighted that the experiments and analyses were interesting and thorough to you. Below, we address your concerns in detail. We hope that these will further clarify our work and lead to a favorable increase of the score.\n\n&nbsp;\n\n> *W1 & Q1: Can the proposed pipeline or pretrained model generalize to out-of-domain data?*\n\nThank you for your constructive comment. We agree that out-of-domain generalization is crucial. First, we would like to clarify that we trained and tested our models with a **diverse set of device types (more than 30)**. Our pretraining set included **18** smartwatch and fitness-tracker models (e.g., *Google Pixel Watch 1-3, Fitbit Sense 1-2, Charge 4-6, Versa 1-4, Inspire 2-3*). The activity downstream dataset used the same devices, while the metabolic downstream dataset has an **additional 17** device types (e.g., *Fitbit Flex, One, Zip, Charge HR, Charge 3*). A complete device distribution will be provided in the Appendix.\n\nAs a result, we are able to demonstrate generalizability on the metabolic test set by comparing performance on **in-domain (N=83,243)** and **out-of-domain (N=6,943)** devices. We follow the setting in the main paper (**Sec. 5.1**) and report classification (AUROC) and regression (MAE) results:\n\n| |Anxiety $\\uparrow$|Hypertension $\\downarrow$|Age $\\downarrow$|BMI $\\downarrow$|\n|-|-|-|-|-|\n|ID|0.65|0.61|9.24|5.74|\n|OOD|0.72|0.55|8.55|5.79|\n\nAs the table confirms, the model exhibits sound OOD generalization: while “BMI” and “Hypertension” predictions show slight variation, “Anxiety” and “Age” predictions may improve under OOD conditions.\n\nMoreover, although training a foundation model on every possible device is impractical, our approach mitigates this challenge by operating on **minutely features** (e.g., heart rate, step count) derived from raw sensor data, which are inherently more consistent across devices. To support this, we present the distribution of “heart rate” measurements across representative devices:\n\n|Device|HR|\n|-|-|\n|Pixel Watch 2|79.6$\\pm$3.9|\n|Charge 6|78.2$\\pm$3.8|\n|Sense 2|79.3$\\pm$3.6|\n|Versa 4|78.5$\\pm$3.7|\n|Inspire 3|78.2$\\pm$3.6|\n\nThis confirms the robustness of the derived features across hardwares. We will add the results to the revised manuscript.\n\n&nbsp;\n\n> *W2 & Q2: Does the modality alignment strategy weaken modality-specific representations?*\n\nThanks for the question. We provide further discussions and clarifications as follows.\n\nExisting work in the image-language domain indicates that straightforward vision-language alignment objectives, when trained on large datasets, yield vision‑only encoders with SOTA performance [ref 21, ref 33]. Following the empirical scaling behavior observed for sensor-text data (see **Sec. 5.2**), we expect that the same simple objective, applied at scale, will produce strong sensor‑only encoders.\n\nOur experiments support this: **Fig. 5** and **Appendix C.2** compare embeddings learned by our sensor-language model with sensor-only baselines, showing that our model outperforms across tasks and data sizes. This suggests that the alignment strategy preserves modality‑specific information relative to unimodal objectives.\n\nTo summarize, we believe the current design should be extensive and representative for sensor-language modeling research. We hope it will serve as a sound basis and inspire future work. We certainly welcome any further specific suggestions regarding possible extensions.\n\n&nbsp;\n\n> *W3 & Q3: Can SensorLM handle scenarios with missing modalities or varying modality combinations?*\n\nThanks for pointing this out, and we apologize if it wasn’t clear. In fact, in both our pretraining and downstream datasets, sensor missingness arises naturally (e.g., devices like *Fitbit Versa* or *Charge* lack EDA sensors). We have performed thorough preprocessing for data missingness following established practices [ref 6, ref 20]:\nWhen an entire sensor modality is absent, we impute values with the population mean computed from the pretraining data.\nFor temporal gaps within a sensor stream (e.g., dropouts), we apply linear interpolation across gaps and forward/backward filling at sequence boundaries. \n\nConsequently, SensorLM is trained on data with up to **80%** missingness. It learns to model missingness patterns and shows robustness across scenarios.\n\nTo further address the reviewer’s concern about performance under missingness, we conducted additional experiments on (1) subgroup analysis stratified by percentage of missingness, and (2) comparisons when certain sensor modalities are entirely absent.\n\n**First**, we observe only mild degradation as missingness increases from <40% to 60–80%:\n\n|Missing %|Activity|Anxiety|Hypertension|\n|-|-|-|-|\n|<40%|0.94|0.68|0.62| \n|40-60%|0.94|0.65|0.59|\n|60-80%|0.91|0.65|0.65|\n\n**Second**, when specific sensor modalities (e.g., EDA or HRV) are entirely absent, we note a moderate AUROC decrease (0.02–0.09) relative to the full‑modality setting:\n\n|Modality Missing|Activity|Anxiety|Hypertension|\n|-|-|-|-|\n|w/o missing|0.94|0.67|0.62|\n|w/ missing|0.89|0.65|0.60|\n\nThese findings confirm that, although performance degrades slightly under missingness, SensorLM remains robust and flexible with incomplete sensor data. All results reported in the paper are aggregated over the entire test set, which already includes these missing‑data scenarios. We will incorporate the new analyses in the revised manuscript.\n\n\n&nbsp;\n\n> *W4 & Q4: Could the authors provide more details on the design and implementation of the hierarchical caption?*\n\nWe appreciate the reviewer’s interest. We would like to clarify that definitions of the statistical, structural, and semantic levels are presented in **Fig. 1B** and **Fig. 3**, with rewrite templates in **Appendix A.2**. We provided additional representative examples in **Fig. 4** and **Figs. 10, 11**.\n\nWhen levels are combined, captions are concatenated in the sequence: (semantic, structural, statistical). Their interaction and individual contributions are studied in **Table 4** and **Table 23 (Appendix)**, where results indicate that leveraging both shared and modality-specific information improves downstream performance.\n\nFor completeness, we will include a full example in the Appendix. We welcome any further questions regarding specific elements of the caption hierarchy.\n\n\n&nbsp;\n\n> *W5: Given the large number of participants involved … practical guidance on data annotation would improve reproducibility and usability.*\n\nThank you for the suggestion. We would like to refer the reviewer to details in **Sec. 3.2** and **Sec. 5**. \n\nFor **semantic** captions and downstream task labels, all **event-level** information (including user activity, mood, disease diagnoses, demographic data) is *self-reported* by the user. This constitutes our \"hard\" labeling. Sensor-derived **statistics** and **structural** patterns are computed ***programmatically*** from raw signals. We apply sliding windows and fit linear regression within each window to categorize trends (e.g., increasing, decreasing) based on slope thresholds. We also use peak detection algorithms (`scipy.signal.find_peaks`) to identify spikes and drops.\n\nThis design ensures the approach is both **scalable** and **reproducible**. Structural and statistical captions can be derived from any comparable sensor data, and the semantic component can be extended to other self-reported or externally annotated information.\n\nWe fully agree with the reviewer that this pipeline can benefit the broader research community. To further support adoptions, we will open source the captioning code and provide practical documentation in the final release.\n\n\n&nbsp;\n\n> *W6 & Q5: Is the proposed approach extendable to unaligned multimodal data?*\n\nThanks for raising this. We would like to further discuss several aspects.\n\nFirst, for the input features, we compute minutely features from raw sensor signals, which reduces misalignment issues when aggregating data from diverse devices and sensors [ref 6, ref 20]. The OOD performance reported in **W1 & Q1** supports its robustness.\n\nIn addition, when data from certain sensor channels or intervals are missing, we apply imputation and related preprocessing procedures established in prior work. The results reported in **W3 & Q3** confirms that SensorLM generalizes well under these conditions.\n\nBeyond these measures, SensorLM is readily extendable to handle more irregular data. This can be achieved by integrating established methods (e.g., appending a missingness mask or using alternative tokenization strategies [1, 2]) within the sensor encoder, which are orthogonal to our main framework and can be seamlessly incorporated.\n\nWe will add the above analyses and discussions in the revised manuscript.\n\n---\nReferences:\n1. Beebe-Wang et al. \"PAITS: pretraining and augmentation for irregularly-sampled time series.\" arXiv preprint (2023).\n2. Erturk et al. \"Beyond Sensor Data: Foundation Models of Behavioral Data from Wearables Improve Health Predictions.\" ICML 2025.\n\n&nbsp;\n\n---\nWe hope the above responses have addressed all concerns from the reviewer adequately. We would also like to update the reviewer on our commitment to open‑sourcing SensorLM:\n- **Dataset**. Under our IRB‑approved protocol, we have obtained informed consent to release a **de‑identified version** of the downstream dataset, including hypertension, anxiety, age, and BMI labels. Access will be limited to researchers with verified academic affiliations.\n- **Pretrained model**. We will release **SensorLM checkpoints** trained on these data together with example code for inference and fine-tuning.\n- **Code**. We will open-source the complete caption-generation pipeline to facilitate reuse by the broader research community.\n\nOnce again, thanks for your time and constructive feedback. We would really appreciate it if the reviewer could consider raising their scores and championing the paper."}, {"review_id": "MpejlkTkBW", "reviewer": "Reviewer_wWho", "summary": "This work proposes a way to teach foundation models the “language” of wrist-worn sensors. The authors build a hierarchical captioning pipeline that turns each 24-hour, 26-feature sensor window into statistical, structural and semantic sentences. They train transformer-based dual-encoder + decoder models with combined contrastive-and-generative objectives, unifying CLIP, Cap and CoCa styles in their framework. In zero-shot activity recognition, cross-modal retrieval and few-shot health prediction, SensorLM beats large LLM-tabular baselines and sensor encoders (e.g., DINO, MSN, etc) by sizable margins.", "questions": "Please refer to the weaknesses part.", "limitations": "I think some further discussions about fixed feature dependency would be good.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n+ The paper delivers a substantial engineering effort for wrist-sensor data. Combined contrastive + generative training unifies CLIP-, caption- and CoCa-style objectives and is backed by ablation and scaling-law studies. Experiments cover zero-shot, few-shot and linear-probe settings and show clear gains over several baselines.\n+ The pipeline is illustrated step-by-step; model architecture and objectives are mathematically specified; Other details are moved to a well-structured appendix.\n\nWeaknesses:\n\n- All downstream tasks rely on a fixed 26 features. The framework is hard-coded to those 26 features from a single wrist device. It is unclear how it adapts when tasks demand additional statistics or when multiple sensors each supply their own 26-D vectors.\n- Zero-shot evaluation is compared only with Gemini/Gemma (LLM-on-tabular). Strong raw-data time-series foundation models capable of zero-shot representation learning (e.g., Mantis, Moment, Chronos, UniMTS, etc) are omitted, making claims of superiority less convincing.\n- Training and test sets come from the same user cohort and device (Section 5), yet the paper emphasises “generalisation”. A leave-person-out (cross-device) split is required to demonstrate genuine robustness.\n- The architecture largely reuses established image-text paradigms (ITC, CoCa) with minimal conceptual change.", "was_revised": true, "final_justification": "During the rebuttal, the authors have provided new information and new experimental results. However, after considering other reviewers' comments, I would like to keep my score which I think reflects the evaluation of the submitted manuscript.", "rebuttal": "Thank you for your thoughtful and constructive feedback! We are happy to learn that you found the experiments extensive and the paper well-structured. Below, we provide additional results and clarifications, which we hope will encourage the reviewer to increase the score.\n\n&nbsp;\n\n> *W1: All downstream tasks rely on a fixed 26 features... multiple sensors each supply their own 26-D vectors.*\n\nThank you for raising this. We provide further clarifications and results as follows.\n\nFirst, we use 26 features (Table 6, Appendix A.1) as a ***generalizable superset*** derived from common smartwatch sensors (ACC, PPG, EDA, etc.) across **> 30** commercial devices. Our pretraining set covers **18** device models (e.g., *Google Pixel Watch 1-3, Fitbit Sense 1-2, Charge 4-6, Versa 1-4, Inspire 2-3*), while the metabolic downstream set has an **additional 17** devices (e.g., *Fitbit Flex, One, Zip, Charge HR, Charge 3*). The superior performance across devices/populations (**Fig. 5, Table 19**) validates the generalizability of derived features, which reflects **real-world** deployment across hardware platforms.\n\nIn addition, **not all** devices offer the complete set of 26 features. In both pre‑training and downstream sets, sensor missingness arises naturally (e.g., many devices like *Fitbit Versa* or *Charge* lack EDA sensors). For instance, the Metabolic set has on average 11 ($\\pm$ 5) sensor features completely absent; in such cases we imputed values using the population mean computed from pre‑training data, following established practice [ref 6, ref 20]. \n\nConsequently, we report comparisons when certain sensor modalities are unavailable:\n\n|Modality Missing|Activity|Anxiety|Hypertension|\n|-|-|-|-|\n|w/o missing|0.94|0.67|0.62|\n|w/ missing|0.89|0.65|0.60|\n\nThis confirms that SensorLM remains robust to missing sensor modalities (results in the main paper included all missingness scenarios).\n\nWhile we acknowledge that the current feature set may not be exhaustive, we would like to emphasize that the captioning pipeline and modeling framework are **not hard-coded** to these 26 features. The framework is designed for **extensibility**: new features can be incorporated by extending the text description without architectural modification. While our feature set already aligns with sensors on **30+** commercial devices, device-specific models could also be trained and their embeddings/predictions can be aggregated at inference time [ref 1, ref 33].\n\nTo summarize, we believe the current design is extensive and representative for sensor‑language modeling, and we hope it will serve as a sound basis for future work. We will add the above results and discussions on sensor feature dependency in the revised manuscript.\n\n&nbsp;\n\n> *W2: Zero-shot evaluation is compared only with Gemini/Gemma .. making claims of superiority less convincing.*\n\nThanks for pointing out these related works. We would like to offer clarifications and results as follows.\n\nFirst, among the cited methods, Moment [1] and Chronos [2] target zero‑shot *forecasting*, not classification or retrieval. Mantis [3] offers *zero-shot embedding extraction* similar to our SSL baselines. All three require training a classifier for downstream evaluation, placing them in the **few‑shot** rather than actual **zero‑shot** setting (i.e., no tuning or adaptation).\n\n**UniMTS** [ref 36] is the closest approach of comparison. We discussed it in our related work and now we directly compare with it. Specifically, it employs a CLIP [ref 21] objective and treats activity descriptions as text input, corresponding to the *semantic* caption in our framework. We have adapted and trained UniMTS under our setup and report the results below:\n\n| |**Activity (20-class)**|**gym cardio**|**outdoor sports**|\n|-|-|-|-|\n|UniMTS|0.71|0.73|0.65|\n|SensorLM|**0.84**|**0.76**|**0.86**|\n\nThe table confirms that SensorLM outperforms UniMTS across zero-shot tasks. We note that the benefits stem from (1) our hierarchical caption design, where relying solely on semantic captions is **sub‑optimal** across tasks (**Table 4, Sec. 5.3**), and (2) our *modularized* training framework, which enables systematic comparison of objectives (CoCa proves more effective than CLIP; **Table 5, Sec. 5.3**).\n\n\nWe will incorporate this additional baseline and extend the discussion of related works as suggested by the reviewer in the revised manuscript.\n\n&nbsp;\n\n> *W3: Training and test sets come from the same user cohort ... demonstrate genuine robustness.*\n\nWe believe there is a misunderstanding here by the reviewer. We would like to clarify that all tasks in our paper employ a **leave-person-out** evaluation protocol, as specified by the train and test cohorts in **Table 7 (Appendix)**.  Our statement that the data are “from the same population” in Sec. 5 refers only to shared recruitment criteria and overall distribution, not to **any overlap** of participants between training and test sets.  We will revise this wording to eliminate ambiguity and emphasize our leave-person-out evaluation.\n\nOn a related note, in addition to the default leave‑person‑out evaluation, we also assess **cross‑device generalization** on the metabolic tasks by comparing **in-domain (N=83,243)** and **out-of-domain (N=6,943)** devices (as discussed in **W1**).  Following the protocol in Sec. 5.1, we report few-shot classification and regression results:\n\n| |Anxiety $\\uparrow$|Hypertension $\\downarrow$|Age (MAE) $\\downarrow$|BMI (MAE) $\\downarrow$|\n|-|-|-|-|-|\n|ID|0.65|0.61|9.24|5.74|\n|OOD|0.72|0.55|8.55|5.79|\n\nAs shown, the model maintains strong OOD performance: while “BMI” and “Hypertension” show minor variation, “Anxiety” and “Age” exhibit slight improvements under OOD settings.\n\nWe believe the clarified leave‑person‑out protocol and the results on OOD devices address the concern and demonstrate SensorLM’s robustness. We will incorporate the analysis and update the manuscript accordingly.\n\n&nbsp;\n\n> *W4: The architecture largely reuses established image-text paradigms (ITC, CoCa) with minimal conceptual change.*\n\nWe appreciate the comment and would like to clarify our architectural contribution. Although the architecture skeleton follows established image-text paradigms, our work introduces a **modular** and **generic** pre‑training framework that makes it straightforward to *attach, remove, or re‑weight* objectives. This **modularity** allows systematic ablations, enables rapid integration of new components, and reveals design trade‑offs not explored in prior image-text studies.\n\nTo support this claim, we study two extensions that would be less practical without such modular design.\n\n**(1) Loss-weight exploration**. We sweep the relative weights of the contrastive ($\\lambda_{con}$) and captioning ($\\lambda_{cap}$) losses beyond fixed settings by CLIP ($\\lambda_{con}=0$) and CoCa ($\\lambda_{con}=\\lambda_{cap}=1$). We find clear task‑specific trade‑offs: a 1:2 balance improves zero‑shot activity recognition, whereas 5:1 yields the best metabolic‑health prediction. These results indicate that careful tuning, enabled by our framework, yields measurable gains:\n\n|$\\lambda_{con}$|$\\lambda_{cap}$|zero-shot|||Linear Probing|||\n|-|-|-|-|-|-|-|-|\n|||**Activity (20-class)**|**gym cardio**|**outdoor sports**|**Activity**|**Anxiety**|**Hypertension**|\n|1|0.2|0.82|0.77|0.82|0.92|**0.66**|**0.62**|\n|1|1|0.84|0.76|**0.86**|**0.94**|0.65|0.60|\n|1|2|**0.85**|**0.78**|0.85|0.93|0.65|0.60|\n\n**(2) Sensor-reconstruction head.**  We append either an auto‑encoder (AE) or vector‑quantized auto‑encoder (VQ‑AE) decoder to the sensor encoder so that training jointly optimizes *contrastive alignment*, *caption generation*, and *signal reconstruction*. Adding the reconstruction loss improves health‑prediction metrics with only a modest drop in activity recognition. The VQ‑AE variant achieves the best overall trade‑off:\n\n| |Zero-Shot|||Linear Probing|||\n|-|-|-|-|-|-|-|\n||**Activity (20-class)**|**gym cardio**|**outdoor sports**|**Activity**|**Anxiety**|**Hypertension**|\n|Con + Cap|**0.84**|**0.76**|**0.86**|**0.94**|0.65|0.60|\n|Con + Cap + AE|0.76|0.67|0.82|0.89|0.68|0.63|\n|Con + Cap + VQ-AE|0.77|0.68|0.74|0.91|**0.69**|**0.65**|\n\nThese straightforward extensions, enabled by the flexibility of the modular framework, demonstrate its support for rapid experimentation and yield concrete performance benefits. This is consistent with the **NeurIPS reviewer guideline**:\n\n> *”originality does not necessarily require introducing an entirely new method when the work enables new insights or capabilities\"*\n\nWe hope these clarifications and new results highlight the significance of our contribution. We will incorporate the above discussions and results into the manuscript.\n\n---\nReferences:\n1. Goswami et al. \"Moment: A family of open time-series foundation models.\" ICML 2024.\n2. Ansari et al. \"Chronos: Learning the language of time series.\" TMLR 2024.\n3. Feofanov et al. \"Mantis: Lightweight calibrated foundation model for user-friendly time series classification.\" arXiv (2025).\n\n&nbsp;\n\n---\nWe hope the above responses have addressed all concerns from the reviewer adequately. We would also like to update the reviewer on our commitment to open‑sourcing SensorLM:\n- **Dataset**. Under our IRB‑approved protocol, we have obtained informed consent to release a **de‑identified version** of the downstream dataset, including hypertension, anxiety, age, and BMI labels. Access will be limited to researchers with verified academic affiliations.\n- **Pretrained model**. We will release **SensorLM checkpoints** trained on these data together with example code for inference and fine-tuning.\n- **Code**. We will open-source the complete caption-generation pipeline to facilitate reuse by the broader research community.\n\nOnce again, thanks for your time and constructive feedback. We would really appreciate it if the reviewer could consider raising their scores and championing the paper."}, {"review_id": "oESRefVwIc", "reviewer": "Reviewer_oLSX", "summary": "This paper presents a new sensor data captioning strategy, which has three hierarchies: statistical, structural, and semantic captions. Then, the paper presents SensorLM, aiming to generate captions based on sensor data. SensorLM is a hybrid method combining contrastive alignment and text generation. Experiments demonstrate that SensorLM is superior to baselines methods on diverse downstream tasks.", "questions": "What does it mean by \"zero-shot\" in your context? Does it mean the test classes are unseen, or the test datasets are collected separately from your training set?", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The paper is overall well written and organized. It is easy to follow the idea.\n2. Building large datasets is a fundamental part of the community and would largely inspire the following research.\n3. The idea of hierarchical captioning is interesting and meaningful. \n\nWeaknesses:\n1. The main contribution of this work is the sensor-text dataset with hierarchical captions. However, the authors do not publish the dataset. Although the data collection details are presented. Datasets are different from models, which are hard and time-consuming to reproduce. Thus, it is critical to publish the dataset.\n2. The quality of the dataset is not studied. It would be better if the authors could show how existing studies work on the new dataset and how the hierarchical captions impact the performance. \n3. The novelty of the SensorLM model is limited. It combines generative and contrastive learning, which is not new. In addition, the authors use $\\lambda_{con}=1$ and $\\lambda_{cap}=1$ as the default setting, which is CoCa actually. The paper fails to show the benefits of tuning the loss weights $\\lambda$.\n4. In Tables 2 and 3, the results of the baselines are nearly random. It is quite unfair to make such a comparison. The two baselines should be fine-tuned using the proposed dataset first, then make inferences and comparisons.\n5. The paper would be more suitable for the dataset track instead of the research track.", "was_revised": true, "final_justification": "The paper has a good structure and clear writing. The idea is interesting and meaningful, especially the dataset construction part. However, the SensorLM model itself has minor originality, and the experimental studies are specific to the proposed model and datasets, lacking general insights and guidance to the general domain. Thus, while also fine for the research track, I would advocate this paper more for presenting at the dataset track. Furthermore, the reproducibility of this paper is inadequate. Thus, I am on the borderline of this paper.", "rebuttal": "Thank you for your thoughtful feedback, your comments have helped us strengthen the evidence for our approach. However, we believe that there are several **important misunderstandings** which we would like to clarify and address point-to-point:\n\n&nbsp;\n\n> *W1: The main contribution of this work is… it is critical to publish the dataset.*\n>\n> *W5: The paper would be more suitable for the dataset track instead of the research track.*\n\nWe respectfully disagree with this assessment. We would like to clarify that SensorLM’s key contribution **extends well beyond a “sensor‑text dataset”**, addressing the **full pipeline** of sensor‑language pretraining with several advances:\n\n1. **Defining paired language for sensor data where none naturally exist**. We introduce a *scalable* hierarchical caption framework, enabling alignment learning at scale, resulting in the *largest* sensor‑language corpus to date, orders of magnitude larger than prior studies (Table 1).\n2. **Determining what makes for effective captions**. Contrary to the common assumption [ref 36], we show that relying solely on high‑level semantic descriptions is *sub‑optimal* across tasks (see **W2** responses).\n3. **Establishing empirical scaling laws for sensor‑language alignment**. We conduct the *first* large‑scale study of sensor-language alignment and demonstrate clear scaling laws (also noted by Reviewer wWho).\n4. **Providing a *modular and generic* pre‑training framework**. The framework makes it straightforward to *attach, remove, or re‑weight* objectives. This *modularity* allows systematic ablations, fast integration of new components, and reveals design trade‑offs unexplored in prior image-text studies (see **W3** responses).\n\nThese methodological, empirical, and conceptual advances go well beyond dataset construction and align with the expectations of the main track. Importantly, they align with the **NeurIPS reviewer guideline** that\n\n> *”originality does not necessarily require introducing an entirely new method but **can come from new insights, novel combinations, or improved understanding of existing approaches**”*\n\n&nbsp;\n\n**Open‑source commitment.** In addition, we would also like to update the reviewer on our commitment to open‑sourcing SensorLM:\n- **Dataset**. Under our IRB‑approved protocol, we have obtained informed consent to release a **de‑identified version** of the downstream dataset (including hypertension, anxiety, age, and BMI labels) to researchers with verified academic affiliations.\n- **Pretrained model**. We will release **SensorLM checkpoints** trained on these data with example code for inference and fine-tuning.\n- **Code**. We will open-source the complete caption-generation pipeline to benefit the broader research community.\n\nWe hope these clarified contributions, along with our open-sourcing commitment, encourage the reviewer to reassess our paper’s novelty and significance.\n\n&nbsp;\n\n> *W2: The quality of the dataset is not studied…how the hierarchical captions impact the performance.*\n\n\nThanks for the suggestion. We would like to clarify that the dataset quality **has been** carefully benchmarked with established SSL methods.\n\nFirst, **SSL baselines** benchmarked under the few-shot protocol in **Sec. 5.1** (Appendix **Table 19, Fig. 9**) show that models pretrained with SensorLM and hierarchical captions consistently surpass SSL baselines trained on the same data.\n\nIn addition, we adapt **UniMTS** [ref 36], a foundation model capable of zero-shot classification by using a CLIP objective and *semantic* activity caption, to our dataset (row 1 below) and examine how our hierarchical captioning strategies (rows 2-3 below) affect performance:\n\n| |Caption|Zero-Shot|||Linear Probing|||\n|-|-|-|-|-|-|-|-|\n| | |20-class Activity|gym cardio|outdoor sports|Activity|Anxiety|Hypertension|\n|UniMTS|semantic|0.71|0.73|0.65|**0.94**|0.65|0.58|\n|UniMTS + our caption|semantic + structural|0.83|0.75|0.84|0.92|0.66|0.62|\n|UniMTS + our caption|semantic + structural + statistical|0.67|0.63|0.76|0.83|**0.67**|**0.63**|\n|SensorLM|semantic + structural|**0.84**|**0.76**|**0.85**|**0.94**|0.65|0.60|\n\nThe table shows that adding **structural** captions improves zero‑shot activity recognition, whereas **statistical** captions benefits metabolic‑health tasks, consistent with our ablation study (**Table 4, Sec. 5.3**). These findings confirm both dataset quality and the effectiveness of the hierarchical caption strategy.\n\nWe will incorporate the additional baseline and analyses into the revised manuscript.\n\n&nbsp;\n\n> *W3: The novelty of the SensorLM model…benefits of tuning the loss weights $\\lambda$.*\n\nWe respectfully disagree with the claim. Our contribution extends **beyond architectural design** (see our responses to **W1**) and addresses the **full pipeline** of sensor‑language pretraining, offering new insights into aligning wearables with natural language.\n\nWhile our architecture skeleton follows established image-text paradigms, our work introduces a *modular* and *generic* pretraining framework that makes it straightforward to *attach, remove, or re‑weight* objectives. This **modularity** allows systematic ablations, rapid integration of new components, and reveals design tradeoffs unexplored in prior image-text studies.\n\nTo support this claim, we study two extensions that would be less practical without such modular design.\n\n**(1) Loss-weight exploration**. Per reviewer suggestion, we sweep the relative weights of the contrastive ($\\lambda_{con}$) and captioning ($\\lambda_{cap}$) losses beyond fixed settings by CLIP ($\\lambda_{con}=0$) and CoCa ($\\lambda_{con}=\\lambda_{cap}=1$). We find clear task‑specific trade‑offs: a 1:2 ratio boosts zero‑shot activity recognition, whereas 5:1 yields the best metabolic‑health prediction. These results show that careful tuning, enabled by our framework, produces measurable gains.\n\n|$\\lambda_{con}$|$\\lambda_{cap}$|Zero-Shot|||Linear Probing|||\n|-|-|-|-|-|-|-|-|\n|||**Activity (20-class)**|**gym cardio**|**outdoor sports**|**Activity**|**Anxiety**|**Hypertension**|\n|1|0.2|0.82|0.77|0.82|0.92|**0.66**|**0.62**|\n|1|1|0.84|0.76|**0.86**|**0.94**|0.65|0.60|\n|1|2|**0.85**|**0.78**|0.85|0.93|0.65|0.60|\n\n**(2) Sensor-reconstruction head.**  We append either an auto‑encoder (AE) or vector‑quantized auto‑encoder (VQ‑AE) decoder to the sensor encoder, so that training jointly optimizes *contrastive alignment*, *captioning*, and *signal reconstruction*​. Adding the reconstruction loss improves health‑prediction metrics with only a modest drop in activity recognition. The VQ‑AE variant achieves the best overall trade‑off:\n\n| |Zero-Shot|||Linear Probing|||\n|-|-|-|-|-|-|-|\n||**Activity (20-class)**|**gym cardio**|**outdoor sports**|**Activity**|**Anxiety**|**Hypertension**|\n|Con + Cap|**0.84**|**0.76**|**0.86**|**0.94**|0.65|0.60|\n|Con + Cap + AE|0.76|0.67|0.82|0.89|0.68|0.63|\n|Con + Cap + VQ-AE|0.77|0.68|0.74|0.91|**0.69**|**0.65**|\n\nThese extensions, enabled by the modular framework, show that it supports rapid experimentation and yields concrete performance benefits. Again, this is consistent with the **NeurIPS reviewer guideline**:\n\n> *”originality does not necessarily require introducing an entirely new method when the work enables new insights or capabilities\"*\n\nWe hope the clarifications and new results clarify the significance of our contribution. We will incorporate the discussions and results into the manuscript.\n\n&nbsp;\n\n> *W4: In Tables 2 and 3, the results of the baselines are nearly random… fine-tuned using the proposed dataset first, then make inferences and comparisons.*\n\nThank you for the comment. We acknowledge that the LLM baselines in **Tables 2 and 3** were not originally designed for high-dimensional time-series reasoning.\n\nTo address the reviewer’s concern, we fine-tuned **Gemini 2.0 Flash** on our dataset using **supervised activity labels**, following a standard supervised fine-tuning (SFT) procedure. We show zero‑shot balanced accuracy under the same protocol as Table 2:\n\n| |20-class|Environment|Cardio / strength|Locomotion / stationary|Gym cardio|Outdoor sports|\n|-|-|-|-|-|-|-|\n|Gemini 2.0|0.07|0.25|0.52|0.55|0.26|0.29|\n|Gemini 2.0 (SFT)|0.1|0.28|0.53|0.55|0.35|0.32|\n|SensorLM-XL|**0.31**|**0.38**|**0.66**|**0.58**|**0.51**|**0.53**|\n\nThe table confirms that fine‑tuning yields only *marginal* gains over the vanilla Gemini, and still falls well short of SensorLM, despite Gemini 2.0 Flash being substantially larger. \n\nThese results highlight two persistent limitations of (fine‑tuned) LLMs:\n1. Due to intrinsic context‑length constraints, LLM requires downsampling to interpret raw sensor data, limiting their ability to capture fine‑grained dynamics;\n2. SFT on supervised activity labels reduces instruction‑following ability of LLMs in other tasks, leading to limited zero‑shot generalization.\n\nWe will add the fine‑tuned results in the manuscript and include **UniMTS** (see our response to **W2**) as a stronger zero-shot baseline for a more thorough comparison.\n\n&nbsp;\n\n> *Q1: What does it mean by \"zero-shot\" in your context?...collected separately from your training set?*\n\nThanks for the question. By “zero‑shot” we adopt the convention used in CLIP and CoCa: the model is evaluated exactly as trained, with **no** task‑specific tuning or calibration. Our test cohorts are participant‑independent and entirely **disjoint** from the pretraining pool.\n\nAdditionally, for categorical tasks (**Table 2**) and unseen activity classes (**Fig. 6**), we present label names **never** observed during pretraining; the model must map them to sensor patterns through the learnt sensor-language representation.\n\n&nbsp;\n\n---\nWe hope these clarifications and our open-sourcing commitment encourage the reviewer to reassess the contributions of the paper. We are more than happy to answer more questions. If the discussions and results successfully addressed all your concerns, please consider raising your score."}, {"review_id": "TM35VcOFfq", "reviewer": "Reviewer_ZU6J", "summary": "This paper introduces SensorLM, a family of sensor-language foundation models that learn from large-scale wearable sensor data paired with automatically generated textual descriptions. The authors propose a hierarchical captioning pipeline (statistical, structural, semantic) to describe over 59 million hours of multimodal data collected from more than 100,000 individuals, enabling alignment with natural language for downstream tasks. The model integrates contrastive and generative objectives, extending prior multimodal architectures (e.g., CLIP, CoCa) to the sensor-text domain. Extensive evaluations demonstrate SensorLM's superiority in zero-shot recognition, few-shot learning, and cross-modal retrieval, making it a compelling step toward interpretable and generalizable sensor understanding.", "questions": "1. Have the authors evaluated SensorLM’s generalization ability on non-wearable time-series data, such as medical monitoring signals or industrial sensors?\n2. What is the computational overhead of SensorLM at inference time compared to conventional sensor models, especially in edge or real-time environments?\n3. Can the hierarchical captioning pipeline be adapted or extended to support human-written descriptions or other forms of supervision beyond rule-based heuristics?\n4. How robust is the model when some sensor modalities are missing, noisy, or partially corrupted at inference time?", "limitations": "Please see above.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "### [Strengths]\n1. The hierarchical captioning pipeline—comprising statistical, structural, and semantic levels—offers a unique way to supervise time-series signals with interpretable textual labels.\n2. SensorLM demonstrates strong empirical results across a wide range of datasets, tasks, and architectures, showing robustness and generalization beyond conventional sensor models.\n3. The method supports test-time inference without requiring fine-tuning or task-specific labels, which is particularly valuable for real-world deployment in personalized or low-resource settings.\n4. The authors provide a comprehensive evaluation that includes classification, retrieval, and metabolic prediction tasks, highlighting the model's versatility across modalities and objectives.\n\n### [Weaknesses]\n1. The quality of the pretraining data relies on heuristically generated captions, which may introduce noise or bias that affects the downstream alignment performance.\n2. The model is currently evaluated only on wearable sensor data, and its applicability to other time-series domains—such as industrial or medical sensors—remains untested.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for acknowledging the novelty and the contributions of our work! We are glad that you found the paper comprehensive and valuable with strong empirical results. In the following, we address your concerns in detail. We hope that these will further clarify our work and lead to a favorable increase of the score.\n\n&nbsp;\n\n> *W1: The quality of the pretraining data relies on heuristically generated captions, which may introduce noise or bias that affects the downstream alignment performance.*\n\nWe appreciate the reviewer’s concern about potential noise or bias introduced by rule‑based captions, particularly for the statistical and structural components. The semantic captions, however, are derived from *self‑reported* user events. Our hierarchical scheme therefore blends human‑provided information with *programmatically* extracted patterns, reducing reliance on **any single heuristic**. In addition, every caption is paraphrased through diverse templates generated by LLMs (**Appendix A.2**), which broadens linguistic variety and better reflects how users might describe sensor traces. This strategy delivers **scalability** that manual annotation cannot match, allowing us to assemble a large-scale sensor-language corpus.\n\nEmpirically, our results support that SensorLM remains **robust** despite possible noise, achieving strong alignment performance across diverse downstream tasks. **Section 5.3** and **Appendix C.5** analyze the contribution of each caption type and training strategy, offering further insight into alignment dynamics.\n\nWe believe this trade-off between **caption realism** and **scalability** is a practical and necessary step toward enabling large-scale sensor-language learning. In future work, incorporating more naturalistic language data (e.g., user descriptions or clinical notes) could further improve alignment quality and enable more model capabilities. We will add more discussion on the trade-off of rule-based captioning and human annotations in the revised manuscript.\n\n&nbsp;\n\n> *W2 & Q1: The model is currently evaluated only on wearable sensor data, and its applicability to other time-series domains—such as industrial or medical sensors—remains untested.*\n\nThank you for the thoughtful comment. While our current evaluation focuses on wearable sensor data, the SensorLM framework is designed to be **modality-agnostic** and generalizable to other time-series domains. The captioning pipeline for deriving statistical and structural patterns from raw signals, and the ability to incorporate new semantic information via captions, is **not specific** to wearable devices. Our **pretraining objectives** and **modular architecture** can also be readily adapted to structured inputs from industrial or medical sensors. We see this as a promising direction for future work and believe that SensorLM lays the foundation for scalable sensor-language learning across diverse domains. We will add these discussions in the revised manuscript.\n\n&nbsp;\n\n> *Q2: What is the computational overhead of SensorLM at inference time compared to conventional sensor models, especially in edge or real-time environments?*\n\nSensorLM demonstrates competitive inference time overhead compared to conventional baselines. For zero-shot inference, the combined sensor and text encoders require **122 GFLOPs**. In few-shot or linear probing scenarios, SensorLM's sensor encoder, at **92 GFLOPs**, offers comparable efficiency to the SSL baselines (**105 GFLOPs**), maintaining a total cost around **100 GFLOPs**. The complete caption generation process requires **155 GFLOPs**; however, this cost is justified by the model's distinctive interpretive capabilities.\n\nChallenges for deployment in highly constrained environments are similar to those of baseline SSL methods, but these can be mitigated through established optimization strategies like model compression or hybrid on-device/cloud inference.\n\nWe will include a detailed discussion of these considerations in the revised manuscript.\n\n&nbsp;\n\n> *Q3: Can the hierarchical captioning pipeline be adapted or extended to support human-written descriptions or other forms of supervision beyond rule-based heuristics?*\n\nYes, the hierarchical captioning pipeline is modular and can be adapted to incorporate human-written descriptions, clinical notes, or expert annotations in place of or alongside existing captions. Its design supports seamless integration of diverse textual inputs. While large-scale expert annotations are challenging to obtain, our caption generation pipeline and pretraining framework is readily adaptable should such supervised data become available. As a first step, our rule-based approach makes scalable sensor-language pretraining possible. \n\nWe will add a discussion section on the adaptability to human-written or other forms of supervision in the revised manuscript.\n\n&nbsp;\n\n> *Q4: How robust is the model when some sensor modalities are missing, noisy, or partially corrupted at inference time?*\n\nThank you for raising this point. Our model is trained to be robust to missing or partially corrupted modalities. During pretraining we encounter natural sensor missingness and apply imputation strategies following established practice  [ref 6, ref 20], which allows the model to learn such patterns. \n\nTo further understand the effect of missingness on the performance, we conduct additional experiments on (1) *subgroup analysis stratified by percentage of missingness*, and (2) *comparisons when certain sensor modalities are entirely absent*.\n\n|Missing Ratio|Activity|Anxiety|Hypertension|\n|-|-|-|-|\n|<40%|0.94|0.68|0.62| \n|40-60%|0.94|0.65|0.59|\n|60-80%|0.91|0.65|0.65|\n\n|Modality Missing|Activity|Anxiety|Hypertension|\n|-|-|-|-|\n|w/o missing|0.94|0.67|0.62|\n|w missing|0.89|0.65|0.60|\n\nAs shown in the tables above, while missing modalities and increased missingness ratio lead to slight performance degradation, the model maintains strong robustness and generalization across tasks.\n\nFor noisy or corrupted inputs we likewise anticipate some degradation, as is typical for most models; however, consistent with the empirical scaling laws reported in **Appendix C.1**, SensorLM benefits from large-scale, diverse training data that may help mitigate this effect.\n\nWe will report these results and add a robustness discussion in the revised manuscript.\n\n&nbsp;\n\n---\nWe hope the above responses have addressed all concerns from the reviewer adequately. We would also like to update the reviewer on our commitment to open‑sourcing SensorLM:\n- **Dataset**. Under our IRB‑approved protocol, we have obtained informed consent to release a **de‑identified version** of the downstream dataset, including hypertension, anxiety, age, and BMI labels. Access will be limited to researchers with verified academic affiliations.\n- **Pretrained model**. We will release **SensorLM checkpoints** trained on these data together with example code for inference and fine-tuning.\n- **Code**. We will open-source the complete caption-generation pipeline to facilitate reuse by the broader research community.\n\nOnce again, thanks for your time and constructive feedback. We would really appreciate it if the reviewer could consider raising their scores and championing the paper."}]

Author rebuttal:

Meta-review: Paper present a large-scale dataset (of subjects and different devices, up to date the largest collected dataset in this domain), sensor data captioning, and pre-training framework for sensor-language foundation models for understanding wearable sensor data with natural language outputs. Hierarchical captioning is designed to capture statistical, structural, and semantic information from sensor data. Framework utilises and extends existing pre-training architectures to  unified framework. Framework and proposed dataset are experimentally evaluated in human activity recognition and healthcare applications, and compared to generic multimodal LLMs with promising results. 

Proposed work presents timely, interesting, and novel application of multimodal generative models for sensor-language data. Several
strengths are observed. The paper is considered well-written and organised (Reviewer oLSX), and the framework is well-illustrated and defined (Reviewer wWho). Furthermore, introducing  of very large-scale dataset collection and utilisation (Reviewers yhqJ and oLSX) is considered a good contribution and valuable for the field. The evaluations are well-done, extensive, and strong (Reviewers yhqJ, wWho, and ZU6J), and the support of test-time inference is seen valuable and practical (Reviewer ZU6J).

Several weaknesses have been indicated during the review phase, including the questions of model's out-of-domain generalisation capabilities (Reviewer yhqJ), handling of missing modalities (Reviewer yhqJ), limitations of their alignment strategies (Reviewer yhqJ), originality of the model components and loss function approaches (Reviewer oLSX), computational overhead (Reviewer ZU6J), and generalisation of the model to other time-series sensor data applications (Reviewer ZU6J). From evaluation perspectives, limited baseline comparison is indicated (Reviewer oLSX), and from dataset and code perspective, missing details of data processing pipeline (Reviewer yhqJ) as well as missing dataset and code open-sourcing during the review period (Reviewer wWho, oLSX).

During the rebuttal period discussion have been quite active, and authors have clarified the raised issues. Specifically, comprehensive set of new results and clarifications have been produced, which support the claims and addresses the main concerns of the reviewers. What is left is the concerns about data and code open-sourcing for the community. Paper has mixed rating both side of the borderline (from borderline reject to accept), indicating the concerns about data and code publishing and application orientation of the work, in general. Overall, paper has its merits and if all the improvements (including promised data and code releases) are done in the final revision, paper can be recommended for acceptance. Authors are especially encouraged to revise the paper with all promised modifications introduced during the rebuttal phase.
Decision: Accept (poster)