Scaling Wearable Foundation Models

## Abstract

Abstract Wearable sensors have become ubiquitous thanks to a variety of health tracking features. The resulting continuous and longitudinal measurements from everyday life generate large volumes of data; however, making sense of these observations for scientific and actionable insights is non-trivial. Inspired by the empirical success of generative modeling, where large neural networks learn powerful representations from vast amounts of text, image, video, or audio data, we investigate the scaling properties of sensor foundation models across compute, data, and model size. Using a dataset of up to 40 million hours of in-situ heart rate, heart rate variability, electrodermal activity, accelerometer, skin temperature, and altimeter per-minute data from over 165,000 people, we create LSM , a multimodal foundation model built on the largest wearable-signals dataset with the most extensive range of sensor modalities to date. Our results establish the scaling laws of LSM for tasks such as imputation, interpolation and extrapolation, both across time and sensor modalities. Moreover, we highlight how LSM enables sample-efficient downstream learning for tasks like exercise and activity recognition.

## 1 Introduction

Wearable devices that monitor physiological and behavioral signals have become ubiquitous. Increasing evidence suggests that these devices can significantly contribute to promoting healthy behaviors (Ringeval et al., ), detecting diseases (Yang et al., ), and enhancing the design and implementation of treatments (Munos et al., ). These devices generate large volumes of continuous, longitudinal, and multimodal data. However, raw data from sensors such as accelerometers or photoplethysmography (PPG) hardware are often challenging for both consumers and experts to interpret. To address this issue, algorithms have been developed to translate sensor outputs into more meaningful representations, such as step counts and heart rate.

Historically, algorithms for wearable sensors have relied on supervised, discriminative models designed to detect specific events or activities (Lubitz et al., ). This approach, however, faces several significant limitations. First, the limited volume and severe data imbalance of labeled events results in large amounts of valuable *unlabeled* data being left unused. Second, supervised models are typically trained for a single task (e.g., classification), producing representations that may not generalize well to other tasks. Third, training data is often collected from small study populations (usually involving only tens or hundreds of participants), leading to a lack of diversity in the data.

Figure: Figure 1: Scaling foundation models on wearable data. Making sense of physiological and behavioral signals derived from wearables is challenging. (A) We present a systematic scaling analysis of sensor models using up to 40 million hours of multimodal data from over 165,000 people. (B) Using a random masking pretext task, we evaluate on tasks of imputation, forecasting, and downstream classification. (C) Experiments show scaling compute, data, and model size are all effective. Scaling is shown on the random imputation task.
Refer to caption: x1.png

Self-supervised learning (SSL) using generic pretext tasks (Noroozi et al., ; Caron et al., ; Yang et al., ) can yield versatile representations that are useful for a wide range of downstream applications.
SSL allows for the use of a much larger proportion of available data without being restricted to labeled data regions (e.g., a limited number of subjects who self-report labels for exercises/activities).
These advantages have motivated efforts to apply similar training strategies to build models from large volumes of unlabeled wearable data (Adaimi et al., ; Thapa et al., ; Yuan et al., ; Abbaspourazad et al., ) (see Table 1 for a summary).

Building on this, the empirical and theoretical success of scaling laws in neural models (Kaplan et al., ; Bahri et al., ) suggests that model performance improves predictably as compute, data, and model parameters increase. These findings raise a critical research question: Do scaling laws apply to models trained on wearable sensor data? We aim to investigate whether the principles that drive the scaling of neural networks in domains like language and vision also extend to large-scale, multimodal wearable sensor data. Understanding how scaling manifests in this context could not only shape model design but also enhance generalization across diverse tasks and datasets.

In this paper, we present the results of our scaling experiments on the largest and the most diverse wearable dataset published to date, comprising 40 million hours of multimodal sensor data from over 165,000 users (Fig. 1). Leveraging these data, we train a foundation model, referred to as the Large Sensor Model (LSM), which is designed to capture generalizable representations across diverse populations, wearable sensor modalities, and downstream tasks.
We demonstrate the scaling properties of LSM with respect to compute, data size, and model parameters, leading to substantial performance gains on generative imputation, interpolation and extrapolation as well as downstream discriminative tasks. Our contributions can be summarized as follows:

- •
Implementation of the largest study to date on the scaling behavior of sensor foundation models, encompassing 40 millions hours, over 165,000 users and multiple sensor modalities, including accelerometer, photoplethysmography (PPG), electrodermal activity (EDA), skin temperature, and altimeter signals.
- •
Identification of key strategies for training large-scale sensor foundation models (LSM), and the LSM’s scaling properties with respect to compute, data size, and model parameters.
- •
Demonstration of the model’s ability to impute, interpolate, and extrapolate across temporal and sensor modalities, with a particular focus on generalization to unseen users.
- •
Verification that learned representations can be applied to downstream classification tasks, such as exercise and activity recognition, using ecologically valid, user-annotated events.

## 2 Related Work

Sensor Foundation Models.
Recent advances have demonstrated improved accuracy, robustness, and generalizability of models for sensor data by utilizing self-supervised pretraining on large-scale corpora of behavioral and physiological signals (Yuan et al., ; Thapa et al., ; Merrill and Althoff, ).
Existing sensor foundation models primarily leverage contrastive learning, creating positive and negative data pairs (Yuan et al., ; Thapa et al., ; Abbaspourazad et al., ). Yuan et al. () employ time domain augmentations (e.g., reversal, warping, permutation) to formulate the SSL task for motion data. Abbaspourazad et al. () adopt a similar strategy, incorporating Gaussian noise, time and magnitude warping, and channel swapping. Thapa et al. () generate data pairs using different sensory modalities. In contrast, we focus on masked input modeling due to the generative capabilities that it offers and explore its properties when scaling compute, data size, and model size. Compared to prior work we consider more sensor inputs, a larger data sample, and systematically investigate scaling laws (see Table 1). We also present contrastive baselines (Assran et al., ; Chen et al., ) where applicable.

**Table 1: Comparisons of studies on wearable sensor foundation models.**
| Study | # People | # Hours | Sensors | Generative |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | (000s) | (000s) | ECG | PPG | ACC | SCL | TMP | ALT |  |
| Adaimi et al. () | 0.05 | 0.20 | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✓ |
| Abbaspourazad et al. () | 141 | 400 | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ |
| Yuan et al. () | 100 | 15,700 | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ |
| LSM (Ours) | 165 | 40,000 | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |

Time-Series Foundation Models.
Wearable sensor data typically takes the form of multivariate time series. Foundation models for time-series signals have been trained and evaluated on data from domains such as energy use, transportation, finance, and climate.
TimeGPT (Garza and Mergenthaler-Canseco, ) and Lag-Llama (Rasul et al., ) represented early versions of pretrained models for predicting time-series signals. Families of models for general-purpose time series analysis emphasize common properties present in many signals, even those from different sources (Goswami et al., ).
Recent efforts explore different model architectures (Das et al., ) and scaling multiple data sources (Ansari et al., ), examing how language models can perform zero-shot reasoning (Liu et al., ; Merrill et al., ).
Yet, time series from different domains can exhibit considerably different properties. Drawing inspiration from prior work, we focus on the analysis of sensory time-series data, exploring scaling behavior, and interrogating whether they are consistent with other domains or show unique properties.

Scaling Laws in Deep Learning.
The scaling of computational resources, data volume, and model size has driven remarkable advancements in deep learning (Zhai et al., ; Kaplan et al., ; Xie et al., ). Recent investigations indicate that testing loss follows a power law relationship with each of these three resources when the other two are held constant (Kaplan et al., ). Power law behavior has been observed across various domains, including large language models (Kaplan et al., ), large vision models (Zhai et al., ), transfer learning (Hestness et al., ), and multimodal models (Aghajanyan et al., ). In this work, we take a step further and investigate the scaling behavior of training foundation models for multimodal wearable sensor data.

## 3 Data for Wearable Foundation Models

### 3.1 Sensor Data and Processing

Fitbit Sense 2 and Pixel Watch 2 have four *sensors* of highest relevance to this work: PPG, accelerometer, skin conductance, and altimeter/pressure sensors. From these input signals we compute a set of 26 *signals* (features), as described in Table 15 of Appendix E. Raw sensor data is not stored at this scale as it would impact the battery life and memory on the device. Thus, we focus on one-minute resolution signals.

Skin Conductance. The EDA sensor is used to infer sympathetic arousal via changes in micro-sweat levels, a physiological response to stress. Two electrodes on the back of the device measure changes in skin conductance level (SCL), which varies with skin moisture levels. SCL data is sampled at 200 Hz, downsampled to 25 Hz via a boxcar filter, and smoothed with a 5-minute median and low-pass filters (McDuff et al., ). Per-minute tonic SCL slope and magnitude are then calculated. Due to the nature of the sensing mode operation, SCL data is only collected during non-exercise wake-periods.

Skin Temperature. A temperature sensor located near the wrist-facing surface of the device takes measurement every 10 seconds. Per-minute slope and magnitude values are calculated via linear regression. Skin temperature signals are available whenever EDA signals are available.

Photoplethysmography. A validated algorithm (Nissen et al., ) is used to extract heart rate (HR) once per second from PPG. The per-minute HR data was calculated by taking the mean of the interpolated, per-second data across non-overlapping one-minute windows. An on-device peak detection algorithm identified PPG-based R-wave peaks from which RR intervals were calculated. RR intervals are susceptible to noise from multiple sources, including movement, electronic noise, and missed heartbeats. To account for noise, outliers were removed from each sliding 5-minute window using the median-filter based approach (Natarajan et al., ). The percentage of each 5-minute window with valid RR intervals are calculated and referred to as “heart rate variability (HRV) percent good”.
Nine standard HRV metrics (Shaffer and Ginsberg, ) are calculated every minute over a sliding 5-minute window: RR mean, RR median, RR 20^th percentile, RR 80^th percentile, RR Shannon Entropy, RR differences Shannon Entropy, standard deviation of RR, root mean squared difference of RR intervals, and percentage of RR intervals greater than 30ms (PNN30).

Accelerometer. Ten signals are extracted from the 3-axis accelerometer: Jerk, steps, accelerometer log energy and energy ratio, covariance, number of zero crossings and standard deviation.
These signals are extracted by converting the 3-axis accelerometer to root mean squared magnitude (1D), and applying a high-pass filter (HPF) to the remove DC component. In parallel, the 3-axis accelerometer signal is put through a second-order band-pass filter (BPF) and the principal component of the filtered 3-axis signal covariance matrix is calculated. In brief, jerk is a measure based on the time-derivative of the acceleration calculated from the principal component. It is the logarithm of the ratio of the absolute of the t=1 autocorrelation lag over the t=0 autocorrelation lag. Steps is a per-minute count of steps taken. Log energy is the logarithm of the sum of the squared HPF signal over the window. Log energy ratio is the logarithm of the ratio of energy computed from principal-component over the magnitude of the HPF signal. Zero-crossing count is the number of crossings in the principal component. Kurtosis is the kurtosis of the BFP signal.

Altimeter. The standard deviation of the altimeter (pressure sensor) measurements.

Table: Table 2: Details of the datasets. Summary of the demographic composition of our pretraining set and class distribution of our downstream set samples.

All sensor signals were globally normalized (z-score) to remove differences in magnitude due to different units of measurement. As the masked autoencoder cannot process missing data, we imputed minutes that had missing values. Within each 300-minute window, missing data between valid data points was linearly interpolated, and leading missing minutes were backfilled.

### 3.2 Building A Large Scale Pretraining Sensor Dataset

To build the large dataset for our experiments we sampled wearable data from 165,090 subjects during the period January 1^st 2023 to July 2^nd 2024. The subjects wore Fitbit Sense 2 or Google Pixel Watch 2 devices and consented for their data to be used for research and development of new health and wellness products and services. We sub-selected from people wearing one of these devices as older device generations included fewer sensors. The subjects were asked for self-reported sex, age and weight. Table LABEL:tab:demographics summarizes the characteristics of the pretraining data. All data were de-identified and not linked with any other information. To create a dataset that maximized the number of subjects we randomly sampled 10 5-hour windows of data from each subject, for a total of 8 million hours (6.6 million pretrain hours). We further explore the extremes of data scaling by experimenting with a subject-imbalanced 40 million hour pretraining dataset (see Appendix B.1).

The dataset was split 80-20 based on subjects into train-test splits.
We then created several “slices” of the training set to conduct the scaling experiment. The test set remains identical throughout all experiments. In the “sample-scaling” experiments we shuffled the training data and took N samples per experiment. In the “subject-scaling” experiments we grouped the training data by subject identifier and took all samples from N subjects per experiment.

## 4 Sensor Modeling Tasks

### 4.1 Generative Tasks

We posit that defining generative tasks in the training of wearable sensor models may not only result in learned representations that are useful for downstream classification tasks, but also produce models that can impute missing or incomplete data (interpolate) and extrapolate future sensor values (forecast). To train the model and to test these capabilities we define several tasks (see Fig. 2).

Random Imputation. Our primary pretext task involves removing patches randomly from the input sample across the time-axis and signal-axis. During training this requires the model to infer missing values and make predictions based on partial input.

Temporal Interpolation. Sensor inputs can be missing for a number of reasons. Devices need to be removed from the wrist for charging, and certain sensors might be turned off for periods to save on battery life (McDuff et al., ). Interpolation of sensor data is an important and necessary step for many algorithms (see Fig. 2). In this task we test the model’s ability to fill gaps in the data where all sensor data is missing for a period of time, usually between two observations.

Sensor Imputation. Sensor imputation refers to the process of inferring a subset of partially missing sensor-streams, from other continuously online sensing modalities. By leveraging correlations between different physiological signals, sensor imputation ensures that insights can be derived even when some sensor modalities are absent, enhancing the overall versatility and capabilities of multi-sensor systems. Under the constraints of hardware limitations (battery, wireless connectivity, etc.), sensor imputation can enable the delivery of more realistic metrics to the user (e.g., step count, average resting heart rate) even if when sensors are not continuously online.

Temporal Extrapolation (Forecasting). A more challenging task than interpolation is extrapolation of sensor values forward in time. Temporal extrapolation involves predicting future sensor measurements. The ability to anticipate future physiological states based on current and historical data has applications in areas such as health interventions, where extrapolation can be used to schedule recovery times, detect early signs of fatigue, predict wake-up times, and detect anomalies. Accurate signal extrapolation is a key task that can empower wearable devices to provide more just-in-time, proactive, and personalized health recommendations.

Figure: Figure 2: Generative LSM tasks and pretraining. We define four distinct generative tasks: random imputation, temporal interpolation, signal/sensor imputation, and temporal extrapolation (forecasting). Random imputation was empirically chosen as the pretraining task.
Refer to caption: x2.png

### 4.2 Discriminative Tasks

Discriminative tasks focus on classifying or identifying specific activities, states, or conditions based on sensor data. These tasks are essential for translating raw sensor inputs into actionable, personalized, and relevant feedback. Two exemplary tasks are considered here.

Exercise Detection. Exercise detection identifies when a user is exercising, enabling real-time feedback and performance tracking. This task involves recognizing exercise events from continuous sensor data, allowing devices to log workout sessions, track progress, and provide personalized recommendations. Additionally, detecting exercise unlocks related experiences, such as identifying exercise types, marking session start times, or tracking post-exercise feedback. We developed a dataset with windows of user-labeled exercise and non-exercise events (see Table LABEL:tab:activity_events).

Activity Recognition. Activity recognition is the process of classifying different user activities such as biking, running, or walking, based on the patterns detected in sensor data. This allows wearable devices to monitor daily routines accurately, providing insights into fitness levels, activity trends, and overall health. Effective activity recognition enables applications like fitness tracking, lifestyle monitoring, and personalized coaching. Our dataset includes eight user-labeled activities: *Biking*, *Elliptical*, *High-Intensity Interval Training (HIIT)*, *Strength Training*, *Swimming*, *Running*, *Walking*, and *Weightlifting*.

## 5 Experiments & Results

### 5.1 Training Procedures

We pretrain wearable foundation models on a diverse collection of multimodal sensor data from 80% of the 165,090 subjects as described in Table LABEL:tab:demographics. Each sample is processed as a two-dimensional matrix of 26 signals by 300 minutes (see Fig. 2). Our primary pretraining objective is to optimize the masked signal reconstruction loss (i.e., mean squared error), averaged over randomly masked patches from the input sequences (He et al., ). The primary performance metric is the mean squared error on the held-out test set, evaluated across all the normalized signals.

We train our models on Google v5e TPUs with a total batch size of 4096 across 50,000 training steps. The training process uses the AdamW optimizer with a base learning rate of $5e-3$ and weight decay set to $1e-4$. A linear warm-up schedule is applied for the first 2,500 steps, followed by a cosine learning rate decay to zero. All pretraining experiments use an 0.8 masking ratio (masking out random patches that cover 80% of the total input signals). Additional details on implementation and hyperparameters can be found in Appendix C.

Figure: Figure 3: Scaling performance of LSM. We show performance on generative tasks across varying data and model sizes. LSM begins to saturate at approximately $10^{7}$ hours of data. The effects of scaling are more pronounced in imputation, interpolation, and extrapolation tasks. Results indicate that as model size increases, significantly larger data volumes are required to prevent overfitting.
Refer to caption: x3.png

### 5.2 Results & Discussion

Do scaling laws apply to wearable data? We present the Pareto front of the reconstruction loss and downstream performance as a function of compute scaling (see Fig. 1). The front highlights the models with optimal compute allocation across model size, data size and training duration. Over multiple orders of magnitude of compute, the relationship between compute and performance follows a power-law ($L=aC^{b}$), resulting in a nearly linear trend on the log-log plot. However, we observe a saturation effect at the upper end of the compute spectrum, where the largest models do not asymptotically approach zero error. This behavior has also been observed for scaling language models (Henighan et al., ) and vision transformers (Zhai et al., ); therefore, we add an additive constant $c$ to model this saturation effect: $L=aC^{b}+c$.

We illustrate data scaling across various model sizes (Fig. 3(a)). Performance improves monotonically to approximately $10^{5}$ data hours, beyond which the rate of improvement diminishes, particularly around $10^{7}$ hours. We validated that scaling beyond $10^{7}$ hours yields minimal benefits by training with 40 million hours (see Appendix B.1). Consequently, results in Table 5.2 are pretrained with 6.6 million hours of data. Larger models, especially the ViT-110M, continue to benefit from data scaling, showing substantial gains when training on over 1 million hours of data. These observations underscore the large data requirements needed to fully exploit the capacity of larger models, which are far greater than those required by smaller models. A similar trend is observed in discriminative tasks (Fig. LABEL:fig:data_scaling_dis_task). We further note that these trends are based on minutely aggregated wearable data; raw sensor signals are traditionally collected at substantially higher sampling frequencies and it is possible that feature extraction on more fine-grained sensor data may require even larger models.

Model scaling results as a function of data size demonstrate that as both model size and dataset size are scaled, sufficient data is essential to prevent overfitting (Fig. 3(b)). Models trained on smaller datasets exhibit limited generalization capacity, whereas scaling up to $10^{8}$ parameters results in significant gains in test loss and generative zero-shot performance. These findings highlight the need to align model size with adequate data to fully leverage the model’s representational power. Our experiments also show larger models are more sample efficient as illustrated in Fig. LABEL:fig:model_efficiency.

By scaling compute, data, and model size together, LSM achieves improvements of 16% to 23% in temporal interpolation MAE and 20% to 21% in extrapolation MAE across five time durations as compared to the best baseline method (Table LABEL:tab:generative_task_results). Additionally, LSM outperforms baselines in exercise detection and 8-class activity recognition over the supervised baseline by 27% / 29% in accuracy and 57% / 54% in mAP, as detailed in Table LABEL:tab:discriminative_task_results_activity. Our baseline approaches are commonly used in existing sensor algorithms (Gershon et al., ; van Rossum et al., ). More scaling results can be found in Appendix B.

Is scaling subjects or wearable data hours per subject more helpful? As shown in Fig. LABEL:fig:data_subjects_scaling, when training using the same total number of wearable signal hours, reducing the number of subjects (but drawing more hours per subject) can yield similar performance. This suggests that total number of hours rather than number of subjects drives gains. One possible hypothesis to explain this effect is that the diversity of activities per subject (as reflected by the increase in hours per subject) plays a crucial role. Alternatively, subject diversity may become more important when learning representations if we scale up the data sample size from 5 hours (e.g., 7 days vs. 5 hours). As each subject has a finite number of hours, to maximize model generalization, it is important to scale both the number of subjects and the wearable data hours per subject simultaneously. While temporal data is crucial for capturing intra-subject variability, increasing the number of subjects introduces valuable inter-subject diversity. Therefore, scaling both dimensions—subjects and hours—together is essential to fully leverage the model’s capacity and improve performance across tasks.

Can wearable foundation models impute the past and predict the future? As shown in Fig. 3, scaling laws apply to all imputation, interpolation, and extrapolation tasks, with larger models and more data resulting in improved performance. The utility of LSM is further emphasized in Table LABEL:tab:generative_task_results. However, despite these quantitative gains, the qualitative results in Fig. 12 of Appendix B.6 reveal that these tasks remain highly challenging. Imputing large portions of missing data, especially over extended time intervals, often leads to degraded accuracy, with performance deteriorating as the missing data window increases. Similarly, extrapolation further into the future (e.g., several hours ahead) introduces significant uncertainty, making it difficult to predict fine-grained physiological or behavioral patterns. These findings suggest that while scaling helps improve generative capabilities, substantial challenges remain, particularly in handling long-range dependencies and large data gaps.

Are wearable foundation models label efficient on discriminative tasks? Our experiments on probing, fine-tuning, and few-shot learning for activity indicate that wearable foundation models are highly label efficient. As shown in Table LABEL:tab:discriminative_task_results_exercise and LABEL:tab:discriminative_task_results_activity, the performance of the fine-tuned LSM consistently outperforms supervised baselines. A confusion matrix of the best performing model is shown in Fig. 8. As shown in Table 11 of Appendix B.2, even in the low-data regime (e.g., 5-shot, 10-shot), foundation models demonstrate strong generalization capabilities, achieving significantly lower error rates compared to models trained from scratch or with limited supervision. As the number of labeled examples increases, the performance gap widens, with foundation models leveraging pretraining to more effectively transfer learned representations to downstream tasks. T-distributed Stochastic Neighbor Embeddings (t-SNE) plots show the impact of pretraining on more data and fine-tuning are shown in Appendix B.4 (Fig. 9).

Table: Table 3: Comparisons of LSM and competing methods on generative and discriminative tasks.

### 5.3 Further Analysis and Ablation Studies

Ablation of Model Design Choice (Appendix A). We analyze the impact of design choices on LSM performance, including masking ratios and strategies, signal orders, patch sizes, and model sizes.

Qualitative Analysis of LSM (Appendix B.4 & B.6). We further explore the learned feature embeddings to assess their sensitivity to personally identifiable features (e.g., age and gender) and examine the reconstruction quality of the signals.

## 6 Limitations & Future Work

Our experiments indicate promising opportunities in scaling wearable sensor models but also highlight several unresolved questions. Notably, we observe saturation in scaling laws with a dataset size of $10^{7}$ hours and model sizes in 100 millions. We attribute this to three factors: (1) the current pretraining task may not be sufficiently scalable, and decoder-only approaches might better leverage data rather than filling masked inputs; (2) the dataset construction lacks sufficient challenge, and extending the sensor context window from 5 hours to a day or even a week could introduce more complexity that enables the model to learn longer time dependency relationships; (3) our data cleaning process was minimal, and increasing data diversity, akin to large-scale language model training, could significantly enhance model generalization. For example, while our dataset spanned all four seasons, there was an imbalance in temporal coverage, with two years of data from January to June but only a single year from July to December. This uneven distribution could bias the model towards activities more common in the earlier part of the year.

A key characteristic of wearable sensor data is its inherent missingness. Handling missing data in both pretraining and downstream tasks remains an open question. While we used imputation for this study, a more principled approach would involve designing models that naturally account for missing data without introducing imputation biases. The nature of missing data in wearable sensors often correlates with real-world events (e.g., charging the device, loose fitting), which can mean that data is missing not at random (MNAR). Understanding these factors and designing methods to handle them robustly remains an important direction for future work. Lastly, we acknowledge the lack of comprehensive evaluation on more discriminative tasks. Future work will expand the dataset to include a broader range of classification and regression tasks, which will provide a more thorough demonstration of the benefits of our pretrained models.

## 7 Broader Impact

Wearable sensors have been shown to have a positive effect on health and well-being, promoting physical activity, sleep and have potential to surface unseen or unperceived actionable health information. Foundation models increase the potential value of these data for the above applications and hold promise for enabling new insights and opportunities to improve health.

We support open science principles and the value of open data for scientific research; however, we have to balance these considerations with the privacy of the participants and protection of their health data. Although the training data could be de-identified, some of the data streams could not be fully anonymized.
We recognize that the inability to share data of this kind is a limitation; however we believe that the results enable us to share valuable insights to the community.

Meanwhile, LSM serves as the stepping stone towards generating large-scale, realistic synthetic datasets. These synthetic data could mimic real-world sensor patterns without compromising participant privacy and offer a promising resource for cross-institutional research collaboration.
By facilitating data sharing in this way, we can overcome the current limitations in data availability and unlock new opportunities for collaborative insights and advancements for the community.

## 8 Conclusion

We present LSM, a large multimodal foundation model trained on 40 million hours of wearable sensor data from over 165,000 individuals, establishing scaling laws for sensor models. LSM significantly improves performance across generative tasks such as imputation, interpolation, and extrapolation, as well as discriminative tasks like exercise detection and activity recognition. Our results demonstrate that scaling data, model size, and compute leads to substantial gains in generalization and efficiency. LSM highlights the potential of scaling wearable sensor models for real-world health applications, enabling more robust and efficient downstream tasks.

## Appendix

[appendices]

## Appendix A Model Design Choices and Ablations

We perform ablations on the configurations used for our masked autoencoder LSM design. Following the convention of previous works (He et al., ; Huang et al., ), we explore masking ratio, masking strategies, patch sizes, and model sizes. Uniquely, we explore the ordering of sensor signals, as these signals do not share the same explicit ordered dependencies as exist in images and audio spectrograms. For all experiments we employ random masking, a 0.8 masking ratio, ordered sensor signal order, a patch size of 10x5, and a LSM-Base (110M) backbone, unless otherwise specified.

### A.1 Selecting a Masking Ratio

Selecting the appropriate masking ratio is critical for ensuring effective representation learning in our sensor MAE training. We explore different masking ratios, ranging from 30% to 90%, to evaluate their impact on reconstruction quality and model generalization. We find that a masking ratio of 80% yields the best performance on temporal interpolation and extrapolation as shown in Table 4).

Table: Table 4: Ablation study of masking ratios.

### A.2 Selecting a Masking Strategy

To train a wearable foundation model effective for both generative and discriminative tasks, mask-based pretraining proves superior to contrastive pretraining. Choosing the right masking strategy is crucial, as it directly influences the quality of the learned embeddings and the model’s generalizability. In Table 5, we systematically compare five different masking strategies and demonstrate that random masking consistently yields the best performance across the two primary generative tasks. Example visualizations of these masking strategies can be seen in Fig. 6.

Table: Table 5: Ablation study of masking strategies.

Figure: Figure 6: LSM MAE pretrain masking strategies. All strategies employ a masking ratio of 0.8. (A): original, unmasked sensor image, (B): random masking, (C): structured temporal masking, (D): structured sensor masking, (E): temporal extrapolation masking, (F): temporal interpolation masking. Both random and structured temporal masking enable strong down-stream performance. We select random masking for all scaling experiments and evaluations.
Refer to caption: x4.png

### A.3 Selecting a Sensor Signal Order

For multimodal sensor data, the order in which signals are processed by the model can impact performance. Specifically, for architectures, such as vision transformers, that take patched inputs, the clustering of signals in patches may have a profound impact on the learned representation. We evaluate ordering the sensor signals by: (a) sensor types (as in Table 15), (b) randomized order (repeated with several random seeds) and (c) interleaving signals with uncorrelated signals. Cross correlation matrices are shown in Fig. 11. We find that ordering by clustering sensor type generally yields better results (see Table 6), particularly when dealing with heterogeneous sensor modalities like accelerometry, electrodermal activity (EDA), and heart rate. This order allows the model to leverage specific sensor characteristics more effectively, improving performance on downstream tasks.

Table: Table 6: Ablation study of sensor orders.

### A.4 Selecting a Patch Size.

Patch size in our pretraining is defined by time steps and the number of sensor features per patch, both impacting model capacity and computation (gFlops). In contrast to previous works  (He et al., ; Huang et al., ) we expensively sweep across both dimensions of the input. This is critical for sensor models, where both dimensions, of time and features, share unique correlations and dependencies along their corresponding axis.

A time-step of 10 minutes strikes the best balance with low gFlops (15.94) and strong performance (MAE of 0.24 for imputation, 0.37 for forecasting) (See Table 7). Similarly, increasing features per patch shows that five features per patch achieves the best trade-off between accuracy and computational cost, outperforming both smaller (10x1) and larger patches (10x26). Thus, a moderate patch size of 10 minutes by 5 features is what we select.

As a patch-size of 10-minutes x 5-sensors (10x5) cannot evenly patch a input sensor-image of 300-minutes x 26-sensors (300x26), we zero-pad the sensor dimension to 30 resulting in an 300-minute x 30-feature (300x30) input sensor-image.

Table: Table 7: Ablation study of patch sizes.

### A.5 Model Size Variants

In Table 8, we present four variants of the LSM models we trained. The model sizes and naming conventions partially follow the tradition established by T5 (Raffel et al., ). Our results indicate that scaling the model beyond LSM-B offers no additional improvements in either reconstruction loss or downstream task performance. Based on this insight all neural methods in Table 5.2 employ a ViT-110M backbone.

Table: Table 8: Vision transformer size variants used in LSM. An LSM-[size] model indicates a ViT-[size] backbone.

## Appendix B Additional Results and Analysis

### B.1 Results of Scaling Experiments for Generative Tasks

Generative Performance wrt. Data Scaling. Table 9 presents the full results for the generative tasks, evaluated across four model sizes and all data scales, including an experiment on the largest 40 million hour pretraining set. The LSM Base model, trained on 6.6 million hours of data, achieved the best overall performance.

Scaling Pretraining Data to 40 Million Hours. As mentioned in Sections 3 and 5, we derive our dataset, used for presented scaling and downstream task results from 6.6M hours of data balanced across 160K people. To test the extremes of data scaling we also build a dataset comprising of 40M data hours by combining the 6.6M hours with an additional 33M hours of data from a 78569 subject subset of the total 160K subjects. However, as shown in Table 9, we observed that scaling benefits taper off when training the LSM-Base model with this extended dataset. We believe this is due to two key factors: the structure of our dataset and the inherent limitations of the masking pretraining task, as discussed in Section 6. It is also possible that as the additional 33M hours are not evenly distributed across subjects that the careful balance of the 6.6M dataset is disturbed.

### B.2 Results of Scaling Experiments for Discriminative Tasks

Figure: Figure 7: Few shot learning. Activity recognition results.
Refer to caption: x5.png

Discriminative Performance wrt. Data Scaling. In Table 10, we demonstrate that scaling up the dataset significantly benefits downstream discriminative tasks, particularly in the fine-tuning stage. Furthermore, our pretrained LSM model exhibits superior performance in label-efficient transfer learning, as shown in Table 11. Activity recognition few-shot results, as compared to a supervised baseline, are also visualized in Fig. 7. From the visualization it is clear that pretraining helps LSM learn a strong representation of sensor data that enables more sample efficient performance on discriminative tasks.

Convolutional Probe. Following prior work (He et al., ) we explore an intermediary evaluation to linear probing and full-model fine-tuning. Specifically, we explore the learnable pooling of embeddings. This probe takes patch-embeddings, produced by the encoder, and reshapes them to [num. patches $H$, num. patches $W$, embedding dimension], similar to the shape of the original patched sensor-image. This embedding is fed through two shallow convolutional layers and a linear head. We find that with less that 0.2% of the trainable-parameters needed for full-model fine-tuning, we are able to achieve similar performance on exercise detection and activity recognition tasks. These results can be seen in Tables 10 and 11.

Table: Table 9: Detailed Results of Generative Tasks. Performance across Data and Model Sizes on Generative Tasks. Data Size is in hours.

Table: Table 10: Data Scaling on Discriminative Tasks.

Table: Table 11: Few-Shot Performance on Discriminative Tasks.

### B.3 Classification Confusion Matrices

Fig. 8 presents the complete confusion matrix for our activity recognition task from the full-model fine-tuned LSM-B model. Note that many classes get mistaken for Walk. This is likely as there are significant periods of walking in the 5-hour inputs, even if the activity is labeled otherwise.

Figure: Figure 8: Activity recognition confusion matrix. Results for the full-model fined-tuned LSM Masked Auto-Encoder.
Refer to caption: x6.png

### B.4 Feature Embeddings

We present t-distributed Stochastic Neighbor Embedding (t-SNE) plots. In Fig. 9 illustrates that scaling pretraining data results in noticeable, albeit subtle, improvements of clustering across activities in the learned representation. We also find that fine-tuning the model is critical to effectively discriminate between activities.
In Fig. 10 we see that the learned representation does embed some subject dependencies. This can be attributed to variance in the physiology and activity definitions for individuals (e.g., a hard run may look very different for two different people).

Figure: Figure 9: t-SNE Embeddings for Pretraining and Fine-tuned Models Labeled by Activity. t-distributed Stochastic Neighbor Embedding (t-SNE) plots showing that there are differences (albeit subtle) between pretrained embeddings using data from almost 50k and 6.6 hours.
Refer to caption: extracted/5921236/fig_08.png

Figure: Figure 10: t-SNE Embeddings Labeled by Gender, Age and Subject. t-distributed Stochastic Neighbor Embedding (t-SNE) plots showing that the learned embeddings do capture subject specific information (and therefore also exhibit some subtle gender and age clusters). Age was not available for all subjects.
Refer to caption: extracted/5921236/fig_09.png

### B.5 Signal Correlations

The 26 signals used as input to our model come from four sensors (accelerometer, PPG, temperature, altimeter). As a result, some signals are more correlated with certain ones than with others.. A signal diagonal correlation matrix was calculated to show the pairwise correlations between signals. Fig. 11 shows the correlation matrix for signals clustered by sensor and for signals ordered to minimize the absolute correlation coefficient between adjacent features.

Figure: Figure 11: Sensor Signal Diagonal Correlation Matrix. The pair-wise correlation between the 26 sensor features based on our training set.
Refer to caption: x7.png

### B.6 Examples of Reconstructions

A qualitative example of ground-truth signals and corresponding reconstructions are shown in Fig. 12. The gray regions are sections that were masked in the input. Additional sensor-image level reconstructions, across generative down-stream tasks (eg. imputation, extrapolation, interpolation) can be seen in Fig. 13. Examples of the often visually subtle affects of scaling on reconstruction can be seen in Fig.  14.

Figure: Figure 12: Example of Signal Reconstructions. Comparison between the ground-truth (blue) and reconstruction (black) for a 5-hour sample. Gray regions were masked in the input. 80% Random Masking (Patch Size 10 mins x 5 sensors). Note: model outputs are only shown for the masked regions in the reconstructions.
Refer to caption: x8.png

Figure: Figure 13: Examples of Signal Reconstructions Across Generative Down Stream Tasks. The top row of each sample shows the original sensor signal image. Subsequent row-pairs plot the masked input followed by the model reconstruction below. All reconstruction come from LSM-Base based LSM employing a 10x5 patch size and pretrained with 80% random masking. Note: model outputs are only shown for the masked patches in the reconstructions.
Refer to caption: x9.png

Figure: Figure 14: Examples of Signal Reconstructions with Respect to Scaling. These plots illustrate the (often visually subtle) affect of A compute, B data, and C model scaling for sensor models. Note: model outputs are only shown for the masked patches in the reconstructions.
Refer to caption: x10.png

## Appendix C Details of Training and Hyperparameters

Hyperparameters. This section provides details about the pretraining and fine-tuning of LSM and other baseline methods. The pretraining hyperparameters, detailed in Table 12, were chosen with hyperparameter sweeps. In Table 13, we include hyperparameters for linear probe and fine-tuning. The hyperparameters for supervised baseline training are detailed in Table 14. Note that hyperparameters used for the few-shot experiments found in Table 11 are similar to those found in Tables 13 and 14 with slight changes in learning rate.

Training Augmentations. Traditional image augmentations are not always valid when applied to sensor-images. For example, random crop and resize, often applied in contrastive pretraining are invalid for sensor images, as a random crop may remove a subset of senor signals. Thus, we define a subset of augmentations valid for sensor-images. These are random Flip: a flip along the temporal axis; Stretch: a stretch along the temporal axis of and subsequent crop of back to original time length; and Noise: the addition of Gaussian noise.

**Table 12: Hyperparameters for pretraining with MAE (He et al., ), MSN (Assran et al., ), DINO (Caron et al., ) and SimCLR (Chen et al., ). A solitary row value indicates that the value was used for all methods.**
| Configuration | MAE | MSN | DINO | SimCLR |
| --- | --- | --- | --- | --- |
| Training Steps | 50000 |  |  |  |
| Warmup Steps | 2500 |  |  |  |
| Optimizer | AdamW (Loshchilov, ) |  |  |  |
| Opt. momentum [$\beta_{1},\beta_{2}$] | [0.9, 0.95] | [0.9, 0.99] | [0.9, 0.99] | [0.9, 0.99] |
| Base learning rate | 0.005 | 0.001 | 0.004 | 0.001 |
| Batch size | 4096 |  |  |  |
| Weight decay | 0.0001 |  |  |  |
| Gradient clipping | 1.0 | 3.0 | 3.0 | 3.0 |
| Dropout | 0.0 |  |  |  |
| Learning rate schedule | Linear Warmup & Cosine Decay |  |  |  |
| Loss Function | Mean Squared Error |  |  |  |
| Data resolution | 26 (signal)$\times$300(minute) |  |  |  |
| Augmentation | Flip, Stretch, Noise |  |  |  |

**Table 13: Hyperparameters for Linear Probing and Fine-Tuning on Discriminative Tasks detailed in Section 4.2. A solitary row value indicates that it was used for all methods. LP=Linear Probe. FT=Fine-Tune (full model).**
| Task | Exercise Detection | Activity Recognition |  |  |
| --- | --- | --- | --- | --- |
| Configuration | LP | FT | LP | FT |
| Training Steps | 400 | 400 | 300 | 300 |
| Warmup Step Percent | 20 | 20 | 15 | 15 |
| Optimizer | AdamW (Loshchilov, ) |  |  |  |
| Opt. momentum [$\beta_{1},\beta_{2}$] | [0.9, 0.95] |  |  |  |
| Base learning rate | 0.5 | 0.00005 | 0.5 | 0.00005 |
| Batch size | 128 |  |  |  |
| Weight decay | 0.0001 |  |  |  |
| Gradient clipping | 1.0 |  |  |  |
| Dropout | 0.3 |  |  |  |
| Learning rate schedule | Linear Warmup & Cosine Decay |  |  |  |
| Loss Function | Balanced Softmax Loss (Ren et al., ) |  |  |  |
| Data resolution | 26 (signal)$\times$300(minute) |  |  |  |
| Augmentation | Noise |  |  |  |

**Table 14: Hyperparameters for Supervised Training on Discriminative Tasks A solitary row value indicates that it was used for all methods.**
| Configuration | Exercise Detection | Activity Recognition |
| --- | --- | --- |
| Training Steps | 400 | 300 |
| Warmup Steps | 20 | 15 |
| Optimizer | AdamW (Loshchilov, ) |  |
| Opt. momentum [$\beta_{1},\beta_{2}$] | [0.9, 0.95] |  |
| Base learning rate | 0.0001 | 0.0005 |
| Batch size | 128 |  |
| Weight decay | 0.0001 |  |
| Gradient clipping | 1.0 |  |
| Dropout | 0.0 |  |
| Learning rate schedule | Linear Warmup & Cosine Decay |  |
| Loss Function | Balanced Softmax Loss (Ren et al., ) |  |
| Data resolution | 26 (signal)$\times$300(minute) |  |
| Augmentation | Noise |  |

## Appendix D Description of Pretraining and Baseline Methods

### D.1 Pretraining Methods

There are two main approaches to pretraining, one based on contrastive learning and the other based on the reconstruction or prediction of input features. At a high-level contrastive methods where representations are learned for different views of the same training example (positives), and dissimilar embeddings for different training examples (negatives). However, there are challenges or drawbacks to this approach. First, in the sensor domain it can be non-trivial to create augmentations that do not alter the meaning (label) of the sample. For example, does stretching data for someone *running* mean that it more closely resembles the data when they *walk*? Second, generative capabilities are attractive as imputing missing data and forecasting signals into the future are useful in and or themselves. As such, purely contrastive set-up has limitations and a pretraining task based on the reconstruction of masked input tokens is attractive. A masked autoencoder is one example of such an approach that is effective at scalable learning of representations (He et al., ). Below we describe the pretraining methods used for LSM and our baselines.

Masked Auto Encoder (MAE) (He et al., ).
MAE is a self-supervised learning method where the input data is randomly masked, and the model is trained to reconstruct the missing parts. It operates on the principle that forcing the model to predict missing information helps it learn meaningful representations. MAE has shown strong performance in various vision and signal tasks, particularly in cases where large-scale unlabeled data is available.

SimCLR (Chen et al., ).
SimCLR is a contrastive learning framework that learns representations by maximizing agreement between different augmented views of the same data sample. The method uses a contrastive loss, which encourages the model to pull together similar views of the same sample while pushing apart views of different samples. SimCLR has been widely used in both vision and sensor data for representation learning without requiring labeled data.

Masked Siamese Network (MSN) (Assran et al., ).
MSN combines the benefits of invariance-based pretraining with mask denoising. MSN operates by matching the representation of an image view with randomly masked patches to the representation of the corresponding unmasked image. This pretraining strategy leverages Vision Transformers by processing only the unmasked patches, significantly enhancing scalability. The framework enables the generation of semantically rich representations, which perform competitively in low-shot image classification tasks.

DINO (Caron et al., ).
DINO is a self-distillation method that trains the model using knowledge distillation, without the need for labeled data. It leverages a teacher-student network architecture, where the teacher generates target representations for the student to learn from. DINO has demonstrated success in generating robust representations that can be transferred to various downstream tasks.

### D.2 Generative Baselines

We define a number of baselines for our generative tasks. Similar methods are common-place in the image domain (often used for up-sampling) (Han, ), and the Internet of Things (IoT) sensor domain (often for imputing corrupted and/or missing data) (Adhikari et al., ).

Mean Fill.
Mean Fill is a simple baseline for generative tasks, where the missing values for a sensor stream are replaced by the mean value of the sensor data present in a given sample. Though naive, this method provides a reasonable estimate in certain contexts where missing values are randomly distributed.

Nearest Neighbor Fill.
Nearest Neighbor Fill imputes missing data by using the value of the nearest observed neighbor for a given feature along the temporal axis. In the absence of a past and future neighbors this method mirrors back/forward fill. This method works well when there is a high degree of local similarity in the data.

Linear Interpolation.
Linear Interpolation fills missing values by interpolating linearly between known values along the temporal dimension. In the absence of a past and future neighbors this method mirrors back/forward fill. This baseline is often used in time-series and spatial data, where the assumption is that changes between data points occur in a smooth, continuous manner.

For all generative baseline methods, in the rare cases where the sensor feature is completely missing, the feature values are replaced with zeros. This remains a valid strategy as all features are z-score normalized and centered around zero.

### D.3 Classification Baselines

Vision Transformer (ViT) (Dosovitskiy, ).
The Vision Transformer (ViT) is a transformer-based architecture that treats image patches or signal segments as input tokens, similar to how transformers handle sequences in natural language processing. ViT has shown competitive performance across various classification tasks, especially when trained with large amounts of data, and serves as a strong baseline in both vision and sensor classification tasks.

## Appendix E Additional Details of Dataset

In Table 15 we detail the 26 derived sensor signal features leveraged by out method.

**Table 15: Sensor Feature Definitions. Names, units and definitions of the 26 Accelerometer, PPG, skin conductance and altimeter features we use.**
| Feature | Unit | Definition |
| --- | --- | --- |
| SCL Skin Conductance |  |  |
| Skin Conductance Value | $\mu$Siemens | Center of linear tonic SCL value fit. |
| Skin Conductance Slope | $\mu$S/Min | Intraminute slope of SCL values. |
| TMP Skin Temperature |  |  |
| Skin Temperature Value | °C | Mean skin temperature. |
| PPG Photoplethysmography |  |  |
| Heart Rate | Beats/Min | Mean of instantaneous heart rate. |
| RR Percent Valid | % | % of 5-minute window with valid RR intervals. |
| RR 80^th Percentile | Msec | 80^th percentile of 5-minute window of RR ints. |
| RR 20^th Percentile | Msec | 20^th percentile of RR ints. |
| RR Median | Msec | Median RR interval. |
| RR Mean | Msec | Mean RR interval. |
| Shannon Ent. RR | Nats | Shannon entropy of the RR intervals.^∗∗ |
| Shannon Ent. RR Diffs | Nats | Shannon entropy of the RR interval differences.^∗∗ |
| PNN30 | % | % of successive RR ints. that change by $>$ 30ms. |
| RMSSD | Msec | Root mean squared st. dev. of RR ints. |
| SDNN | Msec | Standard deviation of RR intervals. |
| On Wrist | Boolean | If optical-sensor off-wrist within a 30-second window, then false. |
| ACC Accelerometer |  |  |
| Jerk Autocorrelation Ratio | a.u. | Ratio of lag=1 autocorrelation to energy in 1st 3-axis principal component. |
| Step Count | Steps | Number of steps. |
| Log Energy | a.u. | Log of sum of 3-axis root mean squared magnitude. |
| Covariance Condition | a.u. | Estimate of condition number for 3-axis covariance matrix. |
| Log Energy Ratio | a.u. | Log of ratio of sum of energy in 1st 3-axis principal component over energy of 3-axis root mean squared magnitude. |
| Zero Crossing St.Dev. | Seconds | Standard deviation of time between zero crossing of 1st 3-axis principal component. |
| Zero Crossing Average | Seconds | Mean of time between zero crossing of 1st 3-axis principal component. |
| Robust Arm-Tilt | a.u. | Log of mean square root of squared X & Z axes. |
| Kurtosis | a.u. | Kurtosis of 3-axis root mean squared magnitude. |
| Sleep Coefficient | a.u. | Sum of 3-axis max-min range, binned into 16 log-scaled bins. |
| ALT Altimeter |  |  |
| Altimeter St.Dev. Norm | Hectopascals | Standard deviation of altimeter readings. |

## Appendix F Code Acknowledgements

We build our methods upon the Scenic project (Dehghani et al., ), an open source codebase for vision tasks implemented in JAX with Flax. Scenic provides rich infrastructure for attention-base vision models and common vision baselines. The project page can be found here: [github.com/google-research/scenic](https://github.com/google-research/scenic).

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: Please refer to the weaknesses for detailed description of each points.

1. How the model would generalize to additional wearable activity tracking tasks? or event wearable health tasks?
2. Would the model and the representation generalize to unknown classes?
3. Can the authors provide more comparisons with time series models?
4. Describe the significance of the key take away of the paper. How it benefits the scalability of the model/approach in different settings.
5. Can the model be use for more diverse down stream tasks?
6. Clarify if you would open source the data and model?
7. Polish the write up and compress the text a little bit.
soundness: 2
strengths: Strengths:
1. The experiments are conducted on a large scale, and modeling a variety of signals including activities of heart, skin, and motion.
2. This work empirically shows that the scaling law of modeling can also be applied to the modality of wearable signals, in terms of scaling up computability, size of dataset, and size of model. 
3. The work includes reasonable baseline comparison with vision-based models.
summary: This work proposed a foundation model aiming to serve as a general data encoder for wearable time series data. The author leverages signals capturing a variety of physiological activities for pre-training including activities of heart, skin, and motion. The model is evaluated on the tasks including time series imputation, forecasting, and human activity recognition. The author also shows that the scaling law also applied on large models for the modality of wearable signals.
weaknesses: Weaknesses:
1. The model has fixed shape input, which raises concern about generalization. For example, it is very common that different devices and scenarios have different sets of sensors with different sampling rate. Some wearables have PPG + inertial sensor data only. Some have other more sophisticated sensors like Galvanic Skin Response and Electrocardiogram. It’s unclear exactly how such a model would work with different sets of input modalities with diverse sampling and data resolution settings. 

2. It is not clear why modeling the time series wearable signal as an image only as opposed to other alternatives like spectral representations (fourier transformation, wavelet transformation). Wearable signals are essentially time series, and are more similar to audio modality. What is the intuition behind this design choice? Also, no comparison has been provided.
[I] Huang, Po-Yao, et al. "Masked autoencoders that listen." Advances in Neural Information Processing Systems 35 (2022): 28708-28720.
[II] Ansari, Abdul Fatir, et al. "Chronos: Learning the language of time series." arXiv preprint arXiv:2403.07815 (2024).

3. Insufficient description of modeling strategy and motivation with lack of model description. The paper can benefit from adding a more comprehensive comparison with general time series models like Chronos [2].

4. The key take away “the larger the model the better the performance” “the larger the dataset the better the performance” are fairly standard statements in the AI/ML area at this point. The paper can be improved by bringing in more insights with respect to the model size, scalability in different conditions (e.g., resource constraint/edge computing setting for example, or a distributed setting). Also, large offline dataset during training is hard to come by and often comes with significant privacy concerns. So when it comes to scalability, the computational resource and privacy are the main bottleneck. I think a bit more discussion around the key takeaways and the model’s scalability contributions would be important. 

5. Only a single real world downstream task of activity recognition has been explored in the current version of the work. While I highly appreciate the large-scale data collection efforts (that enabled the work) and the large computational model, only a single downstream task fails to highlight its capability. The paper can benefit from incorporating other real world mobile health applications such as stress/fatigue modeling with similar wearable data or may be other physiological conditions modeling.

6. Activity recognition is a well studied area. One additional task that could be interesting is whether such a model generalizes across unseen classes (e.g., dancing, cooking, and other activities for daily living). This could further highlight the capabilities of the model.

7. One of the core strengths of the paper is its large dataset. Making it available to the broader community could help research and development in this area. I could not find any mention about open sourcing the dataset. Also, no mention about open sourcing the model either. I understand that often there are IRB restrictions but I think making the model and data available could be a game changer in this area.

8. Relatively minor comment but there are some repetitive statements. The paper can be shrunk a bit and can be made it a bit more compact.

---

confidence: 3
contribution: 4
initial_rating: 8
presentation: 3
questions: 1. How was decided the feature space?
2. How many individual training samples per person were used? 
3. Did you use user-stratified splits? Or what is the data split method?
4. Why are the baseline methods for interpolation and imputation only Mean, NN and Linear?
soundness: 3
strengths: 1. Several tasks have been defined for multimodal model evaluation: imputation, interpolation
2. A robust set of activities and extensive dataset for training 165k people.
3. Comparative analysis across model parameters and data size through an ablation study.
4. Analysis of multiple tasks: Classification, interpolation, reputation and extrapolation.
summary: This paper presents a foundational model training for daily activity recognition from wrist worn portable devices. It uses a multimodal approach including 26 features out of four modalities (Acc, PPG, skin Temp  and conductance).
The model presents an improved performance compared to other methods in classification tasks of activities after user-based self-labelled classes (mostly on sports activities), as well as in interpolation and imputation tasks. 
The paper analyzes the model efficiency, and complexity needed by comparing on number of subjects, and samples needed.
weaknesses: 1. 16 features extracted from PPG, Acceleration, Skin temperature and conductance, and altimetry - seems a very reduced space for learning. 
2. Temporal interpolation at the scale of 1 minute is not a very accurate task for wearable human data, as it only holds under strong assumptions of human behaviour. E.g. continuous activity, unchanged environment, no external inputs, among others.
3. It needed to be clarified the number of individuals used for training and the methods for data labelling. 
4. Baseline method comparison for imputation, interpolation and extrapolation was rather simple and did not included other advance methods, see 1.
[1] Maksims Kazijevs, Manar D. Samad. Deep imputation of missing values in time series health data: A review with benchmarking. (2023) Journal of Biomedical Informatics
5. The title is too broad and uninformative --> this model is trained for sports activity and proof given in that space only of the wrist wearable data.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: As mentioned above.
soundness: 3
strengths: - The data processing and model training steps are clearly outlined, and the study systematically scales the model across key factors such as data volume, computational resources, and model size.
- The paper includes a variety of tasks that effectively demonstrate the model’s utility across different contexts, enhancing the applicability of the LSM model in both generative and classification domains.
summary: This study scales a multimodal foundation model, LSM, across compute, data, and model size. Using a dataset of 40 million hours from 165,000 individuals, covering heart rate, heart rate variability, electrodermal activity, accelerometer, skin temperature, and altimeter data. The paper built a sensor foundation model and investigated strategies for training such models with considering data size, computation cost and model size.
The data processing steps are clearly explained and the the whole framework is tested on a variaty of tasks including generative and classification types of tasks.
The authors’ consideration of data processing and task diversity is a valuable contribution to the wearable and ubiquitous computing community, positioning this work as a significant step forward in multimodal sensor modelling. However, there are important areas for improvement, particularly regarding challenges specific to wearable computing and comparisons with state-of-the-art (SOTA) techniques.
weaknesses: **Areas for Improvement**

- Background and Related Work: The paper is thin in covering prior works in multimodal wearable models. The chosen model (ViT) and baselines focus on non-wearable modalities, overlooking well-established approaches tailored to multimodal sensor data such as (but not limited to):

> [1] Saeed, A., Ungureanu, V. and Gfeller, B., 2021. Sense and learn: Self-supervision for omnipresent sensors. Machine Learning with Applications, 6, p.100152.

> [2] Deldari, S., Spathis, D., Malekzadeh, M., Kawsar, F., Salim, F.D. and Mathur, A., 2024, March. Crossl: Cross-modal self-supervised learning for time-series through latent masking. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining.

> [3] Haresamudram, H., Essa, I. and Plötz, T., 2021. Contrastive predictive coding for human activity recognition. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 5(2), pp.1-26.


- Dataset Clarity and Statistical Insights: The paper does not clearly specify the dataset utilized. The statement, “In this paper, we present the results of our scaling experiments on the largest and the most diverse wearable dataset published to date…” implies that the dataset is previously published, yet it lacks explicit details. The authors should clarify the dataset's origin and provide comprehensive information regarding its attributes. Clarification on the dataset’s availability and publication status is necessary.
Detailed statistical insights, including class distributions and sampling strategies, would enhance understanding of the dataset’s structure. This information would be particularly useful for evaluating how well the model might generalize across various activities and sensor modalities.

- Data Sampling and Task Appropriateness: The paper lacks clarity regarding the time window chosen for training and inference (e.g., It is infered that 5-hour windows is used for both generative and classification tasks). This is a critical point, as activities typically span only a few minutes, making it unclear whether such extended windows are suitable for classification. Details on data sampling strategies and the selection of window sizes should be provided to better assess model performance across tasks.

- Specific Challenges in Ubiquitous Computing: As a foundation model for wearable sensing, LSM should ideally address unique challenges in ubiquitous computing. For example:
  - the authors could discuss handling missing data at various stages (pre-training, fine-tuning, and inference), as highlighted by [2]. This would enhance the model’s robustness for real-world applications.
  - To demonstrate the model’s generalizability, it would be beneficial to extend experiments to include additional datasets covering diverse tasks, devices, and activities to analyse adaptability to a wider range of scenarios in wearable sensing.

---

confidence: 5
contribution: 4
initial_rating: 8
presentation: 4
questions: There is a lot of work that directly deal with accelerometer data in scale and recently also PPG data. The claim that these streams are hard to interpret and therefore not in scope - doesn’t seem a valid explanation. I do understand practical constraints in collecting these streams at a high enough sampling rate and so should suffice to explain the use of intermediates such as steps or heart rate.

The graphs with the scaling laws with the marker size and color simultaneously varying and with different y-axis scales is confusing. In fact, some marker size differences are small enough to not even stand out while accounting for a significant scale increase. Request to use different marker types instead of just size and make it more obvious, and also maintain y axis scale to compare more effectively.
soundness: 4
strengths: An original research work focused on a multimodal model with true large scale of data and model size that is comparable to language foundation models. The scaling laws validate proof of principle for this area. Sample applications chosen have practical significance.

The article is demonstrating high level of original research due to the scaling law contributions. They also gain additional validation by doing so with multi modal sensor streams. There have been other notable work showing the power of pre training on single or multiple sensor streams in the past at sufficient scale but nothing addressing the scaling question.

The results and analysis were presented with sufficient clarity. Some room for improvement has been suggested in the following section.
summary: The authors have introduced a multimodal foundation model for sensor data. The input is mix of raw sensor samples and some preprocessed discrete derived samples due to some understandable practical constraints. They have set up a frame work to prove generative and inferential capabilities allowing the model to be regarded as foundational. With a good representative choice of model architecture, scales of data and a diverse set of scaling experiments the authors have derived a first proof of power scaling laws showing validation of feasibility for the concept of sensor foundation models.
weaknesses: While the work is generally comprehensive, some areas of improvement remain. 

The limited choice of downstream tasks raises questions on the true foundational nature of the model and its few shot capability. It seems important to understand the ability to generalize to tasks beyond classification eg. regression tasks or system identification tasks.

The choice of generative evaluation goals while interesting fails to address the practical utility of the performance shown, especially with large errors in imputation. Also, there exist sophisticated imputation and interpolation schemes that are more performant comparators eg. MICE which might be the real performance benchmarks for the application. 

The definition of some of the features and preprocessing is incomplete and not in the language of mathematics. Some descriptions are highly redundant eg “Kurtosis is kurtosis”. Recommend use of the language of accurate mathematical description to define the features clearly.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. In section 3.1 (data preprocessing) provide an explicit statement of which physiological metrics ("features") are calculated on-device vs, which are calculated on the stored centralized data.  This section mentions broadly that the raw sensor signals are not stored centrally or on-device (which seems to imply that all of the features are calculated on-device), but whether this is the case should be stated clearly. 
2. Toward the end of section 3.1 (data preprocessing) the authors state “Within each 300-minute window, missing data between valid data points was linearly interpolated, and leading missing minutes were backfilled.”  Was the **test set** that was used for evaluating generative tasks free of these missing values that had been infilled by linear interpolation/backfilling?  If so, these time points should be excluded from the calculation of generative task performance (MAE/MSE) because they do not represent real measurements (but rather synthetic values created in data preparation using a human-chosen method).  Please comment on how this issue was handled.  
3. Typo(?) on line 202.  “Kurtosis is the kurtosis of the BFP signal”.  Should (I think) be BPF.
4. Why was the data input length of 5h chosen?  What other lengths were considered, and how were these compared to ultimately land on the choice of 5h.
soundness: 3
strengths: ## This paper represents strong contributions in the following areas: 

### Originality:  This paper is the first to evaluate self-supervised learning methods on a multi-modal physiological data set at this scale.  The authors describe a novel modeling approach as well as a comprehensive description of their ablation experiments. 

### Quality:  The authors present rigorous, systematic and thorough evaluation of performance on the generative tasks as a function of pre-training data volume, and model size. 

### Significance: The analysis of generative tasks is comprehensive and may be of interest both within the problem domain (models for multi-modal data from wearable devices) as well as more broadly.  However, the lack of compelling evidence for model generality across a range of downstream tasks may limit the broad interest. 

### Clarity:  The paper is fairly clear regarding the data source (including featurization), modeling approach, performance analysis and related ablation experiments.   Some details (discussed in the section below) are missing, but this is not likely to impact the general conclusions.
summary: This paper describes self-supervision training methodology (infilling of randomly-masked patches) applied to a very large data set of multi-modal time series data collected by wearable devices under natural usage.   The authors describe a comprehensive analysis of the model performance as a function of training data volume, compute resources and model complexity across a set of generative tasks (imputation, extrapolation/forecasting and interpolation).  Additionally they also evaluate performance of the resulting pre-trained encoder (and compare this with a supervised learning baseline) on a pair of discrimination tasks: binary exercise/non-exercise classification and multi-class activity classification.
weaknesses: # Brief Summary #
This paper had some important weaknesses that are listed briefly below, with more detailed comments and some suggested steps to address these weaknesses following the list:

1. [Most concerning] Lack of variety in downstream task evaluation.  The paper evaluates only two closely-related downstream tasks (binary and multi-class activity classification) that utilize the learned representations.   Given that one of the main characteristics of ‘Foundation Models’ is that they are readily adaptable to a wide variety of downstream tasks, it is a stretch to consider the LSM model to be a foundation model rather than a tool for time series interpolation/extrapolation. 
2. No analysis of model performance in a manner that is agnostic to the downstream task (e.g. using measures of unsupervised embedding quality)
3. Incomplete comparison of model performance on downstream tasks against baselines— LSM performance was compared with SL-trained model having the same ViT architecture, but not against other classifier types.
4. Absence of some details and explanation on the problem setup for downstream classification tasks.
5. The authors are missing discussion/citation of some past publications that represent closely-related research (learning general-purpose representations from multi-modal physiological time series captured by wearable devices) 


# Additional Discussion Details #

## Issue 1: [Most concerning] Lack of variety in downstream task evaluation.
Other than an extensive analysis of the interpolation/imputation/extrapolation tasks (which are useful proxies), the paper evaluates performance on only two closely-related downstream tasks utilizing the LSM embeddings: binary exercise/non-exercise classification, and multi-class activity classification.  This makes it hard for the reader to have confidence that LSM can truly be considered a “foundation model”.  Per the proposed terminology set forth in Bommasani, et al., ‘On the Opportunities and Risks of Foundation Models’, in addition to being trained via self-supervision on diverse large-scale data, a FM “can be adapted to a wide range of downstream tasks”.    Moreover, there would be particular value and impact in demonstrating that the LSM embeddings are useful for ***physiologically-relevant*** downstream tasks.  

### Suggestions to address Issue 1:
The data source described by the authors contains some additional metadata/labels that could be used as targets for downstream tasks, building evidence for the general-purpose utility of the LSM embeddings.   The metadata includes subject-reported age, sex, and weight (possibly BMI, which is listed in table 2a) that can be used as regression and/or classification targets.  

Other targets, particularly those with some direct physiological relevance, would also provide additional evidence for general-purpose utility of the model.  I recall that Fitbit generates a ‘Cardio Fitness Score' that represents an estimate of VO2max.  If VO2max is available it could be utilized as either a regression target or a classification target (simple binning into high/low or tertiles/quartiles).  This would be a good choice of task, since there is a first-principles argument that the temporal relationship between physical effort and heart rate response *should* contain information about VO2max.  

## Issue 2: No analysis of model performance in a manner that is agnostic to the downstream task
In a complement to suggestions for issue 1 above (more downstream task variety) providing some task-agnostic analysis of embedding ‘quality’ would also help the argument that LSM is in fact a general-purpose foundation model. The authors provide some t-SNE visualization of embedding distribution in Figures 8 and 9, but no accompanying quantitative analysis.  

### Suggestions to address Issue 2:
Include some quantitative measure of how well-distributed the model embeddings are, and/or how well 5h data segments generated by the same individual can be distinguished from segments generated by other individuals.  

Multiple measures of embedding “quality” have been reported that are useful for evaluating and comparing models in a manner that is agnostic to the downstream task (for a recent review/comparison, see Tsitsulin, *et al.* ‘Unsupervised Embedding Quality Evaluation’ https://arxiv.org/abs/2305.16562).  

## Issue 3: Incomplete comparison of model performance on downstream tasks against baseline.
The authors compare the performance of their pre-trained model against a supervised learning baseline for the two downstream classification tasks.   However, this baseline (ViT with 100M parameters) may not be optimal, particularly for the relatively simple task of exercise vs. non-exercise binary classification.   It’s worth noting that based on the test set summary in Table 2b simply guessing the majority class (non-exercise) every time would produce ~65% accuracy, but the 100M-parameter ViT only achieves 70.9% with supervised learning (Table 3b).  

### Suggestions to address Issue 3:
Compare performance on both downstream classification tasks using an alternate modeling approach such as random forest or regularized logistic regression (or other boilerplate ML classifiers).  Given the problem statement (classify a period of time as exercise vs. non-exercise, or by exercise type, using a time series of 27 features), it is possible that RF or LR could outperform the SL ViT.  Even if they don’t, it could give an indication of the performance gap between these ‘non-deep’ supervised ML methods and the ViT SL baseline.  
 
## Issue 4: Absence of some details and explanation on the problem setup for downstream classification tasks.
Section 4.2 on discriminative tasks is very light on details, to the extent that it would be difficult or impossible for other researchers to reproduce the findings (using their own data and models).  The data segments are 5 hours long (so presumably only a subset of that time contains the activity/exercise of interest) but it is not clear how the activity period within the segment is labeled or how the classification is evaluated. Tables 3b and 3c report mAP as a performance metric, which makes me think that there is some time segmentation or drawing of boundary-boxes involved in the downstream task, but this is not explained in the text.  In contrast, Appendix section B.3 states “This is likely as there are significant periods of walking in the 5-hour inputs, even if the activity is labeled otherwise”, which implies that the entire 5h inputs are assigned a single class label.  This is confusing (or at least unclear) to the reader.  

### Suggestions to address Issue 4:
Provide more detail on the downstream classification problem setup, accompanied by a diagram or illustration (at least in the Appendix) that illustrates visually how the model outputs are evaluated against the ground truth labels. 

## Issue 5: The authors are missing discussion/citation of some past publications that represent closely-related research.  

Some relevant published work that is fairly similar in scope (learning general-purpose representations from multi-modal physiological time series captured by wearable devices) exists, but these are not cited or discussed.  One clear example is  D. Spathis *et al.*, Self-supervised transfer learning of physiological representations from free-living wearable data (2020, https://arxiv.org/abs/2011.12121) which utilized pretext tasks to pre-train models for multi-modal inputs from wearables (heart rate and raw IMU), then evaluated the resulting embeddings on a variety of downstream tasks.  

### Suggestions to address Issue 5:
At least include the Spathis *et al.* paper (listed above) among the citations.  Given the close match in scope, this paper should probably also be included in Table 1 despite not referring to its work as a “foundation model” (being from 2020, it predates the popularization of the “foundation model”).    It may also be worth doing an additional literature review specifically looking for pre-2021 papers that do not use the term “foundation model” but are clearly pursuing the same objective.

Meta-review: This paper proposes a pretrained foundation model based on a large-scale wearable sensor data from 165,000 individuals, collected over than 40,000 hours, consisting of multiple sensor modalities, i.e. accelerometer, PPG, EDA, skin temperature, and altimeter sensor. This is the largest pretrained foundation model on sensor data to date. As such, the authors provided new insights into the scaling laws and training strategies applicable to wearable sensor data, including sample scaling and subject scaling.
Initially, there were a shared concern by the reviewers that the generality of the foundation model is an overclaim, given the limited downstream tasks, only on imputation, prediction, and discriminative tasks. There are also concerns that the data is not openly shared. 

During the rebuttal phase, the authors have also added new experiments on new downstream tasks, including mood classification, age classification, and sex classification -- increasing the diversity of the tasks, strengthening the claim of generality. The authors have also added more details on the method and experiments, and have better contextualised the work among other similar works in the recent years, and added discussions on limitations. 

The insights into model saturation is also very interesting. 

I believe this paper should be accepted, given the extensive and comprehensive effort, and the strong rebuttal. The contributions of this paper will guide other researchers in the same field, especially those who may not have the same access to the large scale dataset or compute. This paper will also lead to interesting discussion on foundation models training beyond text.

Decision: Accept (Poster)