Beyond Sensor Data: Foundation Models of Behavioral Data from Wearables Improve Health Predictions
Authors: Eray Erturk
Fahad Kamran
Salar Abbaspourazad
Sean Jewell
Harsh Sharma
Yujie Li
Sinead Williamson
Nicholas J Foti
Joseph Futoma
Venue: ICML 2025 poster (2025)
Keywords: Wearables
Health applications
Foundation models
Primary area: applications->health_medicine
TL;DR: This work develops foundation models of behavioral signals from 2.5B hours of wearable data, achieving strong performance on 57 health tasks, excelling in behavior-driven predictions and further improves when combined with sensor data.

Abstract: Wearable devices record physiological and behavioral signals that can improve health predictions. While foundation models are increasingly used for such predictions, they have been primarily applied to low-level sensor data, despite behavioral data often being more informative due to their alignment with physiologically relevant timescales and quantities. We develop foundation models of such behavioral signals using over 2.5B hours of wearable data from 162K individuals, systematically optimizing architectures and tokenization strategies for this unique dataset. Evaluated on 57 health-related tasks, our model shows strong performance across diverse real-world applications including individual-level classification and time-varying health state prediction. The model excels in behavior-driven tasks like sleep prediction, and improves further when combined with representations of raw sensor data. These results underscore the importance of tailoring foundation model design to wearables and demonstrate the potential to enable new health applications.

Beyond Sensor Data: Foundation Models of Behavioral Data from Wearables Improve Health Predictions
Eray Erturk" !
Fahad Kamran 22 Salar Abbaspourazad Sean Jewell Harsh Sharma Yujie Li
Sinead Williamson
Nicholas J Foti #2 Joseph Futoma f2
about person's 's current health state. An important aspect Wearable devices record physiological and behav- ioral signals that can improve health predictions. While foundation models are increasingly used for such predictions, they have been primarily applied to low-level sensor data, despite behavioral data often being more informative due to their align- ment with physiologically relevant timescales and quantities. We develop foundation models of such behavioral signals using over 2.5B hours of wear- able data from 162K individuals, systematically optimizing architectures and tokenization strate- gies for this unique dataset. Evaluated 57 health-related tasks, our model shows strong per- formance across diverse real-world applications including individual-level classification and time- varying health state prediction. The model excels in behavior-driven tasks like sleep prediction, and improves further when combined with representa- tions of raw sensor data. These results underscore the importance of tailoring foundation model de- sign to wearables and demonstrate the potential to enable new health applications.
Abstract
of health monitoring is detecting a static health state - for instance, whether somcone has a history of smoking, has past diagnosis of hypertension, or is on a beta-blocker. Another crucial problem is detecting a transient health state, such as the quality of someone's s sleep or whether someone is currently pregnant. A key property of the data required for these predictions is that they are typically at the temporal resolution of human behavior (e.g., days and weeks) rather than at the lower-level time scales (e.g., seconds) at which raw sensor data is collected from wearables.
Though a majority of past work has considered model- ing low-level sensor data (or simple features thereof) (Ab- baspourazad et al., 2024b; Narayanswamy et al., 2024; Ab- baspourazad et al., 2024a; Yuan et al., 2024), higher-level behavioral information from wearables such as physical ac- tivity, cardiovascular fitness, and mobility metrics, are the natural data type to help solve these detection tasks. Unlike raw sensors, these higher-level behavioral metrics are cal- culated using carefully validated algorithms derived from the sensors. These metrics intentionally chosen by experts to align with physiologically relevant quantities and health states. Importantly, these data are sensitive to an individual's behaviors, rather than being driven purely by physiology. These characteristics make behavioral data particularly promising for such health detection tasks. For example, mobility metrics that characterize walking gait and overall activity levels may be important behavioral factors to help detect changing health state such as pregnancy.
1. Introduction
Consumer wearables, such smartwatches and fitness trackers, provide rich information across diverse health do- mains, including cardiovascular health (Moshawrab et al., 2023), sleep (Bianchi, 2018), diabetes (Ahmed et al., 2022), mental health (Gomes et al., 2023), respiratory health (Channa et al., 2021). reproductive health (Lyzwinski al., 2024), and more. A common type of prediction problem in wearables is detection, where set of physiological, be- havioral, and/or sensor data are used to predict information
However, behavioral data present unique challenges, includ- ing irregular sampling, missing data, and variable sampling rates across features, requiring specialized approaches for modeling. Moreover, it is common to have massive quanti- ties of unlabeled, real-world wearable data, but only have higher-quality labels on small curated subset. This moti- vates the need for approaches to learn useful representations of these behavioral wearable data in the absence of labels.
Equal contribution +Equal senior authors 'USC (Work done during an internship Apple.) AApple Inc. Correspondence to: Joseph Futoma aifutoma@applc.comm
As such, these data naturally align with foundation models (Bommasani al., 2021), where pre-training on large, di- verse datasets enables transfer learning new tasks with limited labels. Foundation models can capture complex,
Proceedings of the nnnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Beyond Sensor Data: Foundation Models of Behavioral Data from Wcarables Improve Health Predictions
(4) Large-scale behavioral health quantity data from wearables
(b) Wearable health behaviorel foundation model training
(c) Strong performance on 57 real-world health tasks and complementary information to existing sensor foundation model
Systematic archilecture and tokenizer turing
what Alllmsaaiin
B& rhnishonuaistli
K I -ll of Vill
WBM
Large observational longitudinal digital health study IAHMS, 161,855 particicants, 15 billion hourly measurements)
Figure 1. An overview of our approach to solving a wide variety of health detection tasks. We use the Apple Heart and Movement Study to develop foundation model for irregularly sampled behavioral time series data. We systematically optimize tokenizers and architectures for our dataset to build our final model, and we find that it excels in behavior-driven tasks and provides complementary information to an existing PPG foundation model.
nonlinear interactions in wearable data, reducing the need for task-specific models and addressing the challenge of small, often weakly labeled datasets, common limitation in wearable health research. Despite the growing interest in foundation models for wearable data, prior work has primarily focused on modeling low-level biosignals, such as photoplethysmogrrm (PPG), electrocardiogram (ECG), and accelerometer data (Abbaspourazad et al., 2024a; Yuan et al., 2024; Abbaspourazad et al., 2024b), or simple ex- tracted features from them (Narayanswamy et al., 2024). These signals, while valuable, are not consistently available throughout the day, limiting their applicability across di- verse health detection problems. In contrast, behavioral data provide complementary insights by capturing broader pat- terns of health and activity. Integrating behavioral data with raw sensor signals offers a holistic view of an individual's 's health, enabling models generalize across a wider range of detection problems.
Our key contributions include: 1) Strong performance of behavioral data for health detection: We demonstrate WBM's generalizability and real-world utility across di- verse health detection tasks. 2) Integrating behavioral and sensor data: We show that incorporating WBM as a model of behavioral data improves upon an existing PPG foundation model across most tasks, highlighting their com- plementary strengths. 3) Developing a foundation model for wearables behavioral data: We outline the design and development of the first large-scale foundation model for irregularly sampled behavioral time series data.
2. Related Work
2.1. Foundation Models for Wearable Sensors
Recent work has demonstrated the success of large-scale pre- training of foundation models via self-supervised learning (SSL) low-level wearable sensor data, such PPG, ECG, and accelerometer signals, to create general- purppos,, accu- rate models (Abbaspourazad et al., 2024a; Narayanswamy et al., 2024; Yuan et al., 2024; Abbaspourazad et al., 2024b; Xu et al., 2024). These models are typically trained on raw sensor streams or extracted signal features from low-level sensors, leveraging SSL objectives to capture rich represen- tations of biosignals. Our work complementary, focusing instead on modeling higher-level behavioral data derived from wearables, such activity and mobility metrics, which are more computationally efficient and on more physiolog- ically relevant timescales for many health detection tasks. Merrill & Althoff (2023) is the only prior work we are aware of that has explored SSL methods on behavioral data, but it was limited to only three variables (heart rate, step count, and a sleep/wake flag) on small dataset of 5.2K individu- als. In contrast, signals derived from wearables, using a large-scale dataset with tens of millions of participant-weck enabling more robust and generalizable foundation models.
In this work, to realize the goal of accurate health detection we develop a wearable health behavior foundation model (WBM). WBM is trained on behavioral data from wearables, using 162K participants with over 15 billion hourly measure- ments from the Apple Heart and Movement Study (MacRae, 2021; Shapiro et al., 2023). Our model development process was guided by an understanding of the unique challenges posed by behavioral data, such as irregular sampling and missingness. We conducted a principled exploration of existing state-of-the-art approaches from past literature in wearables and time series modeling to understand and test which existing solutions would translate best to our unique data, resulting in a single performant model. We systemat- ically evaluate WBM on 57 health- detection tasks, including both existing and novel tasks that span variety of medical domains. We demonstrate our foundation model's 'ss ability to encode important health information, outperform- ing a strong baseline and complementing an existing PPG foundation model. See Figure for high-level summary of our work.
model a more diverse set of behavioral
2

---

Beyond Sensor Data: Foundation Models of Behavioral Data from Wcarables Improve Health Predictions
2.2. Foundation Models for Time Series
methods. These metrics encode both physiological and behavioral information. Compared to modeling raw sen- sor data, these derived metrics are chosen by experts due to their alignment with meaningful physiological health states, making these metrics particularly suited for down- stream modeling tasks. For example, VO2Max, a measure of cardiovascular fitness estimated from outdoor workouts, reflects both behavioral patterss (e.g. exercise intensity) and cardiovascular health. Other metrics, such as step count, primarily encode behavioral activity.
Behavioral data from wearables can be represented as a mul- tivariate time series, but poses unique challenges compared to traditional time series, including higher rates of missing data, irregular sampling, and varying sampling frequencies across variables. Foundation models for time series are an active area of research, particularly for forecasting and anomaly detection tasks in domains such as finance, en- ergy, and climate (Woo et al., 2024; Garza& & Mergenthaler- Canseco, 2023; Ansari et al., 2024; Rasul al., 2023). These approaches generally use a decoder-only approach. However, as we are not interested in forecasting or predict- ing the wearable data itself, we focus on an encoder-only framework for learning meaningful representations.
Below, we group the metrics into categories based on the types of information they capture. Throughout this paper, we refer to these as behavioral data or behavioral health quantities. However, for some variables, such as cardiovas- cular and vital signs, the measurements reflect a combina- tion of underlying physiology and behavior. For example, exertion (a behavior) increases heart rate, but heart rate itself is also direct physiological signal. A complete description of each variable and its availability is provided in Appendix Table 3.
Despite the progress in time series foundation models, few methods directly address the challenges posed by wearable behavioral data. Irregular time series modeling approaches have often been explored in the context of electronic health records and clinical time series (Chowdhury et al.. 2023: Shukla & Marlin, 2021; Tipirneni & Reddy, 2022: Zhang et al., 2023; Labach et al., 2023), which do not generally capture behavioral information despite having other com- monalities in terms of missingness and irregularity. We con- sider approaches from this past work, conducting principled ablations to find the best model for our data by combining these ideas with other tfate-of-the-arr sequence architec- tures, such as the Mamba-2 architecture (Gu & Dao, 2023; Dao & Gu, 2024) which have shown promise for healthcare applications (Fallahpour et al., 2024; Wornow et al., 2024).
Activity (8 variables): active energy (estimated calories burned), basal energy, step count (phone and watch), ex- ercise time, standing time, and flights climbed (phone and watch).
Cardiovascular (4 variables): resting heart rate, walking heart rate average, heart rate, and heart rate variability.
Vitals variables): respiratory rate (overnight only), blood oxygen, and wrist temperature (overnight).
walking metrics (speed, step length, double support percentage, asymmetry percentage, and steadiness score), stair ascent/descent speed, and fall count.
Gait / mobility (8 variables):
3. Data
We use a variety of wearable data collected under informed consent from participants in the Apple Heart and Move- ment Study (AHMS) (MacRae, 2021; Shapiro et al., 2023; Truslow et al., 2024), an ongoing real-world observational cohort study exploring the relationship between cardiovascu- lar measures, activity, and outcomes. Sponsored by Apple Inc in partnership with the American Heart Association and Brigham and Women's is Hospital, AHMS includes over 278,000 participants with up to 5 years of longitudinal data. Participants, who are U.S. residents aged 18+, provide in- formed consent using the Apple Research app (Truslow et al., 2024) and control the types of data they share. The dataset comprises wearable-derived metrics from the Apple Watch and iPhone, and health outcomes from surveys and medical records. To our knowledge, this is the largest and most diverse wearable data utilized to date (Bycroft et al., 2018; Denny et al., 2019; Merrill et al., 2023).
Body Measurements (2 variables): body mass and BMI.
Cardiovascular Fitness / Functional Capacity (2 vari- ables): VO2max and six minute walk distance, both clini- cally validated measures of fitness and capacity.
While AHMS is one of the largest wearable research datasets collected to date, the study cohort is not fully representa- tive of the broader U.S. population. Participants are iPhone and Apple Watch users who voluntarily opted into a digital research study, which may introduce selection bias related to socioeconomic status, technology access, and health en- gagement. Additionally, despite the large cohort size, cer- tain demographic groups, such as women, older adults, and racial and ethnic minorities remain underrepresented. The demographics of the cohort used to train our foundation model can be found in Appendix Table 6. These considera- tions are important when interpreting results and assessing generalizability. Nonetheless. the scale and richness of the data make it uniquely well-suited for developing foundation
Our work focuses on 27 interpretable Kia quantities' that are calculated from lower-level sensors using validated
httpp::/deeellpp...pp..... www. www.:///wwwoeeetttttooeoeeaaaaaa
3

---

Beyond Sensor Data: Foundation Models of Behavioral Data from Wcarables Improve Health Predictions
Hour
Hour
Hourd
Hour 167
Hour 168
4.2. Model Training
The irregularity of our data significantly differs from typical biosignals, necessitating comprehensive investigation into what techniques work best. Morcover, as this type of behav- ioral data has not been previously used to build foundation models, there is no direct prior knowledge to inform either the best methods for tokenizing variables as inputs nor the best backbone architectures. As such, we perform a princi- pled empirical exploration of different input tokenizers and model backbones most appropriate for our data.
A. e
Hour 2
Hour 167
Mamba-2
Hour te8.
Figure 2. The final WBM embedding pipeline for an individual's s behavioral data. The input data is irregularly sampled, both within a specific variable and across all 27 variables. We transform this into dense matrix of weekly data, where cach row is an hour of data. We also include a missingness mask (not shown) for each variable across the week, resulting
Input Tokenizers Understanding how to correctly represent the data is critical for handling irregularly sampled data. A tokenizer maps an individual's 's week of data to a sequence of vectors that can be used as input to a deep learning model. We consider three tokenizers that can be applied to our input data. We provide a brief overview in this section, and leave specific implementation details to Appendix A.3.
168 x 54 matrix. We project each hour as a patch using single linear layer, and use the resulting patches as input to Mamba-2. Finally, we average the Mamba-2 outputs across time to create single embedding for individual's week of data.
TST. First, we consider dense approach for modeling a week of an individual's data inspired by the Time Series Transformer (TST) approach (Zerveas et al., 2021). We create a 168 27 matrix for each hour in the week and 27 variables. For each of the 27 variables, we crudely impute missing hours with a global average value (i.c., zero z-scored inputs). Though we tried more sophis- ticated versions of imputation (e.g., subject-level averages or weekly averages), the simpler global average resulted in the best performance in initial experiments. We also con- catenate missingness indicators for each variable, resulting in a 168 x 54 feature matrix. We treat each hour of data as patch, and use multi-layer perceptron with one hidden layer map each patch an embedding vector (Zerveas et al., 2021). This sequence of vectors can then be used as input to backbone deep learning model.
models of human physiology and behavior from wearable sensors, as we describe in the next section.
4. Developing a Foundation Model
To improve predictions on a wide variety of health detection tasks, we developed a foundation model capable of encoding an individual's s health and behavioral data over a time win- dow into a single embedding. In Figure 2, we showcase the model architecture and tokenizer that found performed best. In the remainder of this section, we discuss the motiva- tion and empirical analysis that led us to particular dataset and modeling choices.
as
4.1. Pre-training Dataset
We created the pre-training dataset by aggregating behay- ioral health data at hourly intervals, aligning with our goal of detecting health states over longer time scales. Hourly aggregation promotes consistency across variables with un- even sampling rates, and is also common design decision in prior work (Lipton et al., 2016; Xu et al., 2018; Wang et al., 2020). Driven by our goal of detecting health states at temporal resolution of human behavior, weekly windows of aggregated hourly data were chosen as model inputs, bal- ancing prediction granularity, actionability, and scalability.
mTAAN Next, we consider using multi-time attention (mTAN) with masking to appropriately handle missing data. We use the same 168 x 54 feature matrix from TST as input to a mTTA network, akin past work (Chowdhury et al., 2023; Shukla & Marlin, 2021). mTAN returns a sequence of embedding vectors that can then be used as input to backbone deep learning model.
Tuple. Finally, we consider a non-dense approach that treats each observation of data as a tuple containing time (i.e., hour in week measurement was taken), variable type, and numerical value. We adapt techniques from past work which learn one-to me-to-many mappings from scalar-valued time- and variable-values to higher dimensional embedding, and use an embedding lookup table to map each categorical variable type to higher dimensions (Tipirneni & Reddy, 2022: Zhang et al., 2023; Labach et al., 2023). Adding these three embeddings results in a single input token for each hourly variable measurement. Missing values are naturally
The final dataset consists of 15.14 14 million weeks of data (~2.5B hours) from 161.855 unique participants, filtered for adequate watch wear (see Appendix 1.1.1 for details). Data irregularity stems from variable sampling rates, missing data, and participant-specifie inconsistencies. We report detailed statistics on variable availability in Appendix 1. and Table 3. We use the same 80% / 10% / 10% participant- level splits for training, validation, and testing for both pre-training and downstream tasks.
4

---

Beyond Sensor Data: Foundation Models of Behavioral Data from Wcarables Improve Health Predictions
handled, as variables that were not measured given hour simply omitted. The sequence of all tokens can then be used as input to a backbone deep learning model.
portions of the input signal, which may both overemphasize observed variables and be unnecessarily stringent. Though such a task is useful for imputation and interpolation, these tasks are not a focus of this work. In Appendix A.5.3, we show the poor performance of a masked autoencoder in ini- tial experiments, further motivating the use of contrastive approaches in downstream health detection tasks. Though other SSL objectives exist and may be suitable. we leave an exploration of these non-contrastive approaches to future work.
Backbone Model Architectures. Given input tokens of the irregularly sampled data, we describe a few popular sequence-to-sequence architectures to learn meaningful rep- resentations. All models take in a sequence of embedding tokens, and return an output sequence of the same length. To obtain a single output embedding, we average across time over all output embeddings of the model. We provide a brief overview, but leave details to Appendix A.4.
We use subject-level positive sampling contrastive objec- tive (see Appendix A.4.1 for further details), which has been shown to learn informative representations for many of the same health detection tasks we are interested in (Ab- baspourazad et al., 2024a). To encourage learning meaning- ful representations, we sample pairs of augmented segments (i.e., weeks of data) from the same individual. To create the augmented segments, we use input token dropping. Specifi- cally, we consider dropping p% of tokens across time prior to learning representations during training.
Self-Artenn Self-Artention Transformer. Given the irregularity of our input data and the strong results of Transformers across dif- ferent modalities, including wearable data, we first consider using& Transformer to learn representations of the data (Vaswani et al., 2017). Given the importance of represent- ing temporal aspects of the signals, we pay special attention to positional encodings. First, we consider standard self- attention techniques with learnable position encodings.
Rotary Transformer. Absolute positions may matter less than relative positions when working with temporal data such as wearable signals. Hence, we also explore the util- ity of using relative position encodings. In particular, we consider Rotary Position Embeddings (RoPE), due to their strong performance compared to other variants, and strong theoretical properties that show their ability to flexibly en- code both absolute and relative encodings (Shaw et al., 2018; Ren et al., 2021; Dufter et al., 2022; Su et al., 2024).
4.3. Tuning and Best Model Creation
To create a final performant foundation model, we train ev- ery combination of tokenization (Tuple, mTAN, and TST) and architecture (Self-Attention Transformer, Rotary Trans- former, Mamba-2) using the regularized contrastive loss. We perform grid search over all 9 combinations, tuning a subset of parameters for each model, including batch size, KoLeo regularization amount (Sablayrolles et al., 2018; Chen et al., 2020), weight decay, and number of layers (see Appendix 1 for details). To choose between these differ- ent models, we train and evaluate the learned embeddings the representative downstream task of age prediction. As age is predictive of many health conditions, the task of age prediction is often used to measure the quality of learned embeddings (Abbaspourazad et al., 2024a; Narayanswamy et al., 2024). Overall, we find that most models perform well on the down- stream age task on the held-out validation set (Appendix Table 9), with the combination of the simple TST tokeniza- tion and Mamba-2 backbone achieving the lowest error. We hypothesize that TST tokenization outperformed more so- phisticated tokenization methods due to the high level of noise in the data and significant aggregation done in prepro- cessing, such that the naive imputation of missing variables using population global mean was sufficient. Despite the prevalence of Transformers for building foundation mod- els for wearable data, we found that a Mamba-2 backbone resulted in the best model. Importantly, all models were given the same hyperparameter budget, and the best model was not always the largest model. Moreover, we capped the number of layers for Mamba-2 in this initial exploration to the largest Transformer that could fit in memory due to
Ma Continuous time state-space models are natural choices for handling irregularly sampled time series data due to their learned discretization step sizes. Specifically, we consider the recent strand of selective state-space mod- els known as Mamba, as they are efficient, more flexible than classical state-space models, and have been shown to be competitive with Transformers for language modeling (Gu & Dao, 2023; Dao & Gu, 2024). We consider using Mamba-2 as backbone architecture to learn downstream representations. We apply a bi-directional Mamba-2 archi- tecture as in past work using state-space models for time series, allowing the representations to summarize informa- tion in both the forward and backward directions (Wang et al., 2024; Liang et al., 2024).
Pre-Training Loss. We use regularized contrastive ob- jective as our SSL pre-training loss. Similar contrastive ob- jectives have achieved strong performance in health-related time-series signals (Jeong et al., 2023), and in learning performative foundation models from wearable signals (Ab- baspourazad et al., 2024a). Due to the importance of captur- ing sparse but informative variables (e.g., VO2max), we do not consider masked autoencoder pre-training framework. These techniques require the model to be able to recreate all
5

---

Beyond Sensor Data: Foundation Models of Behavioral Data from Wcarables Improve Health Predictions
the attention calculation. Despite this, we found the TST and Mamba-2 combination consistently resulted in the best model across other downstream tasks as well (see Appendix Table 10).
involving time-varying health states. Our hypothesis is that WBM and PPG capture complementary aspects of health, and that combining them will yield the best predictive per- formance across most tasks.
Given this initial finding, we perform a larger hyperparame- ter search using a Mamba-2 backbone with TST tokeniza- tion to create the final foundation model, which we denote WBM. Figure 2 shows the full process of going from an individual's 's imput data an output embedding that can be used for downstream tasks. Though we are unable to release model weights and code due to the specifics of the informed consent for participants in the study, Section 1.5.1 contains all training details and final model configuration. For more details, interested parties can reach out to the authors.
We use linear probing (with a ridge penalty) to fit downstream models using WBM's embed- dings as predictors for each downstream task. Models are fit either on per-week or per-participant basis, as appropriate. We use the same participant- splits as in pre-training (00%/10%/109% combining validation and test sets to yield 80/20% train/test splits, and apply internal cross validation to select a good penalty parameter for each linear model. We also compute 95% bootstrap confidence intervals, and use these to compare different models.
Experimental Set-Up.
5. Downstream Evaluation of Learned Foundation Model
Baseline Models. As a simple baseline, we use the mean and standard deviation of each of the 27 health behavior variables as way to summarize each week of data, rather than using a learned model embedding. To fit participant- level models, similar to averaging week-level embeddings, we average these week-level vectors of statistics across all weeks of available data for a participant. For all downstream tasks except age and sex prediction, we also include relevant demographic information (i.e., age, sex, and BMI) inputs. We denote this combination of statistics and demographics as "Baseline". Comparing against such simple baseline fea- tures shows the utility of foundation models over a typical simple supervised approach for our data. We also com- pare against a PPG foundation model (which we denote as "PPG") from prior work (Abbaspourazad et al., 2024a), as they showed extremely strong performance on many down- stream health detection tasks. Comparing against PPG model allows us to examine situations where modeling be- havioral data may be of more benefit than modeling lower- level sensor data. Finally, we create what we expected to be the strongest model by combining embeddings from the PPG model with WBM, to show the complementary na- ture of modeling behavioral data and lower-level sensor data. Specifically, we combine WBM and PPG embeddings by concatenating the two resulting embedding vectors into one single embedding vector that can be used as input for downstream models.
Table 1. Performance of embeddings on age and sex prediction with 95% bootstrap confidence intervals in parenthesis. The com- bination of WBM and PPG consistently performs best.
Embedding
Age (MAF)
Biological Sex (AUROC)
Baseline
7.89 (6.66, 10.31)
0.931 (0.928, 0.934)
WBM
3.67 (3.63, 3.71)
0.999 (0.999, 0.999)
PPG 2.89 (2.86, 2.93)
0.997 (0.996, 0.997)
WBM + PPG 2.46 (2.43, 2.50)
0.999 (0.999, 1.000)
We now aim to understand whether the learned WBM em- beddings encode meaningful information for a wide variety of health detection tasks.
5.1. Downstream Tasks
Motivation. The central research question driving our work is whether WBM encodes enough information to enable strong predictive performance across a wide range of health detection tasks. Prior work in the wearables space has largely focused on activity recognition and exercise detec- tion tasks from sensor data (Narayanswamy et al., 2024; Yuan et al., 2024) or on single disease states or conditions (Merrill & Althoff, 2023), and do not comprehensively eval- uate on a wide variety of different types of health conditions. Given the unique nature of our large-scale observational study, we have access to much wider variety of potential health, wellness, and medication outcomes via comprehen- sive longitudinal health surveys and medical records. This allows us better assess the generalizability of our models to detect larger variety of meaningful health states that might provide individuals actionable insights. We build on prior work that demonstrated promising performance of foundation models trained on low-level physiological sig- nals like PPG and ECG for static health classification tasks (Abbaspourazad et al., 2024a). In this study, we not only adopt those same benchmarks but also introduce novel tasks
Task Details. To comprehensively evaluate our foundation model's ability to enable diverse health detection applica- tions, selected downstream tasks that fall into two broad categories: inter-subject (i.e., static) and intra-subject (i.e., dynamic or time-varying) detection tasks.
Inter-subject tasks involve predicting stable attributes or baseline conditions across individuals, such as demographic variables or medical history at the time of enrollment. These tasks allow us to benchmark performance against prior foun- dation modeling work in wearables (Abbaspourazud et al., 2024a), ensuring comparability and reproducibility. In par-
6

---

Beyond Sensor Data: Foundation Models of Behavioral Data from Wcarables Improve Health Predictions
Table 2. Performance of embeddings on vire-varying health detection (AUROC) and sleep regression (R²) tasks with 95% bootstrap confidence intervals below in parenthesis. The combination WBM and PPG consistently yields the best performance.
Embeddings
Diabetes (AUROC)
Pregnancy (AUROC)
(AUROC)
Infection
Injury (AURÓC)
Duration (/t2)
Sleep
Efficiency (Jt")
Sleep
Deep Sleep Duration (It")
Ducation (/t")
REM Sleep
0.737
0.804
0.632
0.608
0,104
0.131
0,172
0.128
Raseline
(0.729, 0.744)
(0.795, 0.813)
(0.626, 0,638)
(0.605, (0.605.0.11)
(0.093. 0.112)
(0.127. 0.135)
(0.168, 0.176)
(0.124, 0.131)
0.765
0.864
0.749
0.680
0.590
0,424
0.266
0.326
WBM
(0.758, 0,772)
(0.855, 0.873)
(0.744, 0.755)
(0.677, 0.683)
(0.587, 0.594)
(0.419, 0.429)
(0.261, 0.270)
(0.322, 0,33
0.829
0,873
0.730
0.673
0,110
0.182
0.327
0.230
PPG
(0.823, 0.836)
(0 865, 0,882)
(0.725, 0.735)
(0.671, 0.676)
106, 0.114)
178. 0.186)
(0.323, 0.331)
(0.226, 0.235)
WBM + PPG
0.828
0,921
0.763
0,688
0.601
0,438
0,383
0,393
(0.822, 0.834)
(0.914, 0.928)
(0.757, 0.768)
(0.685, 0.691)
(0.598, 0.605)
(0.433, 0.443)
(0.379, 0.3887
(0.388, 0.397)
Baseline
WBM
PPG
WBM PPG
0.9
AUROC
0.8
0.7
0.6
Artery disease Osteopoross
Heart fallure
Heart attack
Pacomaker Heart disease
Affo
Active smoker
Hip/Knee
Anti psychoticcc
Diabetes
Beta- boo-bbockerrs
Anticoagulatt
Neurapathy
Urinary
Falckum-channl blockere
Sleep apnea
ACE inhibiitors
Blood pressure
Opield painkcilters
Arthrits
Cancer Andplatelets
Stroke TIA
Andeety
Anddepressants sants
Depression
On any medication Chalesteral
Thyroid
Chronic bronchits
Liver
Diuretics
Neck disarder
Anti andeey
Pain
Hearing
Blood pressure med.
Kidney
Chemother apy
Lower back
Sleep medication
Heart rhythm
Allergy
Vision
Asthina
Paincilltrrs
Baseline Disease or Medication Outcame
Figure 3. Results for predicting baseline medical history and medications (AUROC). The learned representations from WBM achieve strong performance and consistently outperform the baseline. Furthermore, the combination of WBM and PPG performs best in almost every task, demonstrating the complementarity of behavioral data and sensor data for health detection tasks. See Appendix Table 26 for the exact numbers for each model and task.
ticular, we include age and biological sex prediction, as they serve both as widely-used proxies for general health status and as sanity checks for representation quality, since well-trained embeddings should reliably encode this infor- mation. While there is some potential for label leakage, given that certain WBM input variables are derived from algorithms that explicitly incorporate age and sex, we con- sider this acceptable in context, as these tasks are primarily used for validation rather than as potential deployment tar- gets, Furthermore, inter-subject predictions may be useful in real- hal-world settings for digital health personalization or screening applications, where quickly inferring baseline characteristics without requiring detailed input from users can improve usability and accessibility.
tasks include detecting weekly sleep quality metrics, dia- betes status via HbAlc labs, self-reported pregnancy, and injury states. These tasks probe whether the learned repre- sentations are sensitive enough to detect changes in health trajectories, a core capability for preventive and personal- ized digital health technologies.
Together, this diverse task suite enables us to evaluate both the generalizability of our model across individuals and its temporal sensitivity within individuals. It spans multiple health domains and leverages both behavioral and physiolog- ical modalities, providing a rigorous assessment of WBM's utility foundation model for real-world health detection.
Full details on preprocessing and task setup can be found in Appendix 1.
Intra-subject tasks, by contrast, are designed to test whether our model can track meaningful within-person variation over time, capturing transient or emerging health states. These tasks are unique to our work and highlight an important use case for wearables: continuous monitoring and detection of dynamic conditions. For example, we frame respiratory in- fection detection as a weekly classification problem, where the model must distinguish healthy vs. sick periods using historical behavior and physiology data. Other intra-subject
5.2. Results and Key Takeaways
Table 1 contains the demographics prediction results, Fig- ure 3 contains the baseline medical history and medication results, and Table 2 contains the intra-subject results. Ad- ditional results within demographic subgroups for a select number of tasks can be found in Appendix A.6.2.
7

---

Beyond Sensor Data: Foundation Models of Behavioral Data from Wcarables Improve Health Predictions
Foundation models of behavioral data encode substantial health information. WBM achieves strong performance across all tasks. Notably, WBM substantially outperforms the simple supervised baseline on all demographic tasks. Moreover, WBM is consistently strongly predictive of most baseline disease and medication health outcomes, and usu- ally outperforms the simple baseline. In total, WBM out- performs the baseline in 39 out of 47 outcomes, with a median AUROC improvement of 0.017 in these outcomes and a statistically significant improvement in 30 out of 47 outcomes. In the eight outcomes in which the baseline model performs best, WBM is still competitive (median AUROC deficit of 0.008). Finally, WBM is able to achieve strong results across intra-subject tasks as well, statistically significantly outperforming the baseline model in all 8 time- varying health detection tasks. As sanity check, we also show that WBM accurately reconstructs the weekly mean values for most of the 27 variables considered, including a majority of the sparser input variables such as mobility metrics, in the Appendix Table 11.
consistently when performing subject-level classification of baseline disease or medication outcomes (29 out of 47 outcomes with median performance gain of .017 and 18 significant performance improvements), as the aggregation of every PPG over the study is sufficient for detecting many baseline outcomes. This is clear when looking at the out- come of antidepressants, where PPG excels as shown in past work (Abbaspourazad et al., 2024a), and behavioral data provides only marginal improvements.
Models of behavioral data provide complementary in- formation to models of low-level sensor data. Finally, we see that across most tasks, the combination of embeddings of WBM and the PPG model result in the most accurate models. The combination achieves the best age prediction performance across all models considered, clearly outper- forming either model in isolation. This trend also holds for participant-level baseline disease and medication outcome prediction, where the combination of both modalities almost always improves upon the best single modality. In total, the combination performs best in 42 out of 47 outcomes, with a median AUROC improvement of 0.009 in these outcomes and with many results being statistically significant (38 over WBM and 33 over PPG). Combining the behavioral data with PPG results in substantial gains over PPG alone in many tasks, most notably in predicting Afi (AUROC gain of 0.034), beta-blockers (AUROC gain of 0.055), and calcium-chanme blockers (AUROC gain of 0.033). Finally, the combination of modalities remains the best model in every time-vrrying health state prediction task except dia- hetes, where PPG alone was sufficient. The combination is remarkably performant for pregnancy prediction, where the combination of both data types achieves an AUROC over 0.9 and provides a relative improvement of about 0.05 over each modality alone. Pregnancy results in substantial changes in both the underlying physiology of an individual, as tracked by raw sensors, and substantial changes to an individual's behavior, as measured by derived metries like exercise minutes, step counts, and gait. Hence, this task acts as a clear example of the complementary nature of modeling both types of data. Combining behavioral data with PPG also improves performance all sleep-related tasks, and in detecting infection (AUROC gain of 0.033) and detecting injury (AUROC gain of 0.015).
Behavioral wearable data contains more signal than low- level sensor data for some tasks. WBM outperforms the PPG model across several tasks, especially those where we expect behavioral information should provide extra signal. This is most apparent in the sleep tasks, where WBM re- sults in consistent performance improvements. This is not surprising- - behavioral data capture information from ev- ery hour of the week, including overnight periods, where we can infer how long someone was asleep by periods of inactivity (e.g., from step count, exercise minutes, and heart rate). However, PPG does not provide as comprehensive view of an individual's week, since it is only measured a few times each day. WBM also provides gains in other tasks such as infection and injury, where behavioral data may provide additional information about an individual's ss activity that results more accurate models compared to lower-level PPG sensors (e.g.. gait and mobility metrics will likely change after a lower limb injury). WBM also outperforms PPG on 18 of the 47 baseline disease and medi- cation outcomes, including four with statistically significant differences. Notably, WBM achieves better performance in predicting beta blocker use, which is consistent with its ability to more reliably detect heart rate reductions through- out the day, information that may be missed by the more sporadic measurements of PPG.
6. Discussion
Low-level sensor data outperforms behavioral data in tasks where physiological information is sufficient. There are many tasks in which the lower-level PPG data is suf- ficient, and so lover-level physiology provides enough in- formation to achieve strong performance. This is most ap- parent in the time-varying diabetes task, where the PPG model outperforms all others, including when combined with the behavioral data foundation model. This also occurs
This work built performant predictive models for meaning- ful health detection tasks. We approached this problem by building WBM, a foundation model trained on the largest and most diverse wearables dataset to date. Specifically, we considered quantities that capture individuals' behaviors and are derived from low-level sensor data collected from wearables. These types of quantities typically are computed
8

---

Beyond Sensor Data: Foundation Models of Behavioral Data from Wcarables Improve Health Predictions
using carefully validated algorithms trained from limited be- spoke data. The quantities are also known to relate to many health detection tasks. Focusing on these derived quantities circunvents a model needing to learn these relationships - which may not even be possible using the observational datasets used in this and other work.
experiments with masked autoencoder training were not promising, we suspect this may be due to the high degree of noise present in behavior data, and hypothesize that joint embedding predictive architectures might perform better. Moreover, as many of our downstream task labels come from self-reported surveys, some labels may be inaccurate. Finally, WBM is not designed to forecast future health states from an individual's history.
The derived behavioral quantities that we model differ from the sensor data (or simple features thereof) considered in previous works due to irregularity, with different sampling frequencies and large amounts of missingness driven by behavior. As such, we performed principled search over tokenization strategies and model architectures. The fact that the best- perforrinn tokenizer was the TST approach that produced a dense input format is somewhat surprising, since we found that imputing missing values by the global mean across all subjects rather than using subject-specific means worked better. We hypothesize that this may be due to high levels of noise when estimating variable means for some subjects, and in future work it is worth exploring if more complex model-based methods for imputation might perform even better. Another surprising finding was that the bi-directional Mamba-2 architecture produced the best model for our tasks and data, given the ubiquity of the trans- former architecture. This motivates further developments of state- sate-space models for encoder-only foundation models of time series and sensor data. A key takeaway of our process is that when developing foundation models in the health space it is crucial to evaluate all approaches when using new types of data, and not to assume previous models will necessarily be the best for your problem and data.
If developed and deployed safely and responsibly, predic- tive models built on wearable data like WBM hold signifi- cant promise for clinical impact. By enabling continuous, non-invasive monitoring and early detection of meaningful health events, such models could support more proactive and personalized care- ciire-partcullaryy for conditions where behavioral signals are strong early indicators. In the fu- ture, these approaches could complement clinical decision- making, help triage patients for follow-up, or support just- in-time interventions, especially in populations where tradi- tional healthcare access is limited. Realizing this potential will require careful attention to model fairness, calibration, interpretability, and robust validation across diverse cohorts.
Impact Statement
Our work is focused on enhancing health detection tasks by modeling behavioral data from wearable devices. Given our focus on wearables, we acknowledge the potential societal impact of deploying advanced health prediction models given data from wearables. Improving these prediction models may exacerbate equity gaps between individuals with access to wearable devices to those without. It remains essential to consider ways to democratize the benefits and findings of our work to other forms of data and devices that ensure equity across different groups. Moreover, as our ultimate goal is to perform predictions that may inform health-related decisions, our work has potential for positive socictal consequences. It is important further evaluate our models in terms of actionability to best realize this goal.
We tested our model across suite of tasks that are repre- sentative of real-world health detection problems some from previous work and others novel. We demonstrated that models of behavioral data encode significant information about an individual's health, outperforming models derived from low-level sensors such as PPG in tasks where behavior is meaningful predictor (e.g., sleep and injury detection), and complementing these models across a majority of other tasks by improving the predictive performance of the joint PPG + WBM model. These results expand the foundation modeling paradigm to derived behavioral quantities benefit- ing a breadth of real-world health detection tasks.
LCKNOWLEDGMENTS
We would like to thank participants in the Apple Heart and Movement Study, Calum MacRae, and study staff at The Brigham and Women's Hospital, a Harvard affiliate, with- out whom this work would not have been possible. We are grateful to Anshuman Mishra and Valerie Marchenko for important contributions to the data processing, training, and evaluation infrastructure. We also thank Andy Miller, Guillermo Sapiro, Ian Shapiro, Jen Block, and the anony- mous reviewers for providing valuable feedback, and Lind- say Hislop. Eduardo Martinez Montes and Laura Rhodes for publication coordination help.
This work focused specifically on incorporating individu- als' behaviors into predictive models for health detection tasks using specific dataset and thus has limitations. We cannot say how well the specific model we developed will perform on other datasets, particularly when data is not col- lected with Apple devices (c.g.. Apple Watch and iPhone). Another limitation is that the contrastive self-supervised loss requires defining positive and negative pairs which can be arduous and may affect the types of downstream tasks the learned representations can address. While our initial
9

---

Beyond Sensor Data: Foundation Models of Behavioral Data from Wcarables Improve Health Predictions
References
Chowdhury, R. R., Li, Zhang, X., Hong, D., Gupta, R. K., and Shang. J. Primenet: Pre-training for irregular multi- variate time series. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 37, pp. 7184-7192, 2023.
Abbaspourazad, S., Elachqar, O., Miller, A., Emrani, S., Nallasamy, U., and Shapiro, I. Large-scale training of foundation models for wearable biosignals. In Interna- tional Conference on Learning Representations, 2024a.
Dao, T. and Gu, A. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. arXiv preprint arXiv:2405. 2024.
Abbaspourazad, S., Mishra, A., Futoma, J., Miller, A. C., and Shapiro, I. Wearable accelerometer foundation mod- els for health via knowledge distillation. arXiv preprint arXiv:2412.11276 2024b.
Denny, J. C., Rutter, J. L., Goldstein, D. B., Philippakis, A., Smoller, J. W., Jenkins, G., Dishman, E., and The All Us Research Program Investigators. The "All of Us" Research Program. New England Journal of Medicine, 381 381(7):668-676, August 2019, ISSN 0028-4793, doi: 10. (0.1056/NE/Mss118999911 URL https: /www .ne jm. org / doi / o 1056/NE.JMs /N. 809937. Publisher:
Ahmed, A.. Aziz, S., Abd-Alrazag A., Farooq, F., and Sheikh, J. Overview of artificial intelligence- wear- able devices for diabetes: scoping review. Journal of Medical Interner Research, 2468):330010 2022.
Ansari, A. F., Stella, L.., Turkmen, C., Zhang, X., Mercado, P., Shen, H., Shchur, O., Rangapuram, S. S., Arango, P., Kapoor, S., et al. Chronos: Learning the language time series. arXiv preprint arXiv:2403.07815. 2024.
Massachusetts Medical Society.
Dufter, P., Schmitt, M., and Schütze, H. Position infor- mation in transformers: An overview. Computational Linguistics, 48(3):733-763, 2022.
Apple. Estimating sleep stages from Apple watch. link, September 2023.
Fallahpour, A., Alinoori, M., Afkanpour, A., and Krish- nan, A. Ehrmamba: Towards generalizable and scalable foundation models for electronic health records. arXiv preprint arXiv:2405. 2024.
Assran, M., Duval, Q., Misra, I., Bojanowski, P., Vincent, P., Rabbat, M., LeCun, Y., and Ballas, N. Self-supervised learning from images with joint-embedding predictive architecture. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 15619- 15629, 2023.
Garza, A. and Mergenthaler-Canseeo, M. Timegpt-l. arXiv preprint arXiv:2310.03558 2023.
Gomes, N., Pato, M., Lourenco, A. R., and Datia, N. survey on wearable sensors for mental health monitoring. Sensors, 23(3):1330, 2023.
Bianchi, M. T. Sleep devices: wearables and nearables, informational and interventional, consumer and clinical. Metabolism, 84:99-108, 2018.
Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:23/2.00052, 2023.
Bommasani, R., Hudson, D. A.. Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg. J., Bosse- lut, A., Brunskill, E., et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108. 078.07258 2021.
Jeong. H., Oufattole, N., Balagopalan, A., Medermott, M.. Chandak, P., Ghassemi, M., and Stultz, C. Event-based contrastive learning for medical time series. 2023.
Bycroft, C., Freeman, C., Petkova, D., Band, G., Elliott, I. T., Sharp, K., Motyer, A., Vukcevic, D., Delaneau, O., O'Connell, J., et al. The UK biobank resource with deep phenotyping and genomic data. Nature, 562(7726): 203-209, 2018.
Labach, A., Pokhrel, A., Huang, X. S., Zuberi, S., Yi, S. E., Volkovs, M., Poutanen, T., and Krishnan, R. G. Duett: dual event time transformer for electronic health records. In Machine Learning for Healthcare Conference, 2023.
Channa, A., Popescu, N., Skibinska, J., and Burget, R. The rise wearable devices during the COVID-19 pandemic: systematic review. Sensors, 21 (17)557777 2021.
Liang, A., Jiang, X., Sun, Y., and Lu, C. Bi-mamba4ts: Bidirectional mamba for time series forecasting. arXiv preprint arXiv:2404. 15772, 2024.
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual rep- resentations. International Conference on Machine Learning, pp. 1597-1607. PMLR, 2020.
Lipton, Z. C., Kale, D., and Wetzel, R. Directly modeling missing data sequences with RNNs: Improved classi- fication of clinical time series. In Machine Learning for Healthcare Conference, pp. 253-270. PMLR, 2016.
10

---

Beyond Sensor Data: Foundation Models of Behavioral Data from Wcarables Improve Health Predictions
Liu, H., Dai, Z., So, D., and Le, Q. V. Pay attention mlps. Advances in Neural Information Processing Systems, 34: 9204-9215, 2021.
Rasul, K., Ashok, A., Williams, A. R., Khorasani, A., Adamopoulos, G., Bhagwatkar, R., Bilok, M., Ghonia, H., Hassen, N. V., Schneider, A., et al. Lag-Ilama: Towards foundation models for time series forecasting. arXiv preprint arXiv:2310.08275: 2023.
Lyzwinski, L.,, Elgendi, M., and Menon, C. Innovative approaches menstruation and fertility tracking using wearable reproductive health technology: systematic re- view. Journal of Medical Internet Research, 26:e45139, 2024.
Ren, H., Wang, J., Zhao, W. X., and Wu, N. Rapt: Pre- training of time-aware transformer for learning robust healthcare representation. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery Data Mining, 2021.
MacRae, C. A. Apple Heart & Movement Study. Clin- ical trial registration NCT04198194, clinicaltrials.gov, July 2021. URL https: http:////nnaaaaaaa////aaaaaaaaaa gov/ study/NCT0419811 submitted: November 13, 2019.
Sablayrolles, A., Douze, M., Schmid, C., and Jégou, H. Spreading vectors for similarity search. arXiv preprint arXiv:1806.033: 2018.
Shapiro, I., Stein, J., MacRae, C., and O'Reilly, M. Pulse oximetry values from 33,080 participants in the Apple
Merrill, M. A. and Althoff, T. Self-supervised pretraining and transfer learning enable flu and COVID-19 predic- tions in small mobile sensing datasets. In Conference Health, Inference, and Learning, pp. 191-206. PMLR, 2023.
Heart & Movement Study. npj Digital Medicine, 6 (1):1-13, July 2023. ISSN 2398-6352. doi: 10.1038/
$41746-023-00851-6 URL https: //ww: nature. com http//ariceesssss11 123111146-------6 0023-0085166 Num- ber: Publisher: Nature Publishing Group.
Merrill, M. A., Safranchik, E.,, Kolbeinsson, A., Gade, P., Ramirez, E., Schmidt, L., Foschini, L., and Althoff, T. Homekit2020: A benchmark for time series classification on large mobile sensing dataset with laboratory tested ground truth influenza infections. Conference on Health, Inference, and Learning, 2023.
Shaw, P., Uszkoreit, J., and Vaswani, A. Self-attention with relative position representations. arXiv preprint MwXiv:/003..r 02..02/55 155, 2018
Shukla, N. and Marlin, B. Multi-time attention networks for irregularly sampled time series. In International Con- ference on Learning Representations, 2021.
Mishra, A., Park, J., Shapiro, I.,
T., Baird, D. D., Suharwardy, S., Zhang, S., Jukic, A. M. Z., and Curry, C. L. Trends in sensor-basee health metrics dur- ing and after pregnancy: descriptive data from the ap- ple women's 's health study. AJOG Global Reports, 4(4): 100388, 2024.
Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.
Tipimeni, and Reddy, C. K. Self-supervised transformer for sparse and irregularly sampled multivariate clinical time-series. ACM Transactions on Knowledge Discovery from Data (TKDD), 16(6):1-17, 2022.
Moshawrab, M., Adda, M., Bouzouane, A., Ibrahim, H., and Raad, A. Smart wearables for the detection of cardio- vascular diseases: a systematic literature review. Sensors, 23(2):828, 2023.
Truslow, J., Spillane, A., Lin, H., Cyr, K., Ullal, A., Amold, E., Huang, R., Rhodes, L., Block, J., Stark, J., Kretlow, J., Beatty, A. L., Werdich, A., Bankar, D., Bianchi, M., Shapiro, I., Villalpando, J., Ravin- dran, S., Mance, I., Phillips, A., Earl, J., Deo, R. C., Desai, S. A., and MacRae, C. A. Understanding ac- tivity and physiology at scale: The Apple Heart & Movement Study. npj Digital Medicine, 7(1):1-11, doi: 10.1038/
Narayanswamy, G., Liu, X., Ayush, K., Yang, Y., Xu, X., Liao, S., Garrison, J., Tailor, S., Sunshine, J., Liu, Y., et al. Scaling wearable foundation models. arXiv preprint arXiv:2410.13635 2024.
Nestor, B., Hunter, J., Kainkaryam, R., Drysdale, E., Inglis, J. B., Shapiro, A., Nagaraj, S., Ghassemi, M., Foschini, L., and Goldenberg, A. Machine learning COVID-19 detection from wearables. The Lancet Digital Health, 5 (4))e182-el1) 2023.
September 2024. ISSN $41746-002-011187-55 URL https: nature. com/ http//rricess///s11/ lisher: Nature Publishing Group.
122-01187-41 Pub-
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, E.,, and Polosukhin, I. Atten- tion is all you need. In Advances in Neural Information Processing Systems, 2017.
Ramachandran, P., Zoph, B., and Le, Q. V. Swish: a self-gated activation function. arXiv preprint 17Xiv:1710.0594/ 7(1):5 2017.

---

Beyond Sensor Data: Foundation Models of Behavioral Data from Wearables Improve Health Predictions
Wang, S., McDermott, M. B., Chauhan, G., Ghassemi, M., Hughes, M. C., and Naumann, T. Mimic-extract: A data extraction, preprocessing, and representation pipeline for mimic-iii. In Proceedings of the ACM conference on health, inference, and learning, pp. 222-235, 2020.
Wang, Z., Kong, F., Feng, S., Wang, M., Zhao, H., Wang, D., and Zhang, Is mamba effective for time series forecasting? arXiv preprint arXiv:2403. 241::24433./44, 2024.
Woo, G., Liu, C., Kumar, A., Xiong, C., Savarese, S., and Sahoo, D. Unified training of universal time series fore- casting transformers. arXiv preprint arXiv:2402.02559, 2024.
Wornow, M., Bedi, S., Hernandez, M. A. F., Steinberg, E., Fries, J. A., Ré, C., Koyejo, S., and Shah, N. H. Context clues: Evaluating long context models for clinical predic- tion tasks on EHRs. arXiv preprint arXiv:2412. 16178, 2024.
Xu, M. A., Narain, J., Darnell, G., Hallgrimsson, H., Jeong, H., Forde, D., Fineman, R., Raghuram, J., Rehg. J. M., and Ren, S. Relcon: Relative contrastive learning for motion foundation model for wearable data. arXiv preprint ar 2r 2024.
Xu, Y., Biswal, S., Deshpande, S. R., Maher, K. O., and Sun, Raim: Recurrent attentive and intensive model of multimodal patient monitoring data. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2565-2573, 2018.
Yuan. H., Chan, S., Creagh, A. P., Tong, C., Acquah, A., Clifton, D. A., and Doherty, A. Self-supervised learning for human activity recognition using 700,000 person-days of wearable data. npj Digital Medicine, 7(1):91, 2024.
Zerveas, G., Jayaraman, S., Patel, D., Bhamidipaty, A., and Eickhoff, C. A transformer-based framework for multi- variate time series representation learning. In Proceed- ings of the 27th ACM SIGKDD Conference Knowledge Discovery & Data Mining, pp. 2114-21244 2021.
Zhang, B. and Sennrich, R. Root mean square layer nor- malization. Advances in Neural Information Processing Systems, 32, 2019.
Zhang, J., Zheng, S., Cao, W., Bian, J., and Li, Warp- former: A multi-scale modeling approach for irregular clinical time series. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2023.
Zheng, X., Chen, X., Schürch, M., Mollaysa, A., Allam, A., and Krauthammer, M. Simts: rethinking contrastive representation learning for time series forecasting. arXiv preprint Xrvv:22033 18205, 2023.
12

---

Beyond Sensor Data: Foundation Models of Behavioral Data from Wcarables Improve Health Predictions
A. Appendix
A.I. Additional Dataset Details
A. VARIABLE INFORMATION
We list the different activity variables used as input modalities in our dataset in Table 3. For certain quantities that can be captured by a watch or phone, we separated these out into different variables. We also list how different variables are aggregated into hourly-level statisties when quantity is measured multiple times within the hour. Variables that cumulative (e.g., exercise time, step counts) are aggregated using a sum, while momentary variables (e-g., heart rate, respiratory rate) are aggregated using average. The table also shows the frequency of collection of different variable types, both globally (i.e., what percentage of participant- weeks ever have an observed value for that variable), and per-subject (i.e., what percent of subjects ever have an observed value for that variable). As expected, the most common variable types are quantities such as heart rate (observed in 99.9% of subjects and 91.8% of weeks) and activity metrics such as step count. Less common variables were things like number of falls (only observed in less than 3% of people, as most people do not experience fall), or overnight wrist temperature (which only collected overnight on more recent Apple Watch versions). There is also substantial irregularity in the number of hourly variables recorded per week across the whole dataset. The median number of hourly variables recorded each week is 992, with an interquartile range of 829 to I 1149, showing considerable variation in the number of observed variables each week.
1.1. INFORMATION ON PARTICIPANT DELFRREORTED MEDICAL HISTORY AND MEDICATIONS SURVEYS
We describe the different survey questions about medical questions in Table 4 and medications in Table 5. These questions are found in the AHMS surveys and help define outcomes for participant-level health questions that we use to understand the information encoded in WBM.
We also include two additional survey questions downstream detection tasks: 1) Active Smoker (N 23,8345): Did the participant answer "Yes" when asked about smoking status and "Every day" or "Some days" when asked about how often, or did they answer "No" when asked about smoking status, and "Not at all" when asked about how often, 2) On any medication (N = 23,943): Did the participant answer yes or no to the question "Do you currently take any form of medication"?
1.1.3. TREINIAINING DATASET DETAILS
To create the full pre-training dataset, we filter to only weeks of data with at least days of watch wear (determined by if there are heart rate samples on unique days), and we preprocess such that each week begins at the same relative time, midnight Monday. We further filter to include participants who have at least 5 usable weeks of data, and who have been enrolled in the study for at least 90 days.
We also show below in Table 6 the breakdown of demographics across all 161,855 subjects in the pre-training dataset. Given the large sample sizes, the demographics are nearly identical across the 80% train, 10% validation, and 10% test splits so we report demographics across the full cohort in the table. Note that participants may self-report multiple race/ethnicity columns, so they do not sum to 100%.
A.2. Additional Methods Details
1. SETUP
We assume one hour of wearable data at time t from subject s is a tuple of (t", xf) where t" is the hour of recording, oe the activity type or variable name, and x is the value of the variable time t.. We denote a segment of wearable data as dē. Uler(!* of, .f),, ), or the union of all observations within some week T. Irregularity from the signal comes from 1) events not being sampled at similar rates and 2) single events not being sampled consistently at the same rate. We z-score all input variables with respect to every observation in the training dataset and clip outliers.
A.3. Details on Tokenization of Wearable Data
In the main paper we consider two primary methods for tokenizing each week of data: 1) approaches that model each observation separately (Tuple), and 2) approaches that aggregate all information into regularly sampled dense-matrix (Dense).
13

---

Beyond Sensor Data: Foundation Models of Behavioral Data from Wearables Improve Health Predictions
Weeks
% Subjects
Sampling
Hourly Aggregation
Variable
Category
Notes
With 1+ Value
With 1+ Value
Rate
Flights climbed (phone)
Activity Activity
<Hourly <Hourly
Sum
68.28
98.32
Flights climbed (watch)
Sum
65.06
97.59
Calories burned
Active energy burned
Activity
Hourly
Sum
93.11
99.83
while active
Calories burned
Basal energy burned
Activity
Hourrly
Sum
95.21
99.50
at rest
Step count (phone)
Activity
<Houry
Sum
98.34
99.33
Step count (watch) Exercise minutes
Activity
Hourly
Sum
90.67
99.49
Activity
Houury
Sum
86.35
99.64
Stand time
Activity
<Hourly
Sum
90.42
99.45
Resting heart rate
Cardiovascular
Daily
Mcan
88.45
99.63
Walking heart rate
Cardiovascular
Daily
Mean
84.27
99.39
Heart rate
Cardiovascular
<Huury
Mean
91.76
99.85
Calculated via
Standard Deviation
Roughly every few hours
Heart rate variability
Cardiovascular
Mean
Normal -to Normal
88.56
99.82
Interval
(SDNN)
Respiratory rate
Vitals
<Hourly Roughly every few hours
Mean
Overnight only Only available on Series 6+ Apple Watch Single value, overnight only
35.21
60.79
Blood oxygen saturation
Vitals
Mean
44.31
62.56
Wrist temperature
Vitals
Daily
Mean
3.12
10.42
Walking speed
Mobility Gait
Hourly Hourly
Mean
84.59
91.17
Walking step length
Mobility Gait
Mean
84.53
91.09
Walking double support percentage
Mobility Gait
Hourly
Mcan
83.23
90.86
Walking asymmetry
Mobility Gait
Hourly
Mcan
69.78
90.88
percentage
Stair ascent speed
Mobility Gait
Hourly
Mean
41.64
76.96
Stair descent speed
Mobility Gait
Hourly
Mean
41.72
76.79
Fall count
Mobility/ Gait
Opportunistic
Sum
0.04
2.85
An estimate of stability while walking
Walking steadiness
Mohility Gait
Weekly
Mean
9.64
24.08
From third-party devices manual input
Body
Body mass
Opportunistic
Mean
10.93
54.76
Measurements
From third-pane
Body
Body mass index
Opportunistic
Mcan
devices manual input
7.97
33.84
Measurements
Cardio Fitness / Functional
Requires outdoor walk/run/hike workouts
VO2max
Opportunistic
Mean
16.07
77.27
Capacity
Cardio Fitness /
minute walk distance
Functional
Weekly
Mean
78.82
Capacity
Table 3, List of all derived health and behavioral quantity variables used in modeling, along with the method of aggregation used to create hourly-level aggregates, a brief description, the native sampling rate, and summary statistics (what percentage of individual weeks and subjects have at least one measurement).
Tuple. First, we consider mapping each tuple (t", uZ, E di to a d-dimension token et that can be used as input to a sequential deep-learning model. To do so, we first consider learning mappings dn x - Rd and Φ, : v - Rd From these vectors, we compute the d-dimensional input token ef as (v())) (x(=f) (Tipimeni & Reddy, 2022: Zhang al., 2023; Labach et al., 2023). This results in a single input token for each unique behavioral measurement. segment
14

---

Beyond Sensor Data: Foundation Models of Behavioral Data from Wearables Improve Health Predictions
Table AHMS survey questions about medical conditions. The main question is in form of 'Have you been diagnosed with any of the following conditions?" and participants can answer 'Yes' or 'No" or prefer not to answer' or 'I don't t know". The question for Third column indicates the number of left
vision and hearing loss is different, which explicitly mention in the corresponding participants for evaluation the reason for variations that for each target we exclude participants whose answers were prefer not to answer' 'I don't to know" missing.
Target label
Medical condition
(test)
Pos-
itive (test)
Heart attack
Heart attack (myocardial infarction)
27,009
1.56
Heart discase
Coronary heart discase angina pectoris
26,765
2.45
Blood pressure
High blood pressure (hypertension)
26,519
25.11
Stroke or TIA
Stroke (ccrebral hemorrhage, cerebral thrombosis) or tran- sient ischemic attack (ministroke)
27,022
1.45
Afib
Atrial fibrillation
26,457
3.22
Heart rhythm
Heart rhythm problem other than atrial fibrillation
26,264
8.86
Pacemaker
Pacemaker
27.128
0.56
Artery disease
Peripheral artery disease
26,627
0.74
Heart failure
Heart failure
27,051
0.95
Diabetes
Diabetes
26,861
6.67
Cholesterol
High cholesterol
26,414
27.00
Arthritis
Arthritis
26,588
18.38
Hip/Knee
Hip or knee replacement
27,153
2.54
Lower back
Low back disorder other chronic back defect
26,646
16.27
Neck disorder
Neck disorder other chronic neck defect
26,811
7.64
Sleep apnea
Sleep apnea
25,912
15.78
Ostcoporosis
Osteoporosis
26,731
2.89
Asthma
Asthma
26,588
21.08
Chronic bronchitis
Chronic bronchitis, chronic obstructive pulmonary disease, emphysema
26,919
3.57
Allergy
Rhinitis, hay fever, eye inflammation, dermatitis, food lergy or other allergy (allergic asthma excluded)
26,811
34.58
Kidney
Kidney problems
26,822
4.50
Thyroid
Thyroid disease Cancer
26,696
7.53
Cancer
26,993
5.05
Liver
Cirrhosis the liver
26,985
0.44
Urinary
Urinary incontinence
26,912
3.76
Neuropathy
Neuropathy
26,542
4.36
Depression
Depression
26,284
39.11
Anxiety
Anxiety disorder
26,167
37.97
Hearing
Do you have hearing loss?
24,457
15.64
Vision
Do you have vision loss?
25,663
25.15
15

---

Beyond Sensor Data: Foundation Models of Behavioral Data from Wearables Improve Health Predictions
Table AHMS survey questions about medications. The main question is in form of 'Do you currently take any the following types of medications?" and participants can answer "Yes' or "No' or 'I prefer not answer'. The formatting for the medications is similar their presentation tudy, but may not exactly match format the study application. Third column indicates number of left out participants for evaluation the reason for variations that for each target exclude participants whose answers were prefer not answer' or missing. Third party trademarks used herein are trademarks of their respective owners.
Target label
Medications
N (test)
%
Pos-
itive (test)
ACE-inhibitors
ACE-inhibitors or ARBs (for blood pressure) such as cap- topril, enalapril, lisinopril, losartan, ramipril, or valsartan Anti-anxiety aids such as alprazolam (Xanax clon- azepam (Klonopin") clorazepate (Tranxene diazepam (Valium") )) or lorazepam (Ativan")
15,736
18.44
Anti-anxiety
15,742
17.50
Anti-psychotics
Anti-psychootes such haloperidol (Haldol)), aripipra- zole (Abilify"), risperidone (Risperdal"), quctiapine (Seroquel"), olanzapine (Zyprexa") clozapine (Clozaril"), or lurasidone (Latuda")
15,780
3.97
Anticoagulants
Anticoagulants (blood thinners) such as warfarin (Coumadin ") apixaban (Eliquis") betrixaban (Bevyxxa*), dabigatran (Pradaxa ", edoxaban (Lixiana") ") rivaroxa- (Xarelto")
15,774
3.77
Antidepressants
Antidepressants such as amitriptyline (Elavil"), bupro- pion (Wellbutrin"), citalopram (Celexa"). duloxe- (Cymbalta escitalopram (Lexapro"), fluoxetine (Prozac 2 ). paroxetine (Paxil mirtazapine (Remerou"), sertraline (Zoloft"), ), or venlafaxine (Effexor*)
15,782
36.70
Antiplatelets
Antiplatelets (blood thinners) such aspirin, clopidogrel (Plavix "), prasugrel (Effient"), ), or ticagrelor (Brilinta" beta-blockers (for blood pressure heart rhythm) such as atenolol (Tenormin") bisoprolol (Zebeta"), carvedilol (Coreg ". labetalol, metoprolol (Lopressor", Toprol-XL "). nadolol (Corgard"), nebivolol (Bystolic")) propranolol (Inderal") ), or sotalol (Betapace
15,758
9.29
Beta-blockers
15,722
13.29
Blood pressure med.
Other medications for lowering blood pressure such as clonidine, hydralazine, minoxidil, or sacubitril/valsartan (Fntresto
15,694
4.82
Calcium-channel blockers
Calcium-channel blockers (for blood pressure heart rhythm) such amlodipine (Norvasc") diltiazem. or ver- apamil
15,677
7.13
Chemotherapy
Certain types of chemotherapy such as carboplatin, cis- platin, oxaliplatin, vineristine, vinblastine
15,799
0.44
Diuretics
Diuretics (water pills) such as chlorthalidone, furosemide (Lasix ), hydrochlorothiazide, or spironolactone Opioid painkillers such as codeine, fentanyl, hydrocodone, hydromorphone (Dilaudid"), meperidine (Demerol") mor- phine, oxycodone, Percocet Vicodin" Non- Ste--stcoidal antiiiflaammtoriis (painkillers) such as aspirin, celecoxib (Celebrex") ) diclofenac (Cambia") ") ibuprofen (Motrin /Advil"), ), naproxen (Aleve") Sleeping aids such as eszopiclone (Lunesta ", zaleplon 15,753 (Sonata") or zolpidem (Ambien?)
15,765
10.59
Opioid painkillers
15,809
3.74
Painkillers
15,788
44.14
Sleep medication.
9.35
16

---

Beyond Sensor Data: Foundation Models of Behavioral Data from Wcarables Improve Health Predictions
Covariate
Mean (IQR)
Age
40.7 (30.0 49.4)
Female
0.358
Male
0.620
Sex other or not set
0.022
American Indian
0.022
Asian
0.071
Black
0,057
Hispanic
0.112
Middle Eastern
0.012
White
0.791
Ethnicity missing or preferred not answer
0.031
Ethnicity none describe me
0.013
BMI
28.6 (24.2
Table 6. Demographics of the full cohort of 161,855 participants used for pretraining.
of wearable data can then be represented as a sequence of these learned d-dimensional tokens as cq- Uver (ef). The tuple-based approach is well-suited for modeling irregular time-series, immediately handles irregular sampling for each variable type within segment of wearable sensor data well missing values, they are simply omitted. However, the tuple approach requires separate token for each observation, which may not scale well for longer segments of wearable data, particularly when using nemory-intensiv modeling techniques. We further illustrate Tuple tokenization in Figure 4.
Dense. potential limitation of tuple-based approaches is their poor scalablity and potential down-weighting of sparse but informative variables. Hence, we also consider the more traditional dense approach, where we convert data from tuples to a dense array of values. We create matrix X² E R x where [V] is the number of quantities or variables in the dataset and [7] is the total number of possible hours per segment. For each tuple (te, vf, xx{) dřj, fill in the matrix X with the value of index (t", vf)). We also concatenate missingness indicators to the feature matrix, that the final feature matrix is actually of size [T'] 2 [V]. We next describe two different techniques for tokenizing the resulting matrix Xi. that can handle missingness.
TST. First, we consider the standard technique of imputing missingness values and treating the resulting matrix X as a regularly sampled time-series. initial experiments, we found that the simplest form zero-imputation (i.e., global-average imputation with z-scored inputs) resulted in the strongest learned representations for majority of meaning downstream tasks. We also tried other imputation approaches, such using subject-speciffe means for each variable and week-specific means for each variable for given subject (if available), but these did not work as well in practice. Given this dense matrix representing a time series that is now regularly sampled, we learn a function that maps the vector of size 2. VV from each hour of data to d-dimensional vector of E Rd using a multi-layer perceptron (MLP) as in past work (TST) (Zerveas et al., 2021). These input tokens et = {ef her can then be used as input to an encoder for learning rich representations during pre-training.
mTAN. Second, we consider forgoing imputation and using multi-time attention with masking to tokenize the data (mTAN). We follow past work by (Shukla & Marlin, 2021) and (Chowdhury al., 2023) and use learnable timeeembeddings of size dr as queries Qr and keys Kr, and the embedding matrix Xř. values through an attention computation. We then compute tokens via a masked attention computation as eς (Mf O A4) where A7
and M E {0, | is a masking matrix where means a value is observed, and 0 means a value is unobserved. The resulting columns of size 2 [V] can then be used as token inputs to a downstream deep learning encoder.
A.4. Details on Model Architectures
Given the input segment di the goal is to create a learned representation r E Rdd that summarizes information from the full segment of wearable activity data.
Self-Attention Transformer. First, we consider standard self-attention techniques with learnable position encodings learned
17

---

Beyond Sensor Data: Foundation Models of Behavioral Data from Wearables Improve Health Predictions
Hour
Hour 2
Hour
Hour 4
Hour 168
Variable Embedding
Value Embedding
HK Tokenizer
D.1
Token Representations
Hour
Hour 2
Hour 3
Hour 168
Figure 4. Tuple tokenization framework for irregularly sampled wearable activity data. Different colors refer different variable types. Each variable is represented by a separate token unlike traditional multivariate modeling approaches. The token representation for each variable is the summation of its corresponding variable embedding and value embedding. For models that require learnable time embeddings, e.g. Transformer as in (Vaswani et al., 2017). we also include time embeddings in the summation.
through some model Φ: :t + Rd We augment each token eî with this learnable position encoding similar to past work through addition, where the new token is equal to ef + (t) (Tipirneni & Reddy, 2022).
Rotary Transformer. Next, we consider Rotary Position Embedding (RoPE), due to their strong performance compared to other variants, and their strong theoretical properties that show their ability flexibly encode both absolute and relative encodings (Shaw et al., 2018; Ren al., 2021; Dufter al., 2022; Su al., 2024). We modify the attention scheme of Transformers apply RoPE embeddings every layer using the hours times t as position IDs throughout.
For both schemes, we use a standard Transformer architecture, applying prior to attention and using gated multi-layer perceptron and swish activation functions (Ramachandran et al., 2017; Zhang & Sennrich, 2019; Liu et al., 2021). To create the representation r from the output the Transformer, we calculate the average representation across all input tokens in segment.
Mamba-2. Finally, consider using Mamba-2 backbone architecture learn downstream representations. The Mamba-2 architecture takes in the same tokens the Transformer without any additional positional encoding. As Mamba-2 a recurrent model and the goal is to learn representations of the full signal, we apply bi-directional Mamba-2 architecture as in past work using state-space models for time-series, allowing the representations to summarize information in both the forward and backward directions (Wang et al., 2024; Liang et al., 2024). To create the single representation rf, we first concatenate outputs of the forward and backward runs and use a single multi-layer perceptron to project this back to the original output space Rdd From here, similar the Transformer backbone, we calculate the average representation across all
18

---

Beyond Sensor Data: Foundation Models of Behavioral Data from Wearables Improve Health Predictions
input tokens in the segment.
A.4.1. REGULARIZED INFONCE LOSS
Given batch of positive pairs of segments (i.e., pairs of segments from the same individual), we use both an InfoNCE loss to maximize similarity between positive segments and contrasting these segments from other examples in the batch, as well as a Beaicchnno--eennnlee (KoLeo) regularization to help encourage a uniform span of features in a batch (Sablayrolles et al., 2018; Chen et al., 2020).
To calculate these losses, given a batch of N positive pairs, we pass these through the tokenization and modeling architecture obtain pairs of representations (ri.rš). We map these representations new space of embeddings ht)) using set of multi-layer perceptrons. We then calculate the one half of the InfoNCE loss as:
exp(sim(h), hb)/T)
2 InfONCE
log N İj=1 exp(sim (1)
where sim is the cosine similarity function. The InfoNCE loss encourages learning embeddings that are similar across positive pairs, and contrasts embeddings of segments from different subjects. To obtain symmetric results, calculate the full InfoNCE loss
1 - ImfONCE + LinONNE)
LifoNCE
2
Next, to calculate one half of the KoLeo regularization, we compute:
I KoLco
1 N
Σ
log(min || 11(1P).
##
This loss encourages uniform span of features across examples even further. For symmetry, calculate the full regularization loss as
1 (4kat. + L
LKoLco =
The final self-supervised pretraining loss over each batch can finally be computed as
LIBONNE + ALRot cas
where À is hyperparameter that controls the trade-off between the contrastive and regularization terms.
A.5. Additional Experimental Setup Details
A.5.1. MODELING AND HYPERPARAMETER CHOICES
the Adam W optimization with learning rate of 0.001 and sequential learning schedule. The learning rate begins with linear warm up with start factor of 0.5, followed by exponential strategy with gamma of 0.995. All backbones use feed-forward hidden dimension size (H) of 4 times the input hidden dimension (D) for feed-forward layers inside of the model block, and all Transformer models use 8 attention heads. To map the final representation from each backbone to a new dimension for loss calculation, we use feed-forward neural network of three layers of size input hidden dimension D, input hidden dimension Dx 4, and input hidden dimension D, with batch normalization and dropout of 30%.
For all models,
We conduct separate hyperparameter search jobs. First, perform smaller hyperparameter search across all 9 combinations of models. We describe the different hyperparameters in Table 7. For the ablation test, we use an input hidden dimension D of size 128.
For the final WBM model, we perform a larger hyperparameter search over the space defined in Table 8. For the final model, we use input hidden dimension D of size 256. For all models with a Mamba-2 backbone, the number of layers denotes the full birdirecti model.
19

---

Beyond Sensor Data: Foundation Models of Behavioral Data from Wcarables Improve Health Predictions
The final WBM model is a Mamba-2 backbone with TST tokenization. We use a batch size of 192 samples, hidden dimension of size 256, feed-forward hidden dimension of 1024, weight decay of 0.035, 24 backbone layers, dropout of 2.7%, layer normalization instead of RMS normalization, a À of 0.21 for KoLeo regularization, and dropping 23.3% of input tokens to create augmentations during training. The model was trained with the Adam optimizer using the settings discussed above. The final WBM model was the result of 6 epochs of training which took 16 hours of training time on 8 A100 GPUs The learned model can quickly perform inference, and embeddings can be used easily across many tasks.
Hyperparameter
Search Space
Batch Size
{128, 192]
Number of Layers
{12, 16)
Weight Decay
{0.01, 0.001}
X (Koleeo Regularization)
{0.1 .0.2)
Table Hyperparameter search for ablation experiments.
Hyperparameter
Search Space
Batch Size
(128, 192, 256)
Number of Layers
{8. 12, 16, 20, 24)
Weight Decay
[0.0001, 0.1]
X (KoLco Regularization)
[0.05, 0.4]
Dropout
[0%, 70%]
Token Drop Percentage
[10%, 70%
Normalization
(RMSNorm, LayerNomm
Table 8. Hyperparameter search for final WBM model training.
A.5.2. ABLATION RESULTS
First, we present results a validation set for the downstream task of age prediction. We report mean absolute error (MAE) for predicting age in Table 9. Most models perform similarly, with the Tuple tokenization with Mamba-2 performing the worst, However, the TST tokenization with a Mamba-2 backbone resulted in the best performance overall for this task.
Embedding
Age MAE 1
Tuple with Self-Attention 4.65
Tuple with Rotary Attention
4.29
Tuple with Mamba-2 mTAN with Self-Attention mTAN with Rotary Attention
6.21
4.51
4.39
m mTAN with Mamba-2 TST with Self-Attention TST with Rotary Attention
4.73
4.37
4.43
TST with Mamba-2
4.05
Table 9. Mean absolute error for age prediction for held-out validation task across different foundation model architectures in initial ablation study.
We present further results on the ablation of different tokenization and deep learning architectures. In Table 10, we show the number of times each combination of tokenization and deep learning model was the best performing mode for the 47 survey outcomes presented in the main paper. The relative ordering of model performance holds, with the TST with Mamba-2 model performing best. These results provide more evidence that the TST with Mamba-2 framework is the strongest paradigm for training the final foundation model.
20

---

Beyond Sensor Data: Foundation Models of Behavioral Data from Wcarables Improve Health Predictions
Embedding
Number of Outcomes with Best Performance
Tuple with Self-Attention
1/47
Tuple with Rotary Attention
6/47
Tuple with Mamba-2 mTAN with Self-Attention mTAN with Rotary Attention mTANN with Mamha-2 TST with Self-Attention TST with Rotary Attention TST with Mamba-2
0/47
1/47
2/47
0/47
10/47
2/47
25/47
Table 10. The number of survey outcomes for which each embedding was the best performing model. The relative performance of models consistent with the age prediction performance, with TST with Mamba-2 performing best.
A.5.3. MASKED AUTOENCODER RESULTS
Due to the popularity of masked autoencoder approaches. we consider training the strongest backbone model (i.e., TST tokenization with Mamba-2) using a masked autoencoder approach. We use Mamba-2 as both the encoder and decoder, and perform a similar hyperparameter search over the variables mentioned in Table 8, along with searching over the percentage of signal to mask. We ignore missing variables in the calculation of the loss and try both random masking and temporal masking, akin to past work (Narayanswamy et al., 2024). The best model across all settings achieves an MAE of 6.39 on the task of age prediction, which is substantially worse than all model strained using the contrastive loss. We hypothesize that this could be due to the restrictive nature of masked autoencoder approaches in requiring the entire input signal to be reconstructed. This could result in embeddings that focus too much on reconstructing the most observed variables (e.g.. active energy burned, step count, etc), rather than learning embeddings that learn a holistic view of an individual's 's week of data, including sparse yet informative variables (e.g. VO2max, six minute walk distance, etc). Techniques such as joint embedding predictive architectures (JEPA) (Zheng et al., 2023; Assran et al., 2023) could be an interesting choice to overcome this problem; we leave this exploration for future work.
A.6. Additional Downstream Task Details
A. TASK PREPROCESSING DETAILS
Demographics: The first downstream task is to predict participant age and biological sex (male or female). Prior work has shown that biosignals can often reliably predict demographics (Abbaspourazad et al., 2024a), and since demographics are fairly predictive of many health conditions, this offers a simple proxy task for assessing model quality. All 161,855 participants have self-reported age, and 158,347 have a self-reported biological sex of either male or female (63.4% male). We use the age prediction task find the best combination of settings in the initial ablation analysis.
the intake surveys that ask questions about participant baseline disease and medication history downstream tasks. In order to be included in the dataset for each task, a participant needs to have responded with either yes or no as to whether or not they had the disease or were taking the medication type upon enrollment; participants declining to respond or answering unsure are removed for that task. In total, we consider 47 survey questions as outcomes. The full set of exact survey questions and counts for each condition can be found in Appendix 1.1.2, while results figures show shortened version of the name for each prediction task.
Baseline disease and medications: As in Abbaspourazad al. (2024a),
Sleep metries: Apple Watch can be used to estimate stages of sleep for users who wear a watch overnight, and are derived from an accelerometer-basse algorithm (Apple, 2023). Since our foundation models were not trained using any sleep data, we use weekly summaries of sleep data (in particular, the weekly average total sleep duration, deep sleep duration, REM sleep duration, and sleep efficiency- the percent of time not in a wake state) as sleep-related downstream tasks, to assess the ability of the models to generalize to a new setting. We use data from 35.2K unique users who have sleep tracking data in AHMS. There are 67 1.66 unique participant- weeks of sleep outcomes data that we use, after limiting to weeks where at least 5 nights of sleep were recorded, and where there was sufficient Kialth data as well as at least one PPG segment.
21

---

Beyond Sensor Data: Foundation Models of Behavioral Data from Wcarables Improve Health Predictions
Pregnancy: It is well known that many health metrics change during pregnancy, some of which are directly measurable by wearables, such resting heart rate and HRV (Mishra et al., 2024). Hence, prediction of whether not week of data is from during period when a participant is pregnant is health state that should be detectable by our models. To create a pregnancy dataset, we limit to the 430 pregnancies ending in vaginal or cesarean delivery from 385 unique participants, among participants with adequate HealthKit and PPG data. Weeks in the 9 months hefore the delivery date and month after are categorized as "positive", in the sense that these weeks may show physiological changes for these participants, and all other periods of time are "negative". We further include 24,225 female participants younger than 50 who have reasonable coverage in their health surveys as non-pregnant controls.
Infection: Since substantial prior work has shown that wearables can detect influenza-like illness and COVID-19 (Nestor et al., 2023), we we also use this as downstream task. We predict whether a week of time series data is from within I week before to 4 weeks after self-reporting a respiratory infection, such the common cold, flu, or COVID-19. Other weeks not in proximity to a reported respiratory infection are treated as negatives. We only include weeks of data from participants who reliably fill out the quarterly/monthly health questionnaires. The task dataset includes 9689 self-reported new respiratory conditions (c.g. cold, COVID-19, flu, pneumonia) that developed in 8585 unique participants. We also create additional negative controls from participants who reliably fill out surveys but never self-report a respiratory infection, yielding an additional 83K participants all labeled as negative.
AIC/ / Diabetes Status: To assess how well our models encode information about metabolic health, we predict diabetes status using HbAle, laboratory value typically used for diabetes diagnosis. These values are ascertained from clinical health records, so should be higher quality than the self-reported labels of having diabetes on enrollment into the study, There are 17,984 unique HbAlc labs from 9,847 unique participants, with 1 116.5K weeks of data collected within 30 days of each lab value that we use to make predictions. The prediction task is to classify a participant as either having diabetes or not, based upon the standard threshold of 2 6.5% for the HbAle lc value. In our dataset, 53.2% of labs were normal (< 5.7%), 26.5% were prediabetes (2 5.7% and < 6.5%), and 20.3% were diabetes (2 6.5%).
Injury: A task that we would expect WBM to outperform PPG on is predicting metrics related to changes in mobility, such as whether a participant recently self-reported an accident or injury limiting their mobility. We would expect such events to affect mobility metries captured in activity data, but may not be manifested in a biosignal like PPG. We create a dataset of 26K self-reported injuries that happened during the study to 17.7K unique individuals. We label the week of the injury and the 4 weeks after as positive weeks, omit weeks 5- 16 after injury (in the event that the person recovers quickly, to avoid the potential for ambiguous labels), and otherwise label all other weeks from these participants as negatives. We also include as negative controls data from 75.7K other participants who have adequate data availability and survey coverage but never self-report an injury, with all of their data labeled as negative.
A.6.2. ADDITIONAL RESULTS
Reconstructing Input Signals. To further understand the information encoded in WBM, we probe the embeddings to predict the weekly mean of each input activity variable (Table 11). Specifically, we are interested in understanding whether the embeddings can only reconstruct the highly observed input variables. However, the results show that the embeddings are predictive of a majority of input variables, including irregularly sampled variables such as stair ascent and descent speeds, walking heart rates, and VO2 max. Mcanwhile, active energy burned, frequently measured variable, is unable to be constructed using the embeddings, providing evidence against the hypothesis that the embeddings simply capture the most frequently observed variables. This is likely an artifact of our contrastive pre-training objective different pre-training paradigm such masked autoencoding would likely be better able reconstruct such frequently observed variable. We hypothesize that since active energy burned tends to correlate highly with heart rate, the contrastively trained model simply lears to discard it. Moreover, we hypothesize that we may be able to remove some of these features that do not contribute to the final representation during the training process to get smaller inputs and gain computational efficiency.
22

---

Beyond Sensor Data: Foundation Models of Behavioral Data from Wcarables Improve Health Predictions
Activity Metric
Performance
Resting heart rate, mean
0.942
Heart rate, mean
0.938
VO2max, mean
0.929
Walking double support percentage, mean
0.909
Body mass, mean Walking HR, mean
0.900
0.888
Stand time, mean HRV (SDNN), mean Walking step length, mean Walking speed, mean Step count (watch), mean Respiratory rate, mean Walking Steadiness, mean
0.872
0.868
0.866
0.849
0.847
0.840
0.810
Six minute walk test distance, mean
0.796
Step count (phone), mean Stair descent speed, mean Exercise minutes, mean Sleeping wrist temp, mean Stair ascent speed, mean Walking asymmetry percentage, mean
0.793
0.752
0.746
0.612
0.563
0.533
Flights climbed (watch), mean
0.418
Basal energy burned, mean
0.347
Flights climbed (phone), mean Number of times fallen, mean
0.341
0.096
Active energy burned, mean Body mass index, mean Oxygen saturation, mean
0.011
0.000
0.000
Table 11. R² performance of WBM embeddings for predicting the weekly mean of all activity variables. We subset this analysis to only weeks where given HK variable actually observed, and so has a mean value, in order to ignore artifacts from imputed values skewing the results. Our results show that even irregularly sampled activity variables are captured within the embeddings.
Results Across Demographic Subgroups. We report results of trained models across demographic subgroups represen- tative tasks both at subject-level and segment-level. We report performance across different racc/ethnicity categories, age subgroups, and biological sex. For race/ethnicity categories, we note that many subjects may fall into multiple race/ethnicity categories as subjects can self-report belonging to more than one. These subjects will be considered in multiple evaluations.
For subject-level tasks, we report performance for heart failure, calcium-channed blockers, and smoking status. Results show that most models perform well across different demographic subgroups besides the baseline model. However, this is be expected the baseline this demographic information as input the model. Hence, stratifying on these groups results in poor discriminative performance. One exception the strong performance across subgroups is the WBM model the heart failure outcome. Both models using the WBM embeddings (WBM and WBM + PPG) see poor performance compared to the PPG model, likely indicating that these demographics are encoded and being used in the final learned model.
For intra-subject tasks, we report performance for predicting respiratory infections and pregnancy. Again, we see that most models generally perform well across demographic subgroups except for the baseline model. One notable exception is on predicting pregnancy within Black patients, where the baseline model excels.
23

---

Beyond Sensor Data: Foundation Models of Behavioral Data from Wcarables Improve Health Predictions
Age Group
Baseline
WBM
PPG
WBM PPG
<= 33 years (N ( %); 8922 (0.33))
0.560 (0.468, 0.643)
0,618 (0.519, 0.715)
0.751 (0.650, 0.840)
0.737 (0.639, 0,830)
34 46 years IN (%); 9264 (0.63))
0.569 (0.508, 0.634)
0.763 (0. 700, 0.821)
0.855 (0.797, 0.908)
0.842 (0.781, 0.897)
> 47 years( N (%): 8865 (1.91))
0.593 (0.554, 0.634)
0.815 (0.786, 0.845)
0.889 (0.862, 0.914)
0.884 (0.857, 0.911)
Table 12. AUROC performance on heart failure across different age subgroups.
Ethnicity
Baseline
WBM
PPG
WBM PPG
American Indian (N (%)) 599 (1.50)
0.592 (0.424, 0.756)
0.799 (0.631, 0.943)
0.760 (0.555, 0.941)
0.780 (0.586, 0.941)
Asian (N (%): 1702 (0.41))
0,450 (0.201, 0.688)
0.768 (0.540, 0.988)
0.857 (0.721, 0.986)
0.828 (0.653, 0.982)
Black (N (%): 1483 (1 (1.21))
0,562 (0.438, 0.679)
0.773 680, 0.868)
0.818 (0.696, 0.920)
0.811 (0.683, 0,919)
Hispamic (N (%): 2886 (0.80)
0,611 (0,511, 0.710)
0.734 (0.589, 0.858)
0.903 (0.822, 0.967)
0.900 (0.812, 0.966)
Middle Eastern (N (%): 315 (1.27))
0.235 (0.067, 0.387)
0.775 (0.559, 0.958)
0.797 (0.560, 0.920)
0.727 (0.405, 0.906)
White (N (%): 21749 (1.02))
0.624 (0.592, 0.655)
0.812 (0.784, 0.842)
0.879 (0.849, 0.906)
0.873 (0.842, 0.900)
Table 13, AUROC performance on heart failure across different racc/cthnicity subgroups.
Biological Sex
Baseline
WBM
PPG
WBM + PPG
Male (N (%); 16370 (0.95))
0.632 (0.590, 0.672)
0.830 (0. 790, 0.864)
0.885 (0.849, 0.918)
0.884 (0,849, 0,917)
Female (N (%) 10102 (0.92))
0.597 (0.544, 0.647)
0.802 (0.758, 0.842)
0.881 (0.846, 0.912)
0.879 (0.843, 0.910)
Table 14. AUROC performance on heart failure across different biological sex subgroups.
Age Group
Baseline
WBM
PPG
WBM + PPG
<= 35 years ( N (%): 5082 (1.75))
0.460 (0.400, 0.524)
0.682 (0.624, 0.742)
0.659 (0.599, 0.717)
0.656 (0.596, 0.719)
36 49 years NN (%): 5307 (5.88))
0.492 (0.460, 0.525)
0.749 (0.720, 0.776)
0.732 (0.702, 0.761)
0.735 (0. (0.706, 0.765)
> 50 years N (%): 5288 (13.54))
0.485 (0.463, 0.507)
0.715 (0.695, 0.734)
0.727 (0.707, 0.747)
0.733 (0.714, 0.752)
Table 15. AUROC performance on calcium-channel blockers across different age subgroups.
Ethnicity
Baseline
WBM
PPG
WBM + PPG
American Indian (N (%) 348 (10.34)
0.614 (0.523, 0.701)
0.825 (0.753, 0.884)
0.837 (0.762, 0.903)
0.827 (0.750, 0.895)
Asian (N (%): 712 (7.16))
0.513 (0.435. 0.597)
0.675 (0.598, 0.748)
0.775 (0.709. 0.840)
0.784 (0.722, 0.847)
Black (N (%): 754 (13.40)
0.612 (0.558. 0.671)
0.763 (0.708, 0.815)
0.788 (0.740. 0.832)
0.794 (0.745, 0.840)
Hispanic (N (%) 1357 (5.90))
0.494 (0.433. 0.548)
0.786 (0.731, 0.837)
0.809 (0.753, 0.860)
0.786 (0.727, 0.846)
Middle Eastern (N (%); 163 (6.13))
0.597 (0,467, 0.718)
0.731 (0.509, 0.922)
0.780 (0.574, 0.948)
0.765 (0.541, 0.962)
White (N (%): 13262 (6.83))
0.523 (0.504, 0.541)
0.796 (0.780, 0.810)
0.782 (0.765, 0,800)
0.778 (0.760, 0.796)
Table 16. AUROC performance ciccuum-channell blockers across different race/ethnieity subgroups.
Biological Sex
Baseline
WBM
PPG
WBM + PPG
Male (N (%); 8607 (9.00))
0.521 (0.502, 0.542)
0.770 (0.753, 0.788)
0.786 (0. (6.769, 0.804)
0.791 (0.774, 0.809) 0.789 (0.761, 0.817)
Female (N (%) 6717 (4.69))
0.510 (8.481, 0.542)
0.796 (0.771, 0.823)
0.791 (0.762, 0.820)
Table 17. AUROC performance on calcium-channee blockers across different biological sex subgroups.
Age Group
Baseline
WBM
PPG
WBM + PPG
<= 34 years (N (%): 8253 (7.22))
0.636 (0.614, 0.658)
0.803 (0.785, 0.821)
0.789 (0.770, 0,811)
0.795 (0.776, 0.816)
35.0-46.0 IN (%): 7473 (8.71)
0.654 (0.632, 0.675)
0.824 (0.806, 0.840)
0,809 (0.788, 0,828)
0.816 (0.794, 0,834)
> 47 years IN (%); 8109 (4.72)
0.695 (0.669, 0.719)
0.793 (0.767, 0.818)
0.808 (0. 0.834)
0.813 (0.787, 0.838)
Table 18. AUROC performance on smoker status across different age subgroups.
24

---

Beyond Sensor Data: Foundation Models of Behavioral Data from Wcarables Improve Health Predictions
Ethnicity
Bascline
WBM
PPG
WBM + PPG
American Indian (N (%) 511 (13.31))
0.623 (0.560, 0.688)
0.819 (0.771, 0.864)
0.834 (0.786. 0,883)
0.846 (0.799, 0.892)
Asian (N (%): 1425 (6.95))
0,649 (0.593. 0.705)
0.809 (0.762, 0.850)
0.717 (0.659. 0.779)
0.732 (0.675, 0.791)
Black (N (%): 1248 (8.41))
0.536 (0.479, 0.597)
0.793 (0.749, 0.837)
0.807 (0.759, 0.850)
0.823 (0.776, 0.865)
Hispanic (N (%): 2431 (6.83))
0.614 (0.568, 0.661)
0.780 (0.741, 0.819)
0.740 (0.698, 0.785)
0.759 (0.718, 0.804)
Middle Eastern (N (%); 279 (11.11)
0.666 (0,561, 0.774)
0.773 (0.691, 0.843)
0.827 (0. 756, 0.896)
0.838 (0.771, 0.901)
White (N (%): 19385 (6.61))
0,674 (0.659, 0.688)
0.816 (0.804, 0.828)
0.804 (0.790, 0.817)
0.822 (0.809, 0.834)
Table 19. AUROC performance smoker status across different race/ethnicity subgroups.
Biological Sex
Baseline
WBM
PPG
WBM + PPG
Male (N (%): 14464 (6.34))
0.632 (0.614, 0.651)
0.802 (0.786, 0.817)
0.774 (0.757, 0.791)
0.783 (0.766, 0.800)
Female (N (%)): 8863 (7.71)
0.699 (0.681, 0.719)
0.841 (0.826, 0.857)
0.827 (0.809, 0.844)
0.856 (0.840, 0.872)
Table 20. AUROC performance on smoker status across different biological sex subgroups.
Ethnicity
Baseline
WBM
PPG
WBM + PPG
American Indian (N (%); 14,359 (1.1))
0.678 (0.637, 0.712)
0.717 (0.678, 0.754)
0.692 (0.657, 0.733)
0.702 (0.660, 0.738)
Asian (N (%): 39,501 (0.7))
0,567 (0.533, 0,604)
0.766 (0.743, 0.790)
0,721 (0.693, 0,750)
0.756 (0.731, 0.783)
Black (N (%): 31,822 (0.9))
0.629 (0.603, 0.657)
0.645 (0,616, 0.675)
0.637 (0.607, 0.675)
0.633 (0.600, 0.664)
Hispanic (N (%): 67, (0.9))
0.572 (0.552, 0.597)
0.677 (0.658, 0.696)
0.697 (0.677, 0.718)
0.690 (0.674, 0.712)
Middle Eastern (N (%): 7,550 (0.6))
0.663 (0.574, 0.754)
0.586 (0.527, 0.655)
0.673 (0.607, 0.738)
0.652 (0.590, 0.739)
White (N (%); 666,408 (0.9))
0.628 (0.621, 0.634)
0.749 (0.743, 0.756)
0.729 (0.723, 0.735)
0.762 (0.756, 0.768)
Table 21. Test-set AUROC performance for predicting respiratory infections across different race/ethnicity subgroups. The sample sizes presented are per-week.
Biological Sex
Baseline
WBM
PPG
WBM PPG
Male (N (%): 491,031 (0.8))
0.627 (0.619, 0.635)
0.759 (0.753, 0.766)
0.745 (0.739, 0.752)
0.776 (0.770, 0.783) 0.727 (0.719, 0.735)
Female (N (%): 278,711 (1.1))
0.630 (0.621, 0.638)
0.725 (0.716, 0.733)
0.692 (0.684, 0.699)
Table 22, Test-set AUROC performance for predicting respiratory infections across different hiological sex subgroups. The sample sizes presented are per-week.
Age
Baseline
WBM
PPG
WBM + PPG
<= 37 years (N (%) 269,084 (0.8))
0.615 (0.604, 0.628)
0.738 (0.727, 0.747)
0.707 (0.698, 0.718)
0.741 (0.731, 0.751) 0.757 (0.748, 0.765) 0.760 (0.752, 0,771)
38 51 years (N (%); 262.574 (1.0))
0.641 (0.629, 0.651)
0.752 (0.742, 0.760)
0.728 (0.719, 0.737)
> -22 years (N (%): 255,629 (1.0))
0.633 (0.622, 0.643)
0.749 (0.740, 0.759)
0.724 (0.716, 0.734)
Table 23. Test-sct AUROC performance for predicting respiratory infections across different age subgroups. The sample sizes presented are per-weck.
Ethnicity
Baseline
WBM
PPG
WBM + PPG
Asian (N (%)): 7,023 (I.7))
0.582 (0.543, 0.622)
0.566 (0.506, 0.628)
0.763 (0.721, 0.800)
0.737 (0.685, 0.786)
Black (N (%) 9,897 (0.5))
0.823 (0.785, 0.852)
0.764 (0.690, 0.819)
0.532 (0.459, 0.603)
0.713 (0.643, 0.776)
Hispanic (N (%): 17,718 (1.2))
0.687 (0.651, 0.724)
0.716 (0.683, 0.753)
0.790 (0.757, 0.823)
0.841 (0.807, 0.870)
White (N (%): 153,470 (1.1))
0.817 (0.808, 0.828)
0.879 (0.870, 0.887)
0.879 (0.870, 0.888)
0.926 (0.919, 0.933)
Table 24. AUROC performance for predicting pregnancy across different racc/ethnicity subgroups. Due extremely small sample sizes, not all subgroups were reported.
Age
Baseline
WBM
PPG
WBM + PPG
<= 29 years (N (%): 50,898 (1.6))
0.774 (0.756, 0.789)
0.839 (0. 825, 0.853)
0.858 (0.843, 0,875)
0.895 (0.882, 0.910) 0.883 (0.869, 0.898) 0.925 (0.909, 0.939)
29 33 years (N (%); 26.479 (2.5))
0.767 (0.751, 0.786)
0.812 79,, 0.828)
0.827 (0.808, 0.844)
> 33 years (N (%): 102,840 (0.4))
0.884 (0.870, 0.895)
0.867 (0.848, 0.888)
0.862 (0.845, 0.878)
Table 25. AUROC performance for predicting pregnancy across different age subgroups.
25

---

Beyond Sensor Data: Foundation Models of Behavioral Data from Wcarables Improve Health Predictions
Name
Baseline
WBM
PPG
WBM + PPG
ACE-inhibitors
0.792 (0.783, 0.800)
0.799 (0.791, 0.808)
0.825 (0.817, 0.833)
0.829 (0.822, 0.837)
Active smoker
0.777 (0.765, 76,76,, 0.789)
0.843 (0.833, 0.853)
0.859 (0.848, 0.868)
0.881 (0.871, 0.890)
Afib
0.799 (0.781, 0.814)
0.847 (0.831, 0.862)
0.826 (0.809, 0.842)
0.860 (0.844, 0.874)
Allergy
0.663 (0.656, 0.670)
0.668 (0.661, 0.675)
0.665 (0.659, 0.672)
0.677 (0.670, 0.684)
Anti-anxiety
0.693 (0.682, 0.704)
0.729 (0.719, 0.739)
0.748 (0.737, 0.758)
0.761 (0.752, 0.772)
Anti-psychotic
0.795 (0.774, 0.814)
0.821 (0.803, 0.839)
0.822 (0.803, 0.841)
0.832 (0.813, 0.851)
Anticoagulants
0.710 (0.702, 0.718)
0.768 (0.761, 0.776)
0.856 (0.850, 0.863)
0.866 (0.860, 0.872)
Antidepressants
0.787 (0.776, 0.800)
0.793 (0.781, 0,805)
0.794 (0.780, 0.806)
0,800 (0.788, 0.812)
Antiplatelets
0.767 (0.749, 0.783)
0.830 (0. 814, 0.845)
0.855 (0.839, 0.868)
0.873 (0.858, 0.885)
Auxiety
0.754 (0.748, 0.760)
0.785 (0.779, 0.790)
0.806 (0.800, 0.811)
0,822 (0.816, 0,827)
Artery discase
0.886 (0.863, 0.908)
0.882 (0.852, 0.908)
0.881 (0.848, 0.909)
0.872 (0.840, 0.901)
Arthritis
0.790 (0.783, 783, 0.797)
0.794 (0. 7877 0,800)
0.787 (0.780, 0.793)
0,801 (0.794, 0,808)
Asthma
0.621 (0.612, 0.628)
0.623 (0.614, 0.630)
0.648 (0.640, 0.655)
0.649 (0.641, 0.657)
Beta-blockers
0.800 (0. 790, 0.811)
0.825 (0.815, 0.835)
0.789 (0.778, 0.800)
0.844 (0.834, 0.854)
Blood pressure
0.787 (0. 780, 0.793)
0.799 (0. 793, 0.805)
0.823 (0.818, 0.829)
0.832 (0.826, 0.838)
Blood pressure med.
0.703 (0.683, 0.722)
0.720 (0.699, 0.738)
0.744 (0.725, 0.762)
0.744 (0.725, 0.761)
Calnium-channel Blockers
0.793 780, 0.806)
0.804 (0.790, 0.817)
0.825 (0.812, 0.839)
0.858 (0.846, 0.869)
Cancer
0.805 (0.793, 0.817)
0.792 (0.779, 0,804)
0.804 (0.791, 0.816)
0.800 (0.788, 0,812)
Chemotherapy
0.742 (0.673, 0.797)
0.702 (0.633, 0.767)
0.717 (0.641, 0.784)
0.698 (0.621, 0.768)
Cholesterol
0.753 (0.747, 0.759)
0.758 (0.752, 0,765)
0.768 (0.762, 0.775)
0.774 (0.767, 0.780)
Chronic bronchitis
0.730 (0.713, 0.746)
0.750 (0.735, 0.766)
0.748 (0.733, 0.764)
0.759 (0.743, 0.774)
Depression
0.728 (0.722, 0.735)
0.764 (0.758, 0,770)
0.781 (0.775, 0.786)
0.799 (0.794, 0.805)
Diabetes
0.814 (0.803, 0.824)
0.826 (0.816, 0.836)
0.866 (0.855, 0.875)
0.872 (0.862, 0.881)
Diuretics
0.735 (0.721, 0.748)
0.748 (0.735, 0.761)
0.771 (0.758, 0.782)
0.775 (0.762, 0.787)
Hearing
0.728 (0.720, 0.736)
0.725 (0.716, 0.734)
0.723 (0.715, 0.732)
0.734 (0.725, 0.742)
Heart attack
0.846 (0.825, 0.865)
0.858 (0.838, 0.876)
0.846 (0.825, 0.865)
0.866 (0.845, 0.885)
Heart disease
0,858 (0.842, 0.874)
0.853 (0.837, 0,870)
0.860 (0.844, 0.877)
0.867 (0.849, 0.884)
Heart failure
0.824 (0.794, 0.850)
0.858 (0.832, 0.881)
0.865 (0.835, 0.890)
0.889 (0.864, 0.910)
Heart thythm
0.664 (0.653, 0,676)
0.682 (0.670. 0.694)
0,689 (0.677, 0.700)
0.703 (0.692, 0.715)
Hip/Knee
0.855 (0.840, 0.870)
0.842 (0.825, 0.859)
0.850 (0.835, 0.866)
0.852 (0.836, 0.869)
Kidney
0.698 (0,682, 0.713)
0.708 (0.693. 0.723)
0.707 (0.692, 0.722)
0.714 (0.699, 0.729)
Liver
0.723 (0.675, 0.767)
0.749 (0.698, 0.796)
0.735 (0.691, 0.782)
0.756 (0.709, 0,801)
Lower back
0.690 (0.681, 0.699)
0.700 (0.691, 0.709)
0.696 (0.687, 0.704)
0.710 (0.701, 0.719)
Neck disorder
0.732 (0. 721, 0.742)
0.738 (0.727, 0.749)
0.736 (0.725, 0.748)
0.747 (0.735, 0.758)
Neuropathy
0.794 (0.780, 0.807)
0.818 (0.805, 0.830)
0.816 (0.803, 0.829)
0.829 (0.815, 0.841)
Opioid
0.753 (0. 733,, 0.773)
0.794 (0.774, 0.813)
0.799 (0.780, 0.819)
0.820 80.8 0.839)
On any medication
0.735 (0.729, 0.742)
0.763 (0.757, 0.769)
0.796 (0.790, 0.801)
0,806 (0.801, 0,811)
Osteoporosis
0,872 (0.860, 0.885)
0.862 (0.847, 0.876)
0.862 (0.846, 0.877)
0.867 (0.851, 0.881)
Pacemaker
0.821 (0.782, 0.857)
0.858 (0.823, 0.891)
0.897 (0.859, 0.931)
0,899 (0.860, 0,933)
Pain
0.721 (0. 703, 0.739)
0.728 (0.711, 0.746)
0.718 (0.702. 0.735)
0.737 (0.721, 0.756)
Painkillers
0.605 (0.596, 0.614)
0.609 (O.600, 0.618)
0.609 (0.600, 0.619)
0.618 (0.609, 0.627)
Sleep apnea
0.791 (0. 784, 0.798)
0.802 (0. 795, 0.809)
0.823 (0.815, 0.830)
0.830 (0.823, 0.837)
Sleep medication
0.642 (0.626, 0.655)
0.689 (0.674, 0.702)
0.700 (0.684, 0.714)
0.718 (0.703, 0.731)
Stroke or TIA
0.767 (0. 7424,, 0.790)
0.789 (0.765, 0.812)
0.788 (0.764, 0.811)
0.795 (0.770, 0,817)
Thyroid
0.757 (0.746, 0.768)
0.754 (0.743, 0.765)
0.756 (0.745, 0.766)
0.761 (0.750, 0.772)
Urinary
0.805 (03793, 0.818)
0.814 (0.802, 0.827)
0.804 (0.790, 0.817)
0.813 (0.800, 0.826)
Vision
0.660 (0.652, 0.667)
0.664 (0.657, 0.672)
0.661 (0.653, 0.668)
0.669 (0.662, 0.677)
Table 26. AUROC performance metrics of different models across all baseline discase and medication tasks. The combination of both modalities results in the best performance for most tasks. Bold represents the best model(s) for each label, whereas italics represents results that are significantly better than the other approaches (p < .05)
26


Reviews:
[{"review_id": "g3bReL8HQJ", "reviewer": "Reviewer_Ge7u", "summary": "This paper proposes WBM, a foundation model trained on wearables dataset to improve health predictions. The paper states that behavioral signals including physical activity and mobility metrics align better with physiologically relevant timescales than raw sensor data. The proposed model is trained on over 2.5 billion hours of data from 162k individuals, and is evaluated across 57 health-related tasks. The results suggest that the proposed model has improved performance on behavior-driven tasks like sleep prediction compared to existing models based on raw sensor data.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive comments and constructive feedback to help us improve this work!  We focus on responding to the major themes of your comments:\n\n**Choice of Mamba-2 and comparison to SOTA deep learning models:**\n\nThis is an important point to clarify. As stated in Sections 4.2 and 4.3 and Appendix Section A.5.1, we compared the Mamba-2 model architecture with two alternative architectures: Self-attention Transformer, and Rotary Transformer. These two are among the most commonly used state-of-the-art Transformer architectures in various domains. In addition, for a fair comparison, we did a full grid search over these 3 architectures, and over 3 different tokenizers, as well as sweeping other hyperparameters. Within our hyperparameter search experiment, we observed that Mamba-2 with the TST tokenizer generally outperformed the other alternatives, including the 2 Transformers architectures.  Refer to Table 9 in Appendix Section A.5.2 for a full comparison of how often TST+Mamba-2 achieved the best performance compared to the other 8 architecture+tokenizer combinations.\n\n**Inclusion of more baseline models would improve the rigor of the evaluation:**\n\nWe have included comparisons of WBM with baseline architectures/tokenizers, and a competitive PPG baseline. In addition, we have discussed a baseline comparison with respect to other pre-training methods such as masked autoencoding (Narayanswamy 2024) in Appendix A.5.3, as well as a simple baseline of turning behavioral data to its mean and standard deviation statistics. However, we do agree with that including more baselines can improve our evaluations, and we will discuss this as a caveat in the camera ready version. We welcome any feedback on specific baselines that you feel would particularly strengthen our work.\n\n**Potential demographic biases and performance in demographic subgroups:**\n\nWe agree that characterizing demographic biases are essential for health applications. The Apple Heart and Movement Study has its own limitation and biases (Truslow et al. 2024), for example, the dataset is more biased towards a younger male population. However, given the large scale of this study compared to other studies, our models are still trained and evaluated on a large cohort including participants from diverse demographics. In the camera ready version of the paper, we will include distributions of demographics statistics from our pre-training data. We will also add to the appendix the performance of our models within demographic subgroups on a representative subset of the full set of tasks considered, focusing on tasks where the combination of WBM+PPG performs best.  Specifically, we will show demographic subgroup performance on a representative set of targets: heart failure, active smoker, and calcium-channel blocker baseline tasks, as well as on the pregnancy and infection tasks.  We will add some discussion around potential fairness concerns and demographic biases to the camera ready version of the paper, noting that a complete fairness investigation into our final models was out of scope for this work.", "claims_and_evidence": "The claims made in the submission such as behavioral data provide valuable insights into health conditions beyond raw sensor data, and that the proposed model outperforms baselines in various health detection tasks is supported by experimental results. However, the reason that Mamba-2 is chosen to be the backbone for behavioral data modeling is not stated, nor is it rigorously compared against other deep learning architectures.", "methods_and_evaluation_criteria": "The methods are well-described in Sections 3-5, including dataset preprocessing, model architecture, and evaluation metrics. Evaluation is performed on a broad set of tasks. Inclusion of more baseline models would improve the rigor of the evaluation.", "theoretical_claims": "No theoretical claims in this paper. The parameter details in appendix look solid.", "experimental_designs_or_analyses": "The experiments provide reasonable baselines and comparisons. The evaluation on 57 downstream tasks is impressive, but some analyses lack deeper breakdowns on task-specific performance.", "relation_to_broader_scientific_literature": "The work contributes to wearable-based health monitoring and references important works in this field.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Major contributions include systematic evaluation across 57 health tasks with large datasets. Clear description of model architecture and experiments. Comparison to SOTA deep learning models beyond Mamba-2 is limited.", "other_comments_or_suggestions": "The paper is well-written and has clear and meaningful visuals.", "questions_for_authors": "How does WBM compare against state-of-the-art transformer models?\nHave you analyzed potential biases in the dataset, particularly in terms of demographic representation? How does the model's performance vary across different demographic subgroups?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I9ifC68Awd", "reviewer": "Reviewer_zmiU", "summary": "This manuscript considers the problem of health condition tracking using pretrained foundation models trained on the Apple health movement dataset. In contrast to past work that used raw sensor signals from PPG and ECG, they leverage higher-level ‘behavioral’ metrics that are extracted from IMU (eg steps), user input (BMI) or intermittent sampling (VO2Max). They survey several architectures, noting special challenges in irregularly sampled data. Following network architecture comparison, they used a dense matrix of features per hour that were passed through bidirectional mamba2 and trained using a contrastive loss with pairs of users as positive samples. They examine demographic classifications tasks, inter-subject tasks predicting health states, and intra-subject classification of demographic information like age and biological sex, showing impressive performance that is fairly competitive with PPG on most tasks done using linear probing. They also examine combinations with PPG, and discuss discrepancies.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback and useful comments for enhancing our work. We respond to specific suggestions below: \n\n**Interpretability of WBM**\n\nInterpretability of foundation models is an active area of research that remains extremely important. Unfortunately, it remains non-trivial to understand how input features affect the learned representation in order to ascertain feature importance for any given downstream task. As you suggest, one technique might involve independently perturbing each input sensor to understand its effect on the learned representation. However, understanding the correct way to perturb these irregular data in a meaningful and scalable way remains an important open problem. We will discuss the importance of interpretability and some potential next steps in the camera ready version of the paper. \n\n**Sleep Metrics and IMU**\n\nWhen evaluating using sleep metric labels, our goal was to showcase one example where we expect behavior data to be much more predictive than PPG. Sleep metrics are only estimated when a subject wears their watch overnight, ensuring that we have some amount of PPG overnight alongside the other behavior data fields. In general, a passive measurement of PPG is attempted roughly every 2 hours for most subjects, and the measurement is only retained if the subject is sufficiently quiescent ensuring that the PPG data has low noise.\n\nThe sleep metrics on the Apple Watch are derived only from a continuous stream of IMU (3-axis accelerometer) during a sleep session, and PPG/behavior is not used.  Processing such continuous IMU streams involves the use of complex data pipelines that were not available to us; the volume of such data would make it impossible to scale to using most days and subjects from across the study.  Therefore, including such IMU data in our modeling was out of scope for our study. However, given that sleep labels are derived from IMU, we expect markedly stronger predictions if we include IMU in the input of our models. \n\n**PPG prediction of deep sleep & \"PPG will not provide the same holistic view of an individual’s week\" phrasing**\n\nIt is generally the case that total sleep duration, sleep efficiency, and deep sleep in particular decrease with age.  Since PPG (collected roughly every 2 hours - see above) contains strong age-related signals, we would expect that it should be able to leverage such information to make decent predictions about average sleep metrics for an individual.  Note that the baseline model (which explicitly includes demographics) also performs better on deep sleep prediction, suggesting that demographics plays a more important role.  We will rephrase this to “PPG does not provide as comprehensive a view of an individual’s week, since it is only measured a few times each day”. We will also add additional clarification and caveats around the sleep analyses.\n\n**Improvements on Contrastive Learning Framework**\n\nWe agree that the contrastive framework could be improved upon. We explored the use of a masked auto encoder approach, but found that this resulted in poor performance (see Appendix A.5.3). We hypothesize that this may be due to the high degree of noise and irregularity in the behavior data, making complete reconstruction of the input an overly challenging task that leads to representations that do not generalize well to new tasks. We will expand on this hypothesis further in the camera ready, as well as discuss other techniques that future work could consider adapting to this type of data to improve upon our framework such as joint-embedding predictive architectures (JEPA).  Even though our contrastive learning approach is not set up to capture intra-subject changes, empirically based on our results it still has some ability to do so.\n\n**Clarifying “Human Behavior” Throughout**\n\nThank you for finding this unclear sentence, we will rephrase in the camera ready.  We agree that the use of the term “behavior” may be confusing — we will be sure to carefully go through the manuscript and only use behavior in the intended use-case (i.e., when discussing behavior data) and avoid overloading the term. \n\n**Details on Statistical Significance and Bootstrap Performance**\n\nThank you for the great suggestion for including bootstrap CIs and p-values in the manuscript, we will add these in the camera ready format. We will also clarify how we calculate bootstrap confidence intervals: we resample the test set 1,000 times and recompute performance metrics on each resampled test set for each method. The confidence intervals and p-values are then computed empirically on this bootstrapped set of performance metrics.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "Appropriate work is cited and related, see below.", "essential_references_not_discussed": "THe work is well contextualized.", "other_strengths_and_weaknesses": "**Strengths:**\n\n- The manuscript is polished and clear with a full description and documentaiton of experimental details and methods and clear interpretation of data\n- There are essential baselines included (including a null baseline and fairly SOTA PPG), multiple interesting tasks considered, and ablations.\n- The approach provides fairly large advantages over a baseline in many tasks.\n\n**Weakness**\n\nOverall I think the manuscript is a strong accept, but I am offering some directions that would improve its utility to myself and the field. \n\n- Scaling laws of performance with amount of pretraining data would be helpful.\n- The importance of the 27 features presented is unclear, and they vary quite a bit in their missingness. It would help to delineate which were the most important for the prediction. The R2 from model reconstruction is a start, but not a full interpretability analysis.\n- Clarifying the statistical significance of results. Comparisons in Figure 3 for instance are presented without error bars, and throughout the differences in models are so small that significance should be conveyed. It is also unclear to me how the bootstrap was calculated specifically.\n- For the sleep metrics in particular, my understanding is these are trained from an algorithm consuming the same information as behavior/PPG, except perhaps a raw version of the IMU. Can you comment on how including raw IMU would impact these results. This sensor stream is conspicuously absent from all of these papers", "other_comments_or_suggestions": "- L172 “Driven by our goal of detecting health states at\na temporal resolution of human behavior”  unclear what behavior means here. Behavior is overloaded throughout the manuscript.\n\n\n- As the authors point out, contrastive pairs from the same user don’t necessarily make the most sense in this task, especially for intra-subject tests like sleep staging. Are there other tasks that might be useful they can propose.", "questions_for_authors": "Minor\n\n- L365 “PPG will not provide the same holistic view of an individual’s\nweek, since it is only captured opportunistically a few\ntimes each day” How frequently is this captured? If so, why is the PPG readout of deep sleep so good while normal sleep so bad.\n\n\n- Statsig of different comparisons e.g. classifier performance vs PPG. Should be listed per comparisons.\n- Ablations over feature importance. Given the uneven coverage, would be valuable to inform simplest set of features to construct. May obviate need for masking.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WkLV2YSjYb", "reviewer": "Reviewer_XGWG", "summary": "This paper develops a foundation model for behavioural data from wearables to improve health predictions. The authors process 2.5 billion hours of wearable data and comparing different tokenization strategies and model architectures. They find a Mamba-2 architecture with TST tokenization performs best. The model is tested on 57 health-related tasks including demographic prediction, disease classification and health state detection. Results show that the behavioural foundation model outperforms a statistical baseline on most tasks. The authors also compare WBM to a PPG foundation model, finding WBM performs better on some behavior-driven tasks like sleep prediction, while PPG excels at others. Combining WBM and PPG embeddings yields the best performance across most tasks, indicating complementary information between behavioral data and raw sensor signals.\n\n## Update after rebuttal \n\nThe authors acknowledged most of my concerns. There were two points I added further clarification on A) the ways in which the signals are different for PPG and WBM B) the label leakage. I am well aware that these are monumentally difficult to correct for but they nonetheless affect the results interpretation. A demographic classification model trained using pure physiological signals is not the same claim as a using physiological signals + demographic label conditioning, even if the two models perform to identical levels. However, these limitations are not reasons for rejections and I improved my recommendation 3 to 4 and do urge the authors to discuss these limitations, as they have done in their most recent response, in the paper.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback and suggestions aimed towards improving our work!\n\n**Contextualizing Comparisons between WBM and Baseline/PPG**\n\nWe appreciate your feedback on tempering our claims. \n\nFirst, we clarify the WBM vs baseline comparison. The subject-level tasks in Figure 3 are intentionally simple, as they aggregate a subject’s full history. Basic aggregate feature statistics and demographics may perform well, explaining the small median improvement of WBM.  However, WBM significantly outperforms the baseline in a few key cases (e.g. smoking status and anti-psychotics usage). Its real strength lies in the more difficult time-varying tasks, where it consistently surpasses the baseline in detecting changes in health state on all tasks. \n\nNext, we emphasize that WBM and PPG are complementary. WBM excels in some tasks (e.g., sleep duration and infection), while PPG is stronger in others (e.g., diabetes). However, combining both achieves the best subject-level performance in 42/47 tasks (with a majority being significant), and in all but 1 of the segment-level tasks (where it’s within margin of error). Behavior data should complement, not replace, sensor data when building prediction models from wearables.\n\nWe will edit the language in the camera ready based on your feedback and our response above to better clarify our contributions. \n\n**Combining WBM and PPG Representations**\n\nWe apologize that this was unclear.  We will clarify in the camera ready that we combined WBM and PPG embeddings by concatenating the two 256D embedding vectors into one 512D embedding vector. There are many better ways to build multimodal representations using fusion techniques either at the input or representation level that we did not explore. We will add a discussion of these as future work in the camera ready. \n\n**Relation to Broader Wearable Community**\n\nThanks for raising this point.  We will improve the discussion by connecting our work with the broader space of digital health and wearables, and mention the potential clinical impact such wearables-based health predictions might have in the future if safely deployed at-scale.\n\n**Label Leakage in Downstream Tasks**\n\nThis is a subtle but important point, as label leakage is a major challenge in building foundation models, and you are correct that a small number of our input variables (e.g. basal energy) rely on age and sex as inputs.  However, we clarify that age and sex prediction are not meant to showcase the value of our model.  We view these tasks as sanity checks that our model is able to encode information that we already expect should be partially available. We will make this caveat more clear in the camera ready. We will also emphasize the importance of the other tasks, especially the segment-level tasks as mentioned above. Label leakage should not be a major concern for the 55 other downstream tasks, as none of those labels are used as part of the input variables.  \n\n**Computational Cost of Training and Inference**\n\nThe final WBM was the result of 6 epochs of training which took 16 hours of training time on 8 A100 GPUs. The learned model can quickly perform inference, and embeddings can be used easily across many tasks. We will add these details in the camera ready. \n\n**Robustness to non-Apple devices and other wearing patterns**\n\nYou bring up an important point about generalizing to non-Apple devices and other wearing patterns. During training, we opted to remove weeks with low wear time, so we expect performance will degrade when applied to participant weeks with limited wear time. Evaluations on non-Apple devices is not possible, as much of the data can only be  collected on Apple devices (particularly behavioral features derived via proprietary algorithms). However, our training details and insights provide a useful framework for others to train models on other wearables. We will discuss these limitations in the camera ready.  \n\n**Typos and Writing Suggestions**\n\nWe appreciate the feedback on typos, writing and citation improvements; we will fix these in the camera ready. We clarify that “1-layer multi-layer perceptron” means there is one hidden layer in addition to the input/output layers. \n\n**Miscellaneous Responses**\n\nThank you for bringing up important topics regarding interpretability, selection bias, and evaluation across different demographic groups. Due to space constraints, we point you to responses to R1 and R2 respectively for these topics. \n\nWe will also clarify in the camera ready version that the choice to fit models weekly vs. at a participant-level was made based on the task. For time-varying tasks where we make predictions at every week, we fit models at a weekly level. For static targets, we fit models at a participant level.", "claims_and_evidence": "- The main claim is substantiated with good results of the behaviour model.\n- The claim of the model learning from behaviour might be overstated (see methods).\n- The smaller claims based on comparison of “sensor data” vs “behaviour data” do not appear to be backed up sufficiently.\n- The paper presents a very optimistic narrative about WBM's performance, but the results reveal that WBM only outperforms the PPG model in 18 out of 47 baseline disease and medication outcomes, and of these, only 4 results are statistically significant. A more accurate framing would acknowledge that behavioral data provides valuable information for specific types of tasks (e.g. sleep and mobility), low-level sensor data appears more broadly effective across the majority of tasks.\n- Related to the above, while WBM outperforms the simple baseline on a majority tasks, the performance improvements are quite modest (median AUROC improvement of only 0.017). This modest gain raises questions about whether the additional complexity of foundation models is justified compared to simpler statistical approaches for many applications, particularly when it comes to interpretability, sensitivity and failure modes. These tradeoffs are not discussed sufficiently.", "methods_and_evaluation_criteria": "- The WBM is pre-trained from 27 variables from the wearable device, an Apple watch, to predict age and biological sex. These variables include estimated active energy (calories burned) and basal energy, which can be problematic. I am fairly certain the apple watch uses the individual’s age and biological sex directly to estimate these values, causing label leakage. This would then mean that the model is trained to predict age and biological sex not only on pure behaviour signal, but based on values that are conditioned on the age and biological sex. This could also be the case for VO2max to a lesser extent, typically calculated using the person’s estimated max heart which is in turn a direct function of the person’s age. Although label leakages are almost impossible to avoid in real-world healthcare research, I urge the authors to check if these can be minimised further or, at least, better acknowledged.\n- The approach used for combining WBM and PPG is not discussed.", "theoretical_claims": "The theoretical claims about the complementary nature of behaviour and signal data are mostly supported with the combination of the two models. However this theoretical aspect is not a major claim of the paper.", "experimental_designs_or_analyses": "- The authors make claims on the signal strength of behaviour vs low-level sensor signals that are based on comparisons of WBM and PPG. However these approaches differ fundamentally in more than just signal types. The sampling frequency, data processing, model architectures appear very different. So this seems to be as much of a comparison of sampling frequency and architecture as data signals. It seems a better way to make these claims would have been to ablate the WBM by removing behaviour features and leaving in the “low level” features, i.e. heart rate.  \n- There is possible selection bias in the data representativeness, i.e. limited to Apple watch users. This limitation isn’t sufficiently discussed.", "relation_to_broader_scientific_literature": "This is a weak point of the paper as the discussion of the results in broader context is limited. The discussion does not place the findings in the context of other digital health interventions or wearable technologies beyond a narrow set of foundation models. The authors miss opportunities to connect their work to broader healthcare trends, the potential clinical impact of wearable-based health predictions or how these models might integrate with existing healthcare systems.", "essential_references_not_discussed": "- There are not many prior works in this field apart from the Merrill and Althoff (2023) already cited and an earlier SSL wearable paper that also uses behaviour signals Kolbeinsson et al. \"Self-supervision of wearable sensors time-series data for influenza detection.\" (2021).\n- If the combination of WBM and PPG is ensemble-style, then some references to prior work on ensemble models in health would be appropriate.\n- It would also be clearer to move the seminal citations for rotary transformers and mamba to directly after the paragraph headers where they are named.", "other_strengths_and_weaknesses": "Strengths:\n- The systematic approach to architecture selection is thorough and usually well-justified\n- The dataset size is impressive and allows for robust foundation model development\n- The diversity of downstream tasks provides a comprehensive evaluation framework\n- The behaviour model shows particular promise for sleep and mobility predictions\n\nWeaknesses:\n- Limited discussion of computational requirements and model efficiency\n- Lack of evaluation across different demographic groups to assess fairness\n- The model's interpretability is not discussed, which is important for healthcare applications\n- No discussion of how the model might perform on non-Apple devices with different sensors\n- Limited exploration of more sophisticated fusion techniques when combining WBM and PPG", "other_comments_or_suggestions": "- On line 216 “1-layer multi-layer perceptron”, does this mean a single-layer-perceptron or an MLP with one hidden layer? The current phrasing is a bit clumsy\n- The paper switches between “wearables data” and “wearable data”, the former seems more semantically correct but I do not have a preference as long as it is consistent. \n- L169 “Hourly aggregation ensures consistency across variables” using “ensures” here seems overclaimed, as it is not guaranteed. “Supports” or “promotes” might be more accurate. \n- L295 The split is 80/20 train/test. Should this be 80/10/10 to match with L183?\n- L205 TST is not properly defined and citation could be clearer\n- Caption for table 1 and table 2: WHB → WBM", "questions_for_authors": "- L294 How is the decision to fit to week or participant made? \n- Are some of the features estimated using the individual’s age and biological sex? What implications might that have? \n- How are WBM and PPG combined? \n* What is the computational cost of training and inference for the WBM model compared to traditional approaches?\n* Were any analyses done to model performance across different demographic groups to assess potential biases?\n* Have there been any tests on the model's robustness to different wearing patterns and compliance levels?\n* What privacy-preserving techniques could be implemented alongside this approach for real-world deployment? How might they affect performance?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6LhsbhwXdY", "reviewer": "Reviewer_5c9n", "summary": "This work provides a foundation model for wearable devices on the behaviour data instead of the raw sensor signals. The model was trained on a large-scale wearable dataset totalling over 2.5B hours of wearable data from 162K individuals. This paper has performed extensive experimentations on the choice of tokeniser, model backbone and hyper-parameter optimisation. The best performing model was evaluated on 57 tasks ranging from the the disease detection, health state monitoring (pregnancy) to behaviour monitoring (sleep). The model showed superior performance against a baseline that was trained using summary statistics of the selected behaviour.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive feedback and helpful comments towards improving our work. We focus on responding to the major themes of your comments:\n\n**Measuring Difficulty of Downstream Tasks**\n\nWe agree that it is important to contextualize the difficulty of the tasks. To overcome this, we included baseline statistics of the input data and an existing PPG foundation model as a strong baseline to contextualize the performance of WBM. The performance of these models shows that no task is particularly easy or hard, but the strong performance of WBM relative to these baselines helps show the efficacy of the proposed model. \n\nPer your suggestion, we will include the case/non-case distribution for all binary outcome labels in the camera ready version of the paper to improve our evaluation report. However, importantly, prevalence does not immediately define the difficulty of the downstream task.\n\n**Using Highly Sparse Input Modalities**\n\nThis is a key characteristic of an observational study such as AHMS collected under real-world conditions where these variables are less often logged due to their nature and behavior of the participants. It is true that given the unique nature of these variables, it is harder for the model to leverage their temporal dynamics and changes, but our goal here is to model the data as-is but use state-of-the-art architectures such as state space models and Transformers that can leverage any potential temporal dynamics or inter-variable relationships. We experimented with tokenizers such as “Tuple“ tokenizer that does not do any form of imputation (therefore no constant input for sparse variables); we observed degraded performance with the ”tuple“ tokenizer in our hyperparameter search experiment in Appendix Tables 8 & 9.\n\nWe do find that the learned embeddings retain information for several variables that are natively sampled at a weekly frequency (e.g. six minute walk distance and walking steadiness score, see Table 10 in the Appendix). However, as you point out, there are some variables that we do not capture well in our embeddings. This could be due to their low prevalence or the use of the contrastive loss for training, which we will discuss further in the camera ready. As you suggest, we could remove such variables for computational efficiency, and we will add a line explaining this to the camera ready version of the paper. \n\n**Quality Control of Inputs and Labels**\n\nFor input behavioral data, we do not do a per-modality or per-variable quality control and we use the data as is. However, for turning these behavioral data into week-level segments for training WBM, we perform the following cleaning steps:  1) z-score each variable then clip any outliers to [-5,5], 2) drop weeks whose number of variables are in bottom 5% percentile of all weeks, 3) drop weeks that have less than 5 days of data, 4) drop subjects with less than 5 weeks of usable data, or who were enrolled in the study for less than 90 days. \n\nFor segment-level labels of baseline history and medication, you bring up an important point in that we are working with self-reported labels. In the camera ready version, we will discuss the caveat of parsing the labels from self-reported surveys (although for some of our evaluations we do use more rigorous lab measurements, eg, diabetes).\n\nFinally, we clarify the quality control for defining sleep labels. To obtain sleep labels, a participant must wear the watch overnight in order to get sleep metrics. We further limited to weeks where 5/7 days in the week had sleep metrics, meaning the watch was on overnight. This means that the PPG was recorded overnight as well, ensuring a fair comparison between both techniques. \n\nWe will be sure to clarify all of these points in the camera ready version. \n\n**Making Model Weights Open Source**\n\nUnfortunately the model weights cannot be shared due to the specifics of the informed consent for participants in the study.  We will provide all necessary details in the camera ready and will put a note for interested parties to reach out to the authors for more details. However, for completeness, we will include a model card with the exact details of model size, number of layers, and other necessary training parameters in the camera ready.", "claims_and_evidence": "- Strong performance of behaviour data for health detection: not clear; even though the authors have shown their model performance across a rich set of downstream tasks. It is not clear how difficult the downstream task given that there is no competitive benchmark, also detailed were not provided in terms of case/non-case distribution potentially making the binary classification task very hard or very easy.\n- Integrating behavioural and sensor data: yes, when combining the proposed behaviour model with a PPG model, the authors showed that the integration increased the performance on majority of the downstream tasks\n- Developing a foundation model for wearables behaviour data for irregularly sampled data: yes, this is the one of the first foundation models on behavioural time series data at this scale. But the authors have not talked about sharing the model weights or codebase make it much harder for others to reproduce the work.", "methods_and_evaluation_criteria": "This paper explored the combinations of three tokenisers and three backbone model architectures\n\n- Two types of tokeniser classes were selected one with dense model and another in the form of tuple allowing for a single input token for each behavioural measurement\n- The choices for model architectures were well-motivated including standard transformers and  a state-space model i.e. Mamba-2\n- The choice of the pre-training loss has already been shown to achieve good performance in other data modality e.g. PPG\n\nA key weakness of the method is on incorporating data modalities that are very sparsely sampled particularly around that have samples for ≤10% of the time such as fall count, body mass index and 6 minute walk distance. Even though these matrices are highly relevant for health but when treating them like time series requires a strong motivation as the model always just see some constant input therefore making it difficult to leverage the temporal dynamics. \n\nYour results in Table 10 is painting the same story that the linear probing results on the learnt embedding has a $R^2$ of 0 with body mass index and 0.096  with numbers of times fallen. So at least a portion of input data with low sampling frequency and low variations can possibly be removed to computational efficiency.\n\n\nThe evaluation is the area that requires further justifications on.\n\n- The disease labels for the data used comes from self-report which will might under or over report different clinical outcomes introducing differential bias in the ground truth. Would be to discuss how this can be handled.\n- The case/non-case distributions are not being shown for each disease, so it is challenging to know the difficulty of each that. Would be great if the authors describe the case/non-case distributions for each disease and how the non-case subjects are selected.\n- On the ascertainment for sleep ground truth, the authors did not explain what sort of quality control is done on the sleep labels for example the removal of outliners, minimal wearable time etc.,. Without a careful quality control, it could lead to high measurement error which can potentially explain why the PPG embedding has low $R^2$ 0.1-0.3 to the sleep statistics.", "theoretical_claims": "None", "experimental_designs_or_analyses": "I’ve checked its experimental setup for train and eval which is reasonable to obtain performance metric with confidence intervals for the comparisons for different models.", "relation_to_broader_scientific_literature": "The key contributions of this paper are: \n\n1. Explore the concept of using behaviour level modelling instead of lower-level sensor representation for health inference which will make it easier to leverage longitudinal wearable time series. Previous foundation models in wearable signals are mainly for low-level sensor representation.\n2. It nicely paved a way of integrating behaviour level information with additional model like the PPG and demonstrated how this multi-modal approach could aid in health inference", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strengths: \n\n- The authors have provided extensive reports on their definitions of the behaviour metrics and downstream task creation which is clear", "other_comments_or_suggestions": "1. What sort of quality control did you apply for your input data for each modality.\n2. Can you provide a model card for your optimal model covering model size, numbers of layers and training config?", "questions_for_authors": "Will the model weights and codebase be made open source?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper has received Accept, Strong Accept, Accept, Accept. All reviewers have praised the paper and its contributions. While some claims required softening, and limitations around demographic conditioning and generalizability were acknowledged, the discussion phase successfully led to a consensus. Some reviewers raised concerns, both in their discussions with the authors and with each other/AC, regarding the dataset, particularly its potential demographic biases. **A key point of concern was the lack of public availability of the dataset and the absence of released code or model weights. Open source of the model weights and at least a subset of the dataset is highly required.** Based on the strengths of the work, I suggest for the paper to be accepted, but urge the authors to find a long-term sustainable solution for this problem.
Decision: Accept (poster)