Large-scale Training of Foundation Models for Wearable Biosignals
Authors: Salar Abbaspourazad
Oussama Elachqar
Andrew Miller
Saba Emrani
Udhyakumar Nallasamy
Ian Shapiro
Venue: ICLR 2024 poster (2024)
Keywords: Self-supervised learning
Representation learning
Foundation models
Biosignals
Wearable devices
Health
Photoplethysmography
PPG
Electrocardiogram
ECG
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: We employ self-supervised learning on the unlabeled wearable data from a large health study to train foundation models for two common biosignals, PPG and ECG, and show that these models encode participants' demographics and health conditions.

Abstract: Tracking biosignals is crucial for monitoring wellness and preempting the development of severe medical conditions. Today, wearable devices can conveniently record various biosignals, creating the opportunity to monitor health status without disruption to one's daily routine. Despite widespread use of wearable devices and existing digital biomarkers, the absence of curated data with annotated medical labels hinders the development of new biomarkers to measure common health conditions. In fact, medical datasets are usually small in comparison to other domains, which is an obstacle for developing neural network models for biosignals. To address this challenge, we have employed self-supervised learning using the unlabeled sensor data collected under informed consent from the large longitudinal Apple Heart and Movement Study (AHMS) to train foundation models for two common biosignals: photoplethysmography (PPG) and electrocardiogram (ECG) recorded on Apple Watch. We curated PPG and ECG datasets from AHMS that include data from ${\sim} 141$K participants spanning ${\sim} 3$ years. Our self-supervised learning framework includes participant level positive pair selection, stochastic augmentation module and a regularized contrastive loss optimized with momentum training, and generalizes well to both PPG and ECG modalities. We show that the pre-trained foundation models readily encode information regarding participants' demographics and health conditions. To the best of our knowledge, this is the first study that builds foundation models using large-scale PPG and ECG data collected via wearable consumer devices $\textendash$ prior works have commonly used smaller-size datasets collected in clinical and experimental settings. We believe PPG and ECG foundation models can enhance future wearable devices by reducing the reliance on labeled data and hold the potential to help the users improve their health.

Published as a conference paper at ICLR 2024
LARGE-SCALE TRAINING OF FOUNDATION MODELS FOR WEARABLE BIOSIGNALS
Salar Abbaspourazad Oussama Elachqar, Andrew C. Miller, Saba Emrani, Udhyakumar Nallasamy, Ian Shapiro" Apple
ABSTRACT
Tracking biosignals is crucial for monitoring wellness and preempting the devel- opment of severe medical conditions. Today, wearable devices can conveniently record various biosignals. creating the opportunity to monitor health status without disruption to one's 's daily routine. Despite widespread use of wearable devices and existing digital biomarkers, the absence of curated data with annotated medical labels hinders the development of new biomarkers to measure common health conditions. In fact, medical datasets are usually small comparison to other domains, which is an obstacle for developing neural network models for biosignals. To address this challenge, we have employed self-supervised learning using the unlabeled sensor data collected under informed consent from the large longitudinal Apple Heart and Movement Study (AHMS) to train foundation models for two common biosignals: photoplethysmographhe (PPG) and electrocardiogram (ECG) recorded on Apple Watch. We curated PPG and ECG datasets from AHMS that include data from ~411 participants spanning ~3 years. Our self-supervised learning framework includes participant level positive pair selection, stochastic augmentation module and regularized contrastive loss optimized with momentum training, and generalizes well to both PPG and ECG modalities. We show that the pre-trained foundation models readily encode information regarding participants' demographics and health conditions. To the best of our knowledge, this is the first study that builds foundation models using large-scale PPG and ECG data collected via wearable consumer devices prior works have commonly used smaller-siae datasets collected in clinical and experimental settings. We believe PPG and ECG foundation models enhance future wearable devices by reducing the reliance on labeled data and hold the potential to help the users improve their health.
INTRODUCTION
Recent advances in wearable device technology allow for the recording of various biosignals, which can be used to monitor users' overall wellness. Two of the most commonly collected biosignals from wearable devices are photoplethysmography (PPG) and electrocardiogram (ECG). PPG measures volumetric changes in arterial blood flow and contains wide variety of biological information, and ECG measures cardiac electrical activity and contains information about cardiac function. While these biosignals hold significant potential, the absence of curated datasets with annotated medical labels has been an obstacle for developing digital biomarkers employing deep neural networks. Medical datasets are typically collected in lengthy and expensive health studies, require domain expertise for annotation, and are usually collected from limited number of participants, which can make the learned models less generalizable broad, demographically varied populations.
Large-scale training of foundation models via self-supervised learning (SSL. has shown promise in domains such as natural language processing (Devlin et al., 2019; OpenAI, 2023), computer vision (Chen et al., 2020; 2021; Oquab et al., 2023) and speech recognition (Baevski et al., 2020; 2022). Self-supervised learning often does not require explicit labels, making suitable to pre-train foundation models on unlabeled biosignals, and has been recently proven successful for health applications (Hallgrímsson et al., 2018: Cheng et al., 2020; Kostas et al., 2021: Sarkar & Etemad,
"Corresponding authors: (salarabb, ishapiro) appll.ccom

---

Published as a conference paper at ICLR 2024
2022; Mohsenvand et al., 2020; Gopal et al., 2021; Kiyasseh al., 2021; Mchari & Strodthoff, 2022; Wu et al., 2020; Spathis et al., 2021; Tang et al., 2021; Yuan et al., 2023; Lai et al., 2023).
While there have been recent efforts to apply SSL wearable accelerometer data in free-living conditions (Spathis et al., 2021; Yuan et al., 2023), other SSL. work have mostly used biosignal modalities such as PPG, ECG, and electroencephalogram (EEG) collected in clinical or controlled experimental settings (Cheng et al., 2020; Kostas et al., 2021; Sarkar & Etemad, 2022; Mohsenvand al., 2020; Gopal al., 2021; Kiyasseh al., 2021; Mchari & Strodthoff, 2022: Lai et al., 2023). Developing foundation models for biosignals has several advantages as they: 1) require less amount of labeled data to reach to the same aceuracy as supervised models, which significantly reduces the costs of experimental health studies (Cheng et al., 2020). 2) can be further fine-tuned on downstream targets, leading to better accuracy and requiring less computation resources given faster convergence compared to training from scratch (Hu et al., 2021), 3) provide gegna-tt--mbeddingg models that can be used to calculate similarity scores between users or signals, and allow for faster information retrieval (Mitra & Craswell, 2018).
Here, using the large, longitudinal. multi-year Apple Heart and Movement Study (AHMS), we train biosignal foundation models with PPG and ECG. Our contributions are: 1) Large-scale pre-training for biosignals: To the best of our knowledge, this is the first work for pre-training foundation models on PPG and ECG biosignals using large-scale dataset collected via wearable consumer devices with 141,207 participants. 2) Self-supervised learning framework: We combine techniques used in self- supervised learning for biosignals with techniques from other domains, such as computer vision. Our self-supervised pre-training has a stochastic participant level augmentation module, and the encoder optimized with momentum training with a regularized InfoNCE loss. 3) Studying information encoded by the pre-trained foundation models across various targets: We show that praitreined PPG and ECG embeddings contain information predictive of a broad range of demographic variables and health conditions. We studied the amount of information encoded separately in the PPG and ECG embeddings of pre-trained models with respect to various targets including demographics, health conditions and medications. 4) Ablation studies: We performed various ablation studies to evaluate the importance of participant level positive pair selection, to analyze distinctions in the pre-training and embeddings between PPG and ECG, to enable comparison with SimCLR (Chen et al., 2020) and B BYOL (Grill et al., 2020) as examples of contrastive and non-contrastive pre-training frameworks on dataset, to study the efficacy of different encoder architectures and model sizes, and to study the effect of different augmentation functions.
2 RELATED WORK
Self-supervised learning has recently gained popularity in various domains of deep learning. Our work is most related to a large body of self-supervised learning research using joint embedding architectures (JEA). JEAs are usually trained with well-tailored loss to avoid representation collapse; contrastive losses such InfoNCE avoid the collapse by contrasting various samples in the batch, and non- contrastive losses avoid the collapse using momentum training (Grill et al., 2020), or some form of regularization (Bardes al., 2022), or both (Baevski et al., 2022). Self-supervised pre-trained models have been shown to encode significant amount of information about downstream targets without seeing any labels during pre-training (Chen et al., 2020; Baevski et al., 2020; Mohsenvand al., 2020; Cheng et al., 2020; Kostas et al., 2021; Chen et al., 2021: Gopal et al., 2021; Kiyasseh et al., 2021; Mehari & Strodthoff, 2022; Baevski et al., 2022; Sarkar & Etemad, 2022: Oquab et al., 2023).
Self-supervised learning has also been proven to be effective in health applications, usually with small datasets of few hundred or few thousands participants and recorded in clinical or controlled experimental settings. These datasets include 12-lead ECG in hospital settings (Cheng et al., 2020; Gopal et al., 2021; Kiyasseh et al., 2021: Lan et al., 2021: Liu et al., 2021; Sarkar & Etemad, 2022: Mehari & Strodthoff, 2022; Diamant et al., 2022; Lai et al., 2023), EEG collected in controlled experimental setting (Mohsenvand et al., 2020; Cheng et al., 2020; Kostas et al., 2021; Banville et al., 2021) or PPG from few participants (Ghorbani al., 2023). Recent work have used self-supervised learning for wearable accelerometer signals in free-living conditions and large dataset (Spathis et al., 2021; Yuan al., 2023). However, the best of our knowledge, work presents the first study on pre- pre-training foundation models from large-scale PPG and ECG data collected from wearable consumer devices.
2

---

Published as a conference paper at ICLR 2024
3 SELF-SUPERVISED PRE-TRAINING
We use contrastive self-supervised learning to pre-train deep neural network encoder, as described in Appendix Algorithm 1, and explain the components in details below.
3.1 POSITIVE PAIR SELECTION AND AUGMENTATIONS
self-supervised learning with a JEA often requires positive pairs during pre-training. To motivate the model extract information relevant to participant level physiology, positive pair selection strategy; the positive pairs are selected as augmented views of two different segments from the same participant. We denote segment for participant s as x². therefore positive pairs are randomly selected pairs in form of {I, xz]li * j}. The participant level positive pair selection significantly improves the quality and utility of the learned embeddings (Ablation 5.2.1).
a participant level
Our augmentation module T(-) consists of a stochastic sequence of time-series augmentations, including crop, add Gaussian noise, time warp, magnitude warp, and channel swap, as explained in (Iwana & Uchida, 2021). We assign a configurable probability to each augmentation function in the augmentation set, and at each call of T(-), sequence of augmentation functions the augmentation set is applied given the random binary events drawn from the assigned probability values. In addition to this randomness, each augmentation has internal randomly selected hyperparameters, which we did not tune for the purposes of this study. Given various sources of randomness in the augmentation module, it covers a diverse range of distortions to the input segment, and we control the strength of distortions by changing the probability values. Our augmentation set and the assigned probability values for PPG are: {cut out: 0.4, magnitude warp: 0.25, add Gaussian noise: 0.25, channel permute: 0.25, time warp: 0.15), and for ECG are: (cut out: 0.8, magnitude warp: 0.5, add Gaussian noise: 0.5, time warp: 0.3). ECG segments have only one channel in our datasets (see 4,1), which is why we drop the channel permute augmentation for ECG. Also, to address the fact that ECG has less within-parte participant variability compared to PPG (Ablation 5.2.2), we assign higher probability values our ECG augmentation module to introduce stronger signal distortions.
3.2 REGULARIZED INFONCE LOSS
We use InfoNCE (also known NT-Xent) to maximize the mutual information between the positive pair representations while allowing for contrast to other positive pairs in each batch via the cross- entropy function to avoid representation collapse (Oord et al., 2019; Chen et al., 2020). In addition, to encourage a uniform span of features within the batch, we use Kozachenko-Leonell differential entropy estimator regularization, used in (Sablayrolles al., 2019; Oquab et al., 2023). For each batch of embeddings from N positive pairs (h1, h2), we define the model's objective as below:
(KoLeo)
(1.2) contrastive
1
N
exp(sim (hi hb)/T)
log
N
1 1 + i cxp R2)/T)²
==1
where sim(.,)) is the cosine similarity function. KoLeo regularization also calculated as:
N
Σ
log(min j hi nz|
(2)
N
1=1
The final objective is computed as the regularized InfoNCE loss:
(L (1.2)
contraative + - contrastivee + 2/2 ( + + (2) L
(2.1)
(1)
Both of the contrastive and regularization losses are calculated after (2-normalization the embed- dings and details can be found in Appendix Algorithm 1.
3.3 MOMENTUM TRAINING
While is not necessary to use momentum training to avoid collapse given the contrastive component in our objective function, momentum training create more mismatch between the positive
3

---

Published as a conference paper at ICLR 2024
Table Number of participants/sogmenss average number of calendar days per participant, and total dataset time span (time between the earliest to the latest recorded segment) in our pre-training datasets. We used the data from 80% of the participants for training. 10% for during training validation, and 10% for occasional post-training assessments.
PPG
ECG
Number of participants
141,207
106,643
Number of segments
19,854,101 101
3,743,679
Average number of calendar days per participant Total dataset time span (days)
92.54
23.27
890
1,240
pair representations, further encouraging the network to learn informative representations. This is particularly more helpful for signals such as ECG that have less within participant variability (Figure 2). We apply momentum training to both the encoder and projection head, where the online side of the JEA is updated via backpropagation and the momentum side is a lagging exponential moving average of the online side. The momentum update rule is & <- <- T + (1 - 7)0, where and & denote the weights of the online side and the momentum side, respectively and momentum update rate.
EXPERIMENTS
4.1 DATASETS
We used the PPG and ECG signals recorded on Apple Watch from participants in the Apple Heart and Movement Study (AHMS) (MacRae, 2021). AHMS an ongoing research study designed to explore the links between physical activity and cardiovascular health, which is sponsored by Apple and conducted in partnership with American Heart Association and Brigham and Women's 's Hospital. To be eligible for the study, participants must be at least 18 years of age (21 in some locations), reside in the United States, have access to an Apple Watch, and provide informed consent electronically in the Apple Research app (Shapiro et al., 2023).
PPG pre-training dataset: Apple Watch intermittently and passively records green PPG signals during low-motion periods multiple times per day using light-emitting and light-sensitive diodes. The recorded PPG signals are 60 seconds in duration, sampled at 256Hz or 64Hz, and consist of four separate optical channels corresponding different spatial combinations of transmitting and receiving diodes. We curated ~00 PPG segments from ~ 141K participants for our PPG dataset (Table 1). Random PPG segments were drawn from the full dataset given two conditions: 1) each participant had at least four segments in the pre-training dataset, 2) the number of segments per partieipant was as uniform as possible. PPG segments were pre-processed using dark subtraction (to reject signals introduced by ambient light), followed by bandpass filtering, down-sampling to 64Hz if needed and temporal channel-wise z-scoring for each segment.
Apple Watch Series 4 or later enables users to record ECG using wrist-to-finger (Lead I) dry electrode sensor. One electrode located on the crown and the other electrode is located the back of the watch. When the puts the watch their wrist and makes contact using a finger of opposite hand, they complete a circuit and the ECG signal recorded. The ECG recordings collected on Apple Watch 30 seconds long sampling frequency of 512Hz. We curated ~3.7750 75M ECG segments from ~066 participants under the same two conditions for the PPG pre-training dataset. ECG signal recordings were pre-processed using Apple internal tools to produce 30-second strips equivalent to those stored locally in each participant's Health app. followed by down-sampling 128Hz and temporal z-scoring for each segment. Brief statisties of our curated PPG and ECG datasets are in Table 1, AHMS demographics distribution is demonstrated in Appendix Fig. 5 and example PPG and ECG processed segments are shown in Appendix Fig. 6.
ECG pre-training dataset:
4.2 EVALUATION METRICS
Linear probing for age, BMI and biological sex: We perform linear probing for predicting self- reported age, body mass index (BMI), and biological sex (sex assigned at birth). For the classification tasks, we ridge regression predict scores for binarized targets (0/1) and we quantify the
4

---

Published as a conference paper at ICLR 2024
Table 2: Downstream performance evaluation of PPG and ECG embeddings in predicting age. biological sex and BMI of participants, using two types of positive pair selection strategies during pre-training: 1) participant level, 2) segment level.
Positive pair
Prediction task
PPG
ECG
AUC (pAUC)
MAE J
AUC (pAUC) 1
MAE +
Age classification
0.976 (0.907)
0.916 (0.763)
Age regression
3.19
6.33
Participant level
BMI classification
0.918 (0.750)
0.797 (0.612)
(main)
BMI regression
2.54
3.72
Sex classification
0.993 (0.967)
0.951 (0.841)
Age classification
0.900 (0.762)
0.783 (0.621)
Age regression
6.60
9.01
Segment level
BMI classification
0.766 (0.616)
0.734 (0.557)
BMI regression
4.05
4.11
Sex classification
0.870 (0.685)
0.854 (0.683)
performance with area under curve of receiver's operating curve (AUC), and partial AUC (pAUC) at false positive rate (FPR) of 10%. For regression tasks, we use ridge regression to predict continuous targets and we use mean absolute error (MAE) quantify performance. For age, we classify ages above 50 versus below or equal to 50, as well as a regression estimate of the continuous age value. For BMI, we classify BMI values above 30 kg/m² versus BMI below or equal to 30 kg/m² as well as predict continuous BMI values. For biological sex, we classify male versus female. In all our analyses, we perform the linear probing at participant granularity: we meen-aggreep embeddings associated to each participant so that each participant contributes one and only one sample in the downstream training/evaluation. Also, the downstream training/evaluation splits are stratified based on participants such that the evaluation split's participants have overlap with those in the training split.
all the
During AHMS, participants fill multiple questionnaires containing various questions regarding their historical health record and demographics. The response to these questions are usually form of 'yes' or 'no' for whether the participant has had a history of health condition (c.g.. asthma), or whether they take specific medications (e.g., anti-depressants). or regarding their lifestyle habits (e.g., smoking). Detailed description of questions and how we define targets is in Appendix A.2. To have a baseline, we compare the predictive ability of the pre- pre-trained encoder embeddings vs. baseline features that average heart rate and standard
Linear probing for targets from AHMS survey questions:
include age, biological sex, BMI, and ethnicity, in addition deviation heart rate that are derived from PPG and are known to be well informative of certain conditions. Similar to above, where training/cvaluation splits are stratified based on participants. We quantify linear classification of these targets using AUC from PPG embeddings, and compare these with classification AUC using a baseline feature set consisting only of self-reported demographic information (age, BMI, biological and race/ethnicity) and measured heart rate.
use ridge regression to predict scores for binarized targets (0/1)
Smooth effective rank: Smooth effective rank an unsupervised metric quantifying the entropy of the singular value distribution of the embeddings in a given batch (Roy & Vetterli, 2007: Garrido et al., 2023), and can be used as a crude proxy for downstream evaluations without requiring any labels, as has been shown to positively correlate with downstream evaluations (Garrido al., 2023). We calculated smooth effective rank as its average for the batches of size 256 in the pre-training validation split.
4.3 IMPLEMENTATION DETAILS
Our default encoder is EfficientNet-style ID convolutional neural network (Ablation 5.2.4) with 16 mobile--nvertee bottleneck blocks with squeeze-and ann-eeeetadeeiiiiiioo (Tan & Le, 2020) and we used 256- 256-dimensional embedding (the representation vector after the deep encoder) for all models used in this study. The encoder had 3.3M parameters for PPG and 2.5M for ECG. The projection head was a multi-layer perceptron with one hidden layer of 1024 units, taking the 256-dimensional embedding
5

---

Published as a conference paper at ICLR 2024
a 1.0
b
0.9
ROC AUC (5C8 emboddings)
(PPG emboddingg
ROC AUC
0.7
0.7
0.55
0.6
ROC
[Baseline fealurras)
(Baseline features)
(Age, Sex, BMI, Ethnicity, Mean/std heart rate)
ACE-inhiillar
Ansiely
Chemotherapy
Heart failure
Osteoporoeis
Active sicohol
Artery dieesse
Cholesterol
Heart rhythm
Pacemoker
Active smoker
Arthitis
Chronic branchilis
HiplKnee
Painkilers
Afl
Asthma
Depression
Kidney
Sleep
Allergy
Beta blockers
Diabetes
Liver
Sleep medication
Anti-nniely
cod
Diuretics
Stroke
Anti-ssychotss
cod preasure med.
Hearing
Neck disorder
Thynoid
Anticeagulants
Calcium channel brockers
Heart attack
Ncuropathy
Uninrry
Antidopressants
Cancer
Heart di disease
Opioid painkillars
Vision
Antiplatalets
Figure 1: PPG and ECG foundation models encode participants' health information. The comparison of linear probing evaluation for targets from AHMS survey questions (a) using PPG embeddings, (b) and using ECG embeddings, versus baseline features shown. Each marker represents one of the targets from AHMS questionnaire, the y-axis represents the ROC AUC of binary classification using the embeddings, and the x-axis represents that for the baseline features. The marker color and shapes selected randomly, and described in the legend.
to a 128 18-dimmenional representation subspace where the loss calculated in. For InfoNCE, we used temperature value of 0.04 for both PPG and ECG modalities, and the weight for the KoLeo regularization in our objective function was set to 0. The models were trained using Adam optimizer with gradient descent updates distributed across 32 A100 GPUs. Other implementation details is in Appendix A.1.
RESULTS
5.1 PPG AND ECG FOUNDATION MODELS ENCODE PARTICIPANTS HEALTH INFORMATION
We analyzed the amount of information encoded in the embeddings (the representation after the deep encoder) via linear probing. First, we evaluated linear classification and regression performance for predicting age, biological sex and BMI for both PPG and ECG embeddings, summarized in Table (see 4.2 for details). We observed that PPG embeddings were more predictive of these targets, suggesting that periodically-collecte background PPG signals may contain more information predictive of downstream health predictions compared to user-initiated ECG signals Second, we evaluated linear classification performance for many binary targets derived from AHMS self-reported questionnaires using PPG and ECG embeddings, and compared these with the corresponding classifier performance achieved using baseline feature set consisting only of self-reported demographic information (age, BMI, biological sex and race/ethnicity) and measured heart rate (see 4.2 for details). Fig. la and Fig. 1b show the ROC AUC for predicting these targets using PPG and ECG embeddings compared to the baseline features (also see Appendix Table 9). We made multiple observations: PPG embeddings almost always performed better than baseline features in predicting

---

Published as a conference paper at ICLR 2024
downstream conditions, indicating that the PPG foundation model embeddings readily encode information regarding participant health conditions, and this information is beyond what that can be predicted from the participant demographics and heart rate. This gives us confidence that the encoder not simply using heart rate and demographies (which can be inferred from the biosginals with surprising accuracy shown in Table 2) as a 'shortcut" to predict whether the participant has an age-related condition such as heart disease. 2) While there are various health conditions that are better predicted with ECG compared to the baseline, it appears that ECG embeddings encode less information regarding participant health compared to PPG embeddings. This could have different reasons: ECG signals may contain less information specific to these conditions, and/or our pre- training framework is more optimal for PPG compared to ECG (see Ablation 5.2.2 and Discussion). The gap in performance not explained by the different number segments in the PPG and ECG datasets (20m vs. 3.75m), as we observed that model trained on a smaller PPG dataset which has similar number of segments to the ECG dataset still had significantly better performance in predicting demographies and health conditions (Appendix Table 10).
5.2
ABLATION STUDIES
5.2.1
POSITIVE PAIR SELECTION
We did an ablation analysis regarding the importance of participant level positive pair selection by changing this to segment (instance) level during the pre-training phase. Segment level positive pair selection is conventionally used for self-supervised learning (Chen et al., 2020; Baevski et al., 2020; Mohsenvand al., 2020; Cheng et al., 2020: Kostas et al., 2021; Chen et al., 2021; Gopal et al., 2021; Kiyasseh et al., 2021; Mehari & Strodthoff, 2022; Baevski et al., 2022; Sarkar & Etemad, 2022; Oquab al., 2023), where positive pair two augmented views of an individual segment/sample. We observed that consistently for both PPG and ECG, pre-trained models with participant level positive pairs were significantly more predictive of downstream targets (Table 2). While the evaluations in Table 2 are at participant granularity (see 4.2). we observed similar improvements for evaluations at segment granularity (Appendix Table 11).
5.2.2 DISTINCTIONS IN PPG AND ECG PRE-TRAINING AND EMBEDDINGS
We observed meaningful differences in self-supervised pre-training for PPG compared with ECG signals. To better understand the distinctions, we studied the PPG vs. ECG pre-training metries and the resulting embeddings:
Visual inspection of t-sne representations: We performed t-sne dimensionality reduction (Maaten & Hinton, 2008) on 200 segment embeddings, consisting of 20 segments randomly drawn from 10 random participants, for both PPG and ECG modalities. First, as expected, we observed that participants form clusters in the t-sne subspace (Fig. 2a). However, visually inspecting these plots (only one shown here), we observed that ECG participant clusters tended to be denser than those for PPG. This qualitatively shows that ECG segments, and thus ECG embeddings, within a particular participant as opposed to that for PPG. This is one of the reasons our augmentation cascade (see 3.1) has higher probability values for each augmentation for ECG compared to PPG, to make the pre-training more difficult for ECG by introducing a stronger distortion to the original segments.
similar
In line with the previous observation, we looked at the validation InfoNCE loss for ECG vs. PPG, and observed that the InfoNCE loss was significantly smaller for ECG compared with PPG (Fig. 2b). This further corroborates that pre-training using InfoNCE is an easier task for ECG as opposed to PPG, perhaps due to the fact that it is easier distinguish participants from their ECG segments versus their PPG segments.
InfoNCE loss:
To quantify what we qualitatively observed from visually inspecting reduced t-sne dimensions, we used dispersion ratio metric that is computed as the ratio between within participant variability vs. across participant variability. Fig. 2c represents the dispersion ratio for each of the 256 features in the embeddings for PPG vs. ECG. We observed that, in line with previous observations, ECG embeddings had statistically smaller dispersion ratio across the population compared to that for PPG embeddings.
Dispersion ratio:
7

---

Published as a conference paper at ICLR 2024
PPG
EGG
PPG
PPG
T-SNE dim
dim 2
InfoNCE (val)
EGG
PDT
T-SNE
SNE
20k
40k
60k
0.0
0.5
1.0
1.5
Training iteration
Dispersion ratio
Figure 2: Distinctions in PPG and ECG pre-training and embeddings. a. T-SNE representations for 20 random embeddings drawn from 10 random participants for PPG and ECG. b. InfoNCE validation loss for PPG vs. ECG, where training iteration represents global gradient descent update across all GPUs. c. Dispersion ratio probability density function (PDF) calculated for each feature of the 256-dimensional PPG and ECG embeddings across the population. Dispersion ratio quantifies within participant variability to across participant variability.
These results together demonstrate that ECG and PPG signals have inherent distinctions with respect to the participant- and health- speltticppeiiicc information contained in their embeddings. While our pre- training generalizes well to both modalities, follow up methods could ideally use frameworks to better take into account such distinctions.
5.2.3 PRE-TRAINING FRAMEWORK
We compared our pre-training framework with our variation of SimCLR and BYOL, as popular frameworks for contrastive and non-contrastive self-supervised learning. In fact, to make the compar- ison fair, almost all other training details were similar between these methods, including participant level positive pair selection even though the original frameworks had segment level positive pair selection (Chen al., 2020; Grill al., 2020), augmentation module, or other important implemen- tation details in 4.3. For comparisons, we used smooth effective rank as a crude proxy of general downstream performance, and downstream evaluations for specific targets of age, biological sex and BMI. In general, we observed that contrastive frameworks were more robust and resulted in more informative embeddings (Table and Appendix Table 12). Also, we saw that our pre-training framework was better than these two baseline variations across most evaluations. We observed that for both PPG and ECG modalities, removing KoLeo regularization from our objective function resulted in reduced evaluation metries (Table and Appendix Table 12), demonstrating its impact the model's performance. We also want to emphasize that, as discussed in 5.2.2, unsupervised evaluation metrics such InfoNCE and smooth effective rank are not directly comparable between signal modalities. In fact, we believe low InfoNCE and high smooth effective rank are necessary for good downstream performance but not sufficient these unsupervised objective/metries are correlated with downstream performance but the mapping/relationship is not necessarily monotonic. For instance, we observed that ECG embeddings had lower InfoNCE (Figure 2) and higher smooth effective rank (Table 3) compared to PPG embeddings, while PPG was more predictive of demographics and health conditions (Figure and Table 2).
5.2.4 ENCODER ARCHITECTURE
We used different encoder architectures within our pre-training framework study their importance in extracting informative embeddings. We compared our default ID-EfficienNeet with our variation of ID-ResNet and ID-ViT (see A.1). For comparisons, we used smooth effective rank crude proxy of general downstream performance, and observed that different encoder architectures, with different model sizes, can achieve relatively similar performance, demonstrating that the performance is not unique to the ID-EfficientNet encoder architecture (Table 4). We observed that 1 ID-EEficientNet model yielded similar performance to these alternative encoders with significantly smaller number of parameters, which was one of the main reasons we picked ID-EfficientNe as default backbone encoder given less memory footprint when potentially running on a wearable device. An interesting future direction is scaling up the model size further, particularly with transformer-bat based models (e.g., ViT) given their scalability (Kaplan et al., 2020), and larger datasets to study its effect on downstream performance.
8

---

Published as a conference paper at ICLR 2024
Table 3: Smooth effective rank calculated for different pre-training frameworks (higher is better).
Pre-training
PPG
ECG
Ours
104.13
113,82
Ours (no KoLeo)
101.29
108.84
SimCLR (our variation)
98.87
103.65
BYOL (our variation)
51.18
63.67
Table 4: Smooth effective rank (SER) and number of model parameters (Size) calculated for different encoder architectures within our pre- mre-trannnng framework.
Encoder
PPG
ECG
SER
Size 4
SER f
Size
ID- Df-fefiientNet (our variation)
104.13
3.3M
113.82
2.5M
ID- ID-ResNet (our variation) ID- (our variation)
95.81
16.9M
111.92
16.9M
104.89
7.2M
114.62
7.2M
5.2.5 AUGMENTATIONS
While we did not comprchensively tune our augmentation module (see Appendix A.I), we studied the effect of single augmentation functioon for each modality. To do so, for both PPG and ECG, we kept each of the augmentation functions in isolation separately (with probability 1), and repeated our pre- training framework from scratch. We observed that PPG pre-training of single augmentation functions (more variance in performance), and channel permute and cut out were the most effective augmentation functions in isolation for PPG and ECG, respectively (Appendix Fig. 7). Future work can study more optimal ways to design modality-speciffe augmentation modules.
sensitive the choice
6 DISCUSSION, LIMITATIONS AND FUTURE WORK
Pre-training foundation models using unlabeled data has been proven successful in various domains of deep learning such as natural language processing and computer vision. In this work, have pre-trained foundation models for PPG and ECG, as examples of common biosignals from wearable devices, using supervisdd learning and a large longitudinal dataset. We have shown that our pre- trained foundation models readily encode participant demographics with high accuracy (see A.4 for further discussion), as well as encode information predictive of broad range self-reported health conditions and medication categories. It is worth noting that our target labels are opportunistically constructed from participants' self-reported survey. Therefore, the labels used for training and evaluation the downstream classifiers sub-optimal and future work quantify these predictions using more accurate labels. Also, we would like to note that while PPG/ECG embeddings are predictive of health conditions, other biomarkers including but not limited to heart rate, heart rate variability (HRV), and physical activity still provide valuable insight regarding one's ' health status, and their relationship PPG and ECG embeddings should be examined further in future work. Our pre-training framework generalizes to PPG and ECG biosignals, however, we believe future work could gain improvements by incorporating modality-sppeciff design choices. Future work can also investigate the longitudinal changes in PPG and ECG embeddings, and whether accounting for those can improve downstream predictions. Another future area investigation multi-modal pre-training by: 1) using multi-modal encoder that takes multiple modalities (e.g., PPG and ECG) as input, 2) or CLIP-style multi-modal pre-training by supervising one modality with another one (Radford et al., 2021), or training multiple encoders for multiple modalities simultaneously (Girdhar al., 2023), using a contrastive loss. Last but not least, we observed that the choice of positive pairs significantly affects the quality of the embeddings; future work can investigate the efficacy of different positive pair selection strategies on different health-related targets, by accounting for temporal and other physiological information.
9

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
We would like thank participants in Apple Heart and Movement Study, Calum MacRae, MD, PhD, and study staff at The Brigham and Women's ' Hospital, Harvard affiliate, without whom this work would not have been possible. We also thank Jen Block, Joe Futoma, Sarvesh Kirthivasan, Andy Long and Chris Sandino for providing valuable feedback regarding the manuscript, Zach Ellison and Mayank Gupta for the training platform support, Lindsay Hislop, Eduardo Martinez Montes and Laura Rhodes for publication coordination.
REPRODUCIBILITY STATEMENT
The aggregated data that support the findings of this study can be made available on request from the corresponding author. Request for data will be evaluated and responded to in a manner consistent with the specific language in the study protocol and informed consent form (ICF), Based on the language within the IRB approved ICF for the Apple Heart and Movement Study, we are unable to share sensor data collected in the study. Similarly, code for all data analyses may be available upon request from the corresponding author. Requests for code will be evaluated and responded to manner consistent with policies intended to protect participant confidentiality and language in the study protocol and ICF.
REFERENCES
Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A Framework for Self-Supervised Learning Speech Representations, October 2020. URL http:://aaaaae. org/abs/ 220///abb/00.... 2006. 11477. arXiv:2006.1 1477 [cs, eess].
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. data2vec: General Framework for Self-supervise, Learning in Speech, Vision and Language, April 2022. URL http:: org/ abw./as//a// 03555. mrXiv:2202.03555 [es].
Hubert Banville, Omar Chehab, Aapo Hyvärinen, Denis-Alexander Engemann, and Alexandre Gramfort. Uncovering the structure of clinical EEG signals with self-supervised learning. Journal of Neural Engineering, 18(4):046020, March 2021. ISSN 1741-2552. doi: 10. 10.1088/1741-2552 abcal8. URL https:/ / doi org. 10. 1088 Publishing.
abcal8, Publisher: IOP
Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: Casaamce-nnaarinn---Cviaaii Regular- ization for Self-Supervised Learning, January 2022. URL 04906. wwXiv:2105.04900 [cs].
Florian Bordes, Randall Balestriero, and Pascal Vincent. Towards Democratizing Joibt-Emeeddinng Self-Supervised Learning, March 2023. URL pttp:://axxiv.org/bbbbt.t 0188. 01986. arXiv:2303.0198 [cs].
Kristen Bush, Daniel R. Kivlahan, Mary B. McDonell, Stephan D. Fihn, Katharine A. Bradley, and for the Ambulatory Care Quality Improvement Project (ACQUIP). The AUDIT Alcohol Consumption Questions (AUDIT-C): An Effective Brief Screening Test for Problem Drinking. Archives of Internal Medicine, 158(16):1789-11 1795, September 1998. ISSN 0003-9926. doi: 10.1c/n/e http:///....////acaoaaaeaaa
16.116..78,, URL https:
158 16.1789.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A Simple Framework for Contrastive Learning of Visual Representations, June 2020. URL http: http::////xxiv.o.//bbbb.o 2002. 95709. arXiw:2002.05709 [cs, stat].
Xinlei Chen, Saining Xie, and Kaiming He. An Empirical Study of Training Self-Supervised Vision Transformers, August 2021. URL http://arxiv.orgg/ wtt.oo/gg/abb///aaaaeae//aaa 220./2227....00.. 02057.
arXiv:2104.02057 [cs].
Joseph Y. Cheng, Hanlin Goh, Kaan Dogrusoz, Oncel Tuzel, and Erdrin Azemi. Subject-Aware Contrastive Learning for Biosignals, June 2020. URL http: arriv. wwwpiiivorg//aaae wwg/abs /2007. 04871. arXiv:2007.0487 [cs, eess, stat].
10

---

Published as a conference paper at ICLR 2024
Joshua C. Denny, Joni L Rutter, David B Goldstein, Anthony Philippakis, Jordan W. Smoller, Gwyanne Jenkins, Eric Dishman, and The All of Us Research Program Investigators. The "All of Us" Research Program. New England Journal of Medicine, 381(7):668-67)) 676, August 2019. ISSN 0028-4793. doi: 10. (006//N/Jss11809999 URL ht.tps: www. ne org/ /tgg/doi/// do / 1056 /E.6/NNEMar- 1 809937. Publisher: Massachusetts Medical Society.
Jacob Devlin, Ming- Mei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, May 2019. URL org/abs ab///b///// 111011111.0 04805. arXiv:1810.04805 [cs].
Nathaniel Diamant, Erik Reinertsen, Steven Song, Aaron D. Aguirre, Collin M. Stultz, and Puneet Batra. Patient contrastive learning: A performant, expressive, and practical approach to electro- cardiogram modeling. PLOS Computaational Biology. 18(2)e1000862, February 2022. ISSN URL https: / /jorrnalssaaaa plos org/ 1553-7358. doi: wwt.:ccleeddi....111.m. 1371 journal pcbi 1009862. Publisher: Public ploscompbicn //aartiilleee Library of Science.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, June 2021. URL http: / arxiv. abs 2010. 11929. arXiv:2010.11929 [cs].
Quentin Garrido, Randall Balestriero, Laurent Najman, and Yann Lecun. RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rank, February 2023. URL http: /awp/v/ravvvv//.///////a./...a wtt//ba//////ee 2210. C2885. arXiv:2210.02885 [cs].
Ramin Ghorbani, Marcel J.T. Reinders, and David M.J. Tax. Self-Supervised PPG Representation Learning Shows High Inter-Subject Variability. In Proceedings of the 2023 8th International Conference on Machine Learning Technologies, pp. 127-132, Stockholm Sweden, March 2023. URL https: //////dll /dl acm.
ACM. ISBN 978- 40044000--999-9 doi: org/doi /10. 1 1 145 41/358988338333 01119988333899999 3589902.
Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. ImageBind: One Embedding Space To Bind Them All, May 2023, URL
http:/ // //aaaaiivvv../////aa//aaaa//aaaa attpaaawww.aaaa////////aaaaaaa/a wwtp:/rrgabbb...ooooooo 2305.00665...... arXiv:2305.055: .0566665555 [cs].
Bryan Gopal, Ryan Han, Gautham Raghupathi, Andrew Ng, Geoff Tison, and Pranav Rajpurkar. 3KG: Contrastive Learning of 12-Lead Electrocardiograms using Physiologically-Insiire Aug- mentations. In Proceedings of Machine Learning for Health, 156-167. PMLR, November 2021. wtt.psss......p.../....mmmmm
URL https: http///toceeingng.... mlr wwwssssvssse https////1gg/goa1111111 gww.g/pplllloooo 21aa html. ISSN: 2640-3498.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self-supervised Learning, September 2020. URL http: /tappv///rxxvioo/////
abs/20065 07733. arXiv:2006.07733 [cs, stat].
Haraldur T. Hallgrímsson, Filip Jankovic, Tim Althoff, and Luca Foschini. Learning Individualized Cardiovascular Responses from Large-scale Wearable Sensors Data, December 2018. URL abs 18/188122 1812 01 696. arXiv.1812.016%6 [cs, stat].
http:://rxxt wwwpoogg/////oo/////o/o/eeeeeeee
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recog- nition, December 2015. URL httt:://axxi....t org. abs/1512 03385. arXiv:1512.003: [cs].
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models, October 2021. URL atttp://arxivg httpg///ab//200.. 09685. arXiv:2106.09683 [cs].
Brian Kenji Iwana and Seiichi Uchida. An empirical survey of data augmentation for time series clas- sification with neural networks. PLOS ONE, 1627)4,0, July 2021. ISSN 1932-6203. doi: /o.r37//ju 022.4444 URL https: //so/r//jjoalss plos wrg/ploson...o.oooooooooooooooo. article7id=10 111.1111111111111111111111111111 1.111//journa /journal pone. 0254841. Publisher: Public Library of Science.

---

Published as a conference paper at ICLR 2024
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling Laws for Neural Language Models, January 2020. URL http:/ /arxivivvv/v./////v///a//////iia wwwgabs////eoooooorot...oe..eert at/.a/ass//o.... 08361. arXiv:2001.08361 [es, stat].
Dani Kiyasseh, Tingting Zhu, and David A. Clifton. CLOCS: Contrastive Learning of Cardiac Signals Across Space, Time, and Patients, May 2021. URL htt.tt////rrzziworr///bbba/
13249.
MrXiv:2005.3324 [cs, eess, stat].
Demetres Kostas, Stéphane Oeel--Oelleetee, and Frank Rudzicz. BENDR: Using Transformers and a Contrastive Self-Suue Learning Task to Learn From Massive Amounts of EEG Data. Frontiers in Human Neuroscience, 15:653659, June 2021. ISSN 1662-5161. doi: 10.3389/fnhum. 2021 653659. URL https: : /www. www.oneeeei...o.rrrrtieeeeeeeeee 2021. 653659/ full.
338./ Enhum.
Jiewei Lai, Huixin Tan, Jinliang Wang. Lei Ji, Jun Guo. Baoshi Han, Yajun Shi, Qianjin Feng, and Wei Yang. Practical intelligent diagnostic algorithm for wearable 12-lead ECG via supervuue learning on large-scale dataset. Nature Communications, 14(1):3741, June 2023. ISSN 2041- URL https: / /www. nature. www.re.com/eaaa..e.ea. Number: Publisher: Nature Publishing Group.
1723. doi: 10.
111-677023-397722
Xiang Lan, Dianwen Ng. Shenda Hong, and Mengling Feng. Intra-Inter Subject Self-supervised Learning for Multivariate Cardiac Signals, September 2021. URL http: http://awwww..........///
2109. 08908. 7XXv:2109.08900 [cs, eess].
Han Liu, Zhenbo Zhao, and Qiang She. Self-supervised ECG pre-training. Biomedical Sig- nal Processing and Control, 70:103010, September 2021. ISSN 17468094. doi: 10.1016/ 2.bspc.2021.000010 URL https: http:s///innkighhhtee elsevier. com//etrrive//i.. $1746809421006071
Laurens van der Maaten and Geoffrey Hinton. Visualizing Data using t-SNE. Journal of Machine Learning Research, 986): 2008. ISSN 1533-7928. URL http://jmir. org/ html.
papers/v2s http://////aaa/eaeaeeeeeeeeeeeee
Calum A. MacRae. Apple Heart & Movement Study, Clinical trial registration NCT04198194, clini- caltrials.gov July 2021. URL submitted: November 13, 2019.
Temesgen Mehari and Nils Strodthoff. Self-supervised representation learning from 12-lead ECG data. Computers in Biology and Medicine, 141:105114, February 2022. ISSN 00104825. doi:
abs wtwpg///a///.././//////gg/ wttpbooedd.2221.05555 1021.105114. URL http:// http:://aaxvvvvo//
arXiv:2103. 12676 [cs, eess, stat].
Bhaskar Mitra and Nick Craswell. An Introduction to Neural Information Retrieval. Foundations and Trends@ in Information Retrieval, 13(1):1-126, December 2018. ISSN 1554-0669, 1554-0677, doi:
10.1561/150000061. URL https: // /wwww//ww/.//////////////////// www.nowwwll..... howpuubisseers. com INR- 61..
Publisher: Now Publishers, Inc.
Mostafa Neo Mohsenvand, Mohammad Rasool Izadi, and Pattie Maes. Contrastive Representa- tion Learning for Electroenccephalogaan Classification. In Proceedings of the Machine Learn- ing for Health NewrIPS Workshop, pp. 238-253. PMLR, November 2020. URL https:
//ppoceedings... mlr wtt.igss.............mmem v1 /136/ hwt.///mannna......mmmm html. ISSN: 2640-3498.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation Learning with Con- trastive Predictive Coding, January 2019. URL http:/pp//aaaw...rrr//////.0...0
arXiv:1807.03744 [cs, stat].
OpenAL. GPT-4 Technical Report, March 2023, URL arXiv:2303.08774 [cs].
Maxime Oquab, Timothée Darcet. Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez. Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Pao Huang, Shang -en Li, Ishan Misra, Michael
12

---

Published as a conference paper at ICLR 2024
Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning Robust Visual Features without Supervision, April 2023. URL http::///aar owwpeiiww..rr///oooooo org/ aba//bb//23// 001.3.4..11..... arXiv:2304.07193 [es].
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision, February 2021. URL http: http://wwwwwwor/...e///o abs 2103. 00020. arXiv:2103. 10..000200000000 [es].
Andrea H. Ramirez, Lina Sulieman, David J. Schlueter, Alese Halvorson, Jun Qian, Francis Rat- simbazafy, Roxana Loperena, Kelsey Mayo, Melissa Basford, Nicole Deflaux, Karthik N. Muthu- raman, Karthik Natarajan, Abel Kho, Hua Xu, Consuelo Wilkins, Hoda Anton-Culver, Eric Boerwinkle, Mine Cicek, Cheryl Clark, Elizabeth Cohn, Lucila Ohno-Machado, Sheri D. Schully, Brian K. Ahmedani, Maria Argos, Robert M. Cromin, Christopher O' O'Donnell, Mona Fouad, David Goldstein, Philip Greenland, Scott J. Hebbring. Elizabeth W. Karlson, Parinda Khatri, Bruce Korf, Jordan W. Smoller, Stephen Sodeke, John Wilbanks. Justin Hentges, Stephen Mockrin, Christopher Lunt, Stephanie A. Devaney, Kelly Gebo, Joshua C. Denny, Robert J. Carroll, David Glazer, Paul A. Harris, George Hripesak, Anthony Philippakis, and Dan M. Roden. The All of Us Research Program: Data quality, utility, and diversity. Pat- terns, 3(8):100570, August 2022. ISSN 3899. doi: 10.10 10.10166..aatte:2222....1. 2022.2..022..00577.. URL https: www. ncbi. mmm. www.ii...n.n www.ggo///pooo art www.eseeeeeee000/ee
Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. In 2007 15th European Signal Processing Conference, pp. 606-610, September 2007.
Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, and Hervé Jégou. Spreading vec- tors for similarity search, August 2019. URL http::/arxiv.
03198.
arXiv:1806.03199 [cs, stat].
Pritam Sarkar and Ali Etemad. Self-supervised ECG Representation Learning for Emotion Recogni- tion. IEEE Transactions on Affective Computing, 13(3):154111555 July 2022. ISSN 1949-3045, 2371-9850. doi: 10. W19//AFFC.2000.300111.1 03898. arXiv:2002.03898 [cs, eess, stat].
URL attp://azae org. 2202 2002.
lan Shapiro, Jeff Stein, Calum MacRae, and Michael O'Reilly Pulse oximetry values from 33,080 participants in the Apple Heart & Movement Study. npj Digital Medicine, 6(1):1-13, July 2023.
ISSN 2398-6352. doi: 10. 10.3/08/s41 146-023-00851--66 URL https: //ts://wwwz www. nature. com/ articles 4 74 -23-00851 6.
Number: Publisher: Nature Publishing Group.
Dimitris Spathis, Ignacio Perez- Pozzeloo Soren Brage, Nicholas Wareham, and Cecilia Mascolo. Self-supervised transfer learning of physiological representations from free-living wearable data. In Proceedings of the Conference on Health, Inference, and Learning, pp. 69-78, April 2021. doi: 10. 445533400499.4518666 URL http: httpaaivvvvo... .rgg absp:/as/200000 12121. 12X [cs, cess].
Mingxing Tan and Quoc V. Le. EfficientNer: Rethinking Model Scaling for Convolutional Neural Net- works, September 2020. URL http:://axix/ www.s/iii.....ooooooooooo abs 11946. 11Xiv:190S.11945
[cs, stat].
Chi lan Tang, Ignacio Perez-Pozuelo, Dimitris Spathis, and Cecilia Mascolo. Exploring Contrastive Learning in Human Activity Recognition for Healthcare, February 2021. URL http:///aae org/aba abs///b//at 20112 11542. 1. [cs, eess].
Xian Wu, Chao Huang, Pablo Roblesgranda, and Nitesh Chawla. Representation Learning Variable Length and Incomplete Wearable-Senson Time Series, May 2020. URL abs/2002 03595. arXiv:2002.03595 [cs, eess, stat].
Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tie- Yan Liu. On Layer Normalization in the Transformer Archi- tecture, June 2020. URL http:// arwii. .rg. www./as..//...oooooo./oo..oo.aaa 2002. 04745. arXiv:2002.04745 [cs, stat].
13

---

Published as a conference paper at ICLR 2024
Hang Yuan, Tatiana Plekhanova, Rosemary Walmsley, Amy C. Reynolds, Kathleen J. Maddi- son, Maja Bucan, Philip Gehrman, Alex Rowlands, David W. Ray. Derrick Bennett, Joanne VcVeigh, Leon Straker, Peter Eastwood, Simon D. Kyle, and Aiden Doherty. Self-supervised learning of accelerometer data provides new insights for sleep and its association with mortal- ity, July 2023. URL https: : / www.. medrxiv wwwg///ootett././/oo/ooo////eooe 10tp:////...10000..00.. 23292251v1. Pages: 3023.07.07.2322222.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow Twins: Self-Supervised Learning via Redundancy Reduction, June 2021. URL. http: http::/arxww..rr/aab//////////// 03230. arXiv:2103.03230 [cs, q-bio].
14

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 SUPPLEMENTARY TRAINING DETAILS
Pre-training framework, encoder architectures and other implementation details: Pseudo-code of our pre-traie trainingg framework is in Algorithm and it is visually shown in Fig. 3. Also, brief description of our encoder architecture is shown in Fig. 4. For BYOL, the prediction head multi-layer perceptron with one hidden layer of 1024 units. We observed that batch normalization was necessary to prevent collapse for BYOL, and for consistency, all our models had batch normalization in their projection and prediction (if any) heads. Throughout this study, we used constant momentum update rate of 0.99 for our pre-training framework and BYOL. For all model training in this study (unless otherwise stated), we used batch size of 256, and the initial learning rate was 0.001 (0.00025 for BYOL), and we used step learning rate scheduling for a faster convergence. Regarding batch size in our pre-training, while early contrastive self-supe learning works have shown that larger batch size is necessary to improve performance (Chen et al., 2020; 2021), follow up works removed this dependency by changing the pre-training (Grill et al., 2020; Zbontar et al., 2021), and a recent work has shown that it is possible to train SimCLR on ImageNet with smaller batch sizes without an important drop in performance (Bordes et al., 2023). We also did not see meaningful change in performance with larger batch sizes in our early experiments. We did not comprehensively tune our augmentation module probabilities, we started with preliminary set of probabilities in early experiments based on prior work (Cheng et al., 2020; Tang et al., 2021), for instance, cut out was shown to be the most effective augmentation in (Cheng al., 2020) and we assigned higher probability to it. We did not tune these probabilities afterwards and the only change made was to make the ECG augmentation probabilities stronger (2x probability) allow for more mismatch between positive pair representations. For the ID-ResNet and ID-ViT encoder architectures in 5.2.4, we almost used similar training details as explained for ID-EfficienNNet in 4.3, with the only difference that linear learning rate warm up from 50% of max learning rate for the first 10 epochs was employed for training the ID- encoder to ensure a more stable and optimal training (Xiong al., 2020). Our ID-ResNet was adapted from ResNet-34 with residual connections (He et al., 2015) for 1D time-series, and our ViT had a 6-layer convolutional neural network for token embedding (receptive field around 178 input samples), resulting in 60 tokens with 256 dimensions, followed by a 8-layer transformer with 8 attention heads and 1024 MLP feedforward dimension, and finally global average pooling output tokens to get the final 256-D embedding. In our experiments, observed that convolutional neural network token embedding, as used in prior models for audio signals (Baevski al., 2020; 2022) outperformed linear token embeddings used in the original ViT work (Dosovitskiy et al., 2021).
A.2 SUPPLEMENTARY DATASET DETAILS
AHMS demographics distribution:
The distribution of demographic variables in AHMS is shown for the participants that opted to answer the survey questions regarding each of age. BMI (derived from weight and height), biological sex, and ethnicity in Fig. 5.
Representative PPG/ECG signals: Representative examples of processed PPG and ECG signals shown in Fig.6.
AHMS Survey: AHMS survey is formed of multiple questionnaires which participants fill out multiple times over the course of their participation in the study; for all the analyses in this study, we used participants' latest entry before 01/28/2022. Tables 5 and 6 contain AHMS survey questions about medical conditions and medications, respectively, in addition to the corresponding target labels used in Fig. Table includes AHMS survey questions about drinking and smoking habits, and Table 8 defines our logic to summarize these questions into binary labels for the related targets used in Fig. 1.
A.3 SUPPLEMENTARY RESULTS
Linear probing evaluation of targets from AHMS survey questions: Table 9 includes the linear probing evaluation of targets from AHMS survey questions shown in Fig. 1, using PPG embeddings. ECG embeddings and baseline features.
15

---

Published as a conference paper at ICLR 2024
Comparison of downstream performance for different pre-training frameworks: Table 12 includes the downstream evaluation of age, biological sex and BMI for our pre-training framework and our variation of SimCLR and BYOL.
Effect of single augmentation functions in pre-training: Figure 7 represents the effect of single augmentation functions in pre-training for both PPG and ECG modalities.
A.4 SUPPLEMENTARY DISCUSSION
Predicting demographic variables via representation learning from biosignals in prior work: Prior work have looked into predicting certain demographic variables via representation learning from biosignals: 1) Using activity, sleep, and heart rate, BMI was predicted with ROC AUC of 0.697 (with the binary class cut-off at 30 kg/ m2 similar to us), which is significantly smaller than that with our PPG/ECG embeddings in Table 2. Also, age was predicted with ROC AUC of 0.701 (with the binary class cut-off at years) which is not directly comparable to our evaluation due to different cut-offs (Hallgrimsson et al., 2018). 2) A prior work pre-trane pre-trained models using wearable accelerometer and heart rate data free-living conditions (Spathis et al., 2021), and predicted age, BMI, and biological with ROC AUC of 0.676, 0.694, and 0.934, respectively, where the binary class cut-off to the median for continuous variables such as age and BMI. While the biological sex prediction is more accurately predicted with PPG and ECG embeddings Table the age and BMI predictions are not directly comparable due to different factors including different cut-off for binary labels and different dataset distribution, e.g., the distribution of the age AHMS (Fig. 5) different from the dataset used in this prior work (age ranging from 35 to 65). 3) Using the heart rate time-series data, biological sex was predicted with 0.703 and 0.672 Micro and Macro score (Wu et al., 2020) these reported numbers are lower than that for PPG embeddings (0.973 and 0.969, respectively) and for ECG embeddings (0.892 and 0.869, respectively). This prior work also reports age prediction which is not directly comparable us due to different class buckets.
16

---

Published as a conference paper at ICLR 2024
Recorded biosignal RP
Augmented blosignal RR Txv
Embedding Representation [2
Bt 128xl
Attract positive pairs
Encoder
m
MA
Encoder
Repel negative saiiss
Representation Subpace
Encoder
Attract
Encoder
H
positive
paira
Augmentation module
:LP Projection head
Figure 3: High-level visualization of our pre-training framework shown for mini-batch containing participants and segments. Augmented views of recorded biosignals are passed through an encoder, followed by an MLP projection head to get the representations. The representations are used to calculate the contrastive loss which attracts the positive pairs while repelling the negative pairs. Positive pairs are formed different segments of the same participant. The momentum updates and KoLeo regularization are not shown on the figure.
Algorithm 1: Pseudocode of our pre-training framework in PyTorch-like style.
E, Eim: online and momentum encoder networks, including the MLP projection heads
m: momentum rate
temperature
lambda weight for KoLeo regularization
loss_cont: contrastive loss
loss_reg: KoLec regularization penalty
initialize momentum encoder with enline encoder weights t.r.parmms f.params
training loop
for x1, x2 in loader: load batch of positive pairs from N participants
get the augmented views of the segments
x2 aug(x1) (x1), aug (x2)
get the representations from the online and nomentum encoders hi, h2 f(x1), f(x2)
him, h2m = fm(x1), f.m(x2)
stop gradients for momentum encoder
h1.m, h2.m hi.m.detach (), h2.. detach ()
calculate the rogularized InfoNCE loss (after 12-normalization of embeddings inside loss_cent and loss_reg)
loss = lcss_cont (h1, h2_m, t) + loss_cont (h2, hlm,, t)
(loss reg (h1) loss_rege (h2))
backprop through the online encoder
potimizer..erro.raadd backwaackarrd))
optimizer.step
momentum update for the momentum encoder
m. am..params.. data m f.m. params data + (1 m) f.params. data
17

---

Published as a conference paper at ICLR 2024
b
Input segment
Embedding
MBCanv1D
1DAvgPool
Conv1D
Conv1D D
Sigmold
BatchNorm
MBConv1D
BatchNorm
1DAvgPool
Cenv1D
Swish
Conv1D
BatchNorm
BatchNorm
BatchNorm
Conv1D
Swissh
Conv1D
Swish
Conv1D D
Figure 4: Our EfficientNet-stylee encoder architecture, adapted from (Tan & Le, 2020) for time-series input. a. Encoder architecture with convolutional blocks shown Conv batch normalization as BatchNorm, Swish activation as Swish, mobile inverted bottleneck block as MBConv ID, average pooling as Doogl b. The internal architecture MBConv ID, where Sigmoid activation is shown as Sigmoid and asterisk represents element wise multiplication.
a
b
C
10
20000
Biological sex
Female
Count
5000
Count
Count (log
Male
10000
10
2500
20
40
80
20
40
60
80
Asian
Black
Hispanic
Other
White
Age (years)
BMI (kg/m²)
Ethnicity
Figure 5: The distribution of demographic variables AHMS shown for self-reported a. age, BMI, biological sex and ethnicity.
a
2
Channel 1
Channel 2
PPG (a.u.)
Channel 3
Channel 4
2
10
Time (9)
4
ECG (a.u.)
2
Time (s)
Figure representative example for the processed segments shown for a. PPG, b. ECG, in arbitrary units (a.u.) for 10 seconds.
18

---

Published as a conference paper at ICLR 2024
b
114
Smooth effective rank
104
Smooth effective rank
102
112
100
110
Full augmentation module
augmentation module
108
Add Gaus. noise
Channel permute
out
Magnitude warp
Time warp
Add Gaus. noise
Cut out
Magnitude warp
Time warp
Augmentation
Augmentation
Figure 7: The effect of single augmentation functions in pre-training for PPG, b. ECG. For each augmentation function, we kept in isolation and probability in our augmentation module, and repeated pre-training. We then calculated and compared smooth effective rank with other augmentation functions and the full augmentation module.
19

---

Published as a conference paper at ICLR 2024
Table AHMS survey questions about medical conditions. The main question is in form 'Have you ever been diagnosed with any of the following conditions?" and participants can answer "Yes" or 'No' or 'I prefer not to answer' or 'I don't i know'. The question for vision and hearing loss is different, which we explicitly mention in the corresponding rows. Third column indicates the number of left out participants for evaluation the reason for variations is that for each target we exclude participants whose answers were 'I prefer not to answer' 'I don't 't know" missing.
Target label
Medical condition
N (test)
Heart attack
Heart attack (myocardial infarction)
16,832
Heart disease
Coronary heart disease or angina pectoris
16,715
Blood pressure
High blood pressure (hypertension)
16,574
Stroke TIA
Stroke (cerehral hemorrhage, cerebral thrombosis) transient
16,850
chemic attack (ministroke)
Afib
Atrial fibrillation
16,585
Heart rhythm
Heart rhythm prohlem other than atrial fihrillation
16,428
Pacemaker
Pacemaker
16,902
Artery disease
Peripheral artery disease
16,613
Heart failure
Heart failure
16,851
Diabetes
Diabetes
16,770
Cholesterol
High cholesterol
16,499
Arthritis
Arthritis
16,598
Hip/Knee
Hip or knee replacement
16,932
Lower back
Low back disorder other chronic hack defect
16,682
Neck disorder
Neck disorder other chronic neck defect
16.739
Osteoporosis
Ostcoporosis
16,722
Asthma
Asthma
16,812
Chronic bronchitis
Chronic bronchitis, chronic obstructive pulmonary disease, or em-
16,818
physema
Allergy
Rhinitis, hay fever, eye inflammation, dermatitis, food allergy or
16,729
other allergy (allergic asthma excluded)
Kidney
Kidney problems
16,789
Thyroid
Thyroid disease
16,706
Cancer
Cancer
16,836
Liver
Cirrhosis the liver
16,848
Urinary
Urinary incontinence
16,804
Neuropathy
Neuropathy
16,589
Depression
Depression
16,457
Anxiety
Anxiety disorder
16,380
Hearing
Do you have hearing loss?
15,476
Vision
Do you have vision loss?
16,295
20

---

Published as a conference paper at ICLR 2024
Table 6: AHMS survey questions about medications. The main question form of 'Do you currently take any of the following types of medications?" and participants can answer "Yes" or "No" or prefer not to answer'. The formatting for the medications is similar to their presentation in the tudy, but may not exactly match the format in the study application. Third column indicates the number of left out participants for evaluation the reason for variations is that for each target we exclude participants whose answers 'I prefer not to answer' missing. Third party trademarks used herein are trademarks of their respective owners.
Target label ACE-inhibitors
Medications
N (test)
ACE-inhibitors or ARBs (for blood pressure) such as captopril, enalapril, lisinopril, losartan, ramipril, valsartan
11,043
Anti-anxicty
Anti-anxicty aids such as alprazolam (Xanax"), clonazepam (Klonopin" "aoopiin",,, clorazepate (Tranxene"), diazepam (Valium ) or lo- razepam (Ativan")
11,047
Anti-psychotics
Anti-psychotics such as haloperidol (Haldol"), aripiprazole (Abilify risperidone (Risperdal"), quetiapine (Seroquel" ) olan- zapine (Zyprexa"), clozapine (Clozaril"), lurasidone (Latuda ) Anticoagulants (bload thinners) such as warfarin (Coumadin") apixaban (Eliquis 30 betrixaban (Bevyxxa") ), dabigatran (Pradaxa ) edoxaban (Lixiana"), rivaroxaban (Xarelto")
11,076
Anticoagulants
11,075
Antidepressants
Antidepressants such as amitriptyline (Elavil"), bupropion (Wellbutrin " citalopram (Celexa duloxetine (Cymbalta") esc- italopram (Lexapro"), fluoxetine (Prozac"), paroxetine (Paxil"), mirtazapine (Remeron")) )) sertraline (Zoloft"), or venlafaxine (Effexor"))
11,073
Antiplatelets
Antiplatelets (blood thinners) such aspirin, clopidogrel (Plavix ), prasugrel (Efffient)), or ticagrelor (Brilinta")
11,073
Beta-blockers
Scta-blockers (for blood pressure or heart rhythm) such atenolol (Tenormin bisoprolol (Zebeta"), carvedilol (Coreg ). labetalol, metoprolol (Lopressorr" Toprol-XI. nadolol (Corgard l nebivolol (Bystolic propramolol (Inderal ) or sotalol (Betapace
11,039
Blood pressure med.
Other medications for lowering blood pressure such clonidine, hydralazine, minoxidil, or sacubitril/valsartan (Entresto
11,010
Calcium channel block-
Calcium-channel blockers (for blood pressure or heart rhythm) such amlodipine (Norvase") diltiazem, or verapamil
11,019
ers
Chemotherapy
Certain types of chemotherapy such carboplatin, cisplatin, oxali-
11.084
platin, vineristine, or vinblastine
Diureties
Diureties (water pills) such chlorthalidone, furosemide (Lasix P)
11,067
hydrachlorothiazide, or spironolactone
Opioid painkillers
Opioid painkillers such as codeine, fentanyl, hydrocodone, hydro- morphone (Dilaudid"), meperidine (Demerol") morphine, oxy- codone, Percocet". Vicodin" 3
11,090
Painkillers
Non-steroidal anti-inflammatories (painkillers) such as as- pirin, celecoxib (Celebrex") " diclofenac (Cambia ibuprofen (Motrin " or naproxen (Aleve")
11.069
Sleep medication.
Sleeping aids such as eszopiclone (Lunesta") )) zaleplon (Sonata ).
11,057
or zolpidem (Ambien*)
21

---

Published as a conference paper at ICLR 2024
Table 7: AHMS survey questions about drinking and smoking habits. These are standardized questions from the AUDIT-C questionnaire (Bush et al., 1998) and All of US research program (Denny et al., 2019; Ramirez al., 2022).
#
Question
Answer choices
Q1
your entire life, have you least I drink of any kind of alcohol, not count-
"Yes' Vess//No'/"1 don't know I'I prefer not answer"
ing small tastes sips?
Qib
[If yes Q1] How often did you have drink containing alcohol the past year?
'Never' Dever'/'Moneee M'Montlly less" less//"wo four time a month Tww three times week' / or more times a week'/'l prefer not to an- swer'
Qle
[If yes yes Q1] On a typical day when you drink, many drinks you have?
27'3 4''5 6'7"7 97'
more'/'l prefer not to answer'
Q2
Have you smoked at least 100 cigarettes in your entire life?
Yes'/"No"71 don't know 'II prefer not to
answer"
Q2b
[If Yes Do not know Q2] Do you
'Every day' dom' days 2 Not all'''l pre- fer not to answer'
now smoke cigarettes every day, some days, or not at all?
Table 8: Our logic for defining drinking and smoking targets from questions in Table 7. Third column indicates the number of left participants for evaluation the reason for variations that for each target we exclude participants whose answers did not conclude in our binary 'yes' or 'no" mappings or were missing.
Target label
Logic
N (test)
Active alcohol user
"Yes": answer to QI is 'Yes' and answer Qlb 'Two three
16. 236
times a week' bock'/'Four Four or more times week'
'No': answer to Q1 is 'No', answer to Qlb is 'Never Never'/'Momi /'Motthly Tess'/'Two four time month'
Active smoker
'Yes': answer Q2 'Yes' and answer Q2b 'Every
16,214
day' day'/"Sonn /'Som days'
'No': answer to Q2 is 'No", and answer to Q2b is 'Not at all'
22

---

Published as a conference paper at ICLR 2024
Table Linear probing evaluation of targets shown in Fig
Name
ROC AUC
ROC AUC
ROC AUC
(PPG embeddings)
(ECG embeddings)
(Baseline features)
ACE-inhibitors
0.821
0.773
0.786
Active alcohol user
0,730
0,611
0,623
Active smoker
0,853
0.729
0.704
Afib
0,818
0.818
0.777
Allergy
0,659
0.631
0,652
Anti-anxiety
0,736
0,639
0,629
Anti psyyhhyhosicss
0,808
0.709
0.701
Anticoagulants
0.811
0.785
0.775
Antidepressants
0.837
0.656
0.658
Antiplatelets
0.791
0.756
0.781
Anxiety
0.785
0,688
0.708
Artery disease
0,889
0.866
0,880
Arthritis
0.785
0.746
0.782
Asthma
0,650
0.595
0.618
Beta-blockers
0,801
0.747
0.685
Blood pressure
0,819
0.769
0.770
Blood pressure med.
0.746
0.714
0.707
Caleium-channel blockers
0,816
0.746
0.740
Cancer
0.799
0.754
0.802
Chemotherapy
0,689
0.605
0,643
Cholesterol
0.751
0.716
0.734
Chronic bronchitis
0.759
0.687
0,668
Depression
0.756
0.662
0.673
Diabetes
0.866
0.796
0.804
Diuretics
0.766
0.744
0.734
Hearing
0.733
0.698
0.729
Heart attack
0,865
0,834
0,816
Heart disease
0,874
0.843
0.844
Heart failure
0,802
0.839
0.776
Heart rhythm
0.695
0.694
0.635
Hip/Knee
0,853
0,814
0,849
Kidney
0.696
0.669
0,652
Liver
0.764
0.741
0.690
Lower back
0.694
0.666
0.673
Neck disorder
0.727
0.699
0.713
Neuropathy
0,807
0.786
0,750
Opioid painkillers
0.778
0,692
0,610
Osteoporosis
0.844
0.818
0,848
Pacemaker
0.897
0.874
0.813
Painkillers
0.619
0.585
0.612
Sleep apnea
0,805
0,751
0.779
Sleep medication
0.677
0.598
0.554
Stroke or TIA
0,778
0.732
0.731
Thyroid
0.760
0.728
0.769
Urinary
0,801
0.754
0.795
Vision
0.661
0.631
0.651
Table 10: Downstream performance evaluation of PPG and ECG embeddings in predicting age,
biological sex and BMI of participants, when the number of pre-training segments in PPG similar to that for ECG in Table 1.
Positive pair
Prediction task
PPG
ECG
AUC (pAUC) t
MAE
AUC (pAUC) 1
MAE 1
Age classification
0.974 (0.907)
0.916 (0.763)
Age regression
3.38
6.33
Participant level (main)
BMI classification
0.908 (0.730)
0.797 (0.612)
BMI regression
2.54
3.72
Sex classification
0.989 (0.947)
0.951(0.841)
23

---

Published as a conference paper at ICLR 2024
Table 11: Downstream performance evaluation PPG and ECG embeddings in predicting age. biological sex and BMI of participants, using two types of positive pair selection strategies during pre-training: 1) participant level, 2) segment level, calculated at segment- granularity, where each segment contributes one and only sample in the downstream training/evaluation.
Positive pair
Prediction task
PPG
ECG
AUC (pAUC) f
MAE
AUC (pAUC) 1
MAE +
Age classification
0.961 (0.858)
0.891 (0.739)
Age regression
4.28
7.61
Participant level
BMI classification
0.850 (0.674)
0.723 (0.559)
(main)
BMI regression
3.33
4.18
Sex classification
0.961 (0.870)
0.904 (0.737)
Age classification
0.822 (0.657)
0.748 (0.606)
Age regression
8.43
10.39
Segment level
BMI classification
0.672 (0.555)
0.649 (0.526)
BMI regression
4.50
4.52
Sex classification
0.762 (0.592)
0.806 (0.647)
Table 12:
Downstream performance evaluation of PPG and ECG embeddings in predicting age. biological sex and BMI of participants, using different pre-training frameworks.
Pre-training
Prediction task
PPG
ECG
AUC (pAUC)
MAE
AUC (pAUC) 1
MAE 1
Age classification
0.976 (0.907)
0.916 (0.763)
Age regression
3.19
6.33
Ours
BMI classification
0.918 (0.750)
0.797 (0.612)
BMI regression
2.54
3.72
Sex classification
0.993 (0.967)
0.951 (0.841)
Age classification
0.976 (0.910)
0.912 (0.756)
Age regression
3.22
6.60
Ours (no KoLeo)
BMI classification
0.915 (0.746)
0.797 (0.612)
BMI regression
2.58
3.82
Sex classification
0.992 (0.962)
0.946 (0.822)
Age classification
0.975 (0.905)
0.914 (0.759)
Age regression
3.22
6.41
SimCLR
BMI classification
0.913 (0.744)
0.796 (0.612)
(our variation)
BMI regression
2.59
3.79
Sex classification
0,992 (0.961)
0.947 (0.816)
Age classification
0.969 (0.889)
0.900 (0.734)
Age regression
3.82
6.72
BYOL (our variation)
BMI classification
0.893 (0.721)
0.779 (0.593)
BMI regression
2.88
3.90
Sex classification
0.987 (0.946)
0.939 (0.805)
24


Reviews:
[{"review_id": "JFXPHINAB3", "reviewer": "Reviewer_4W8A", "summary": "This paper introduced foundation models for wearable sensor data using a large-scale population dataset of ECG and PPG signals. Its embeddings demonstrate generalization across an array of downstream tasks related to personalization and health inferences.", "questions": "- The paper could better address previous research. For example, it claims that previous works employed biosignals recorded in clinical or controlled experimental settings, however, both [1] and [2] used large-scale _free-living_ datasets in the wild. \n\n- Regarding the augmentations, it is not clear whether assigned probabilities are found through hyper-parameter tuning or heuristics. I point the authors to this paper for further experimental decisions about the order and impact of these augmentations [3].\n\n- There are very limited details about the availability of the dataset. Should we just assume it is private? Are there any public datasets that we could replicate (some of) the results of the paper? I would appreciate any discussion around these points.\n\n- Given the number of participants, the paper could also attempt to increase the sequence length of the signals and assess whether longitudinal/day-level temporal dynamics can impact downstream tasks.\n\n- The linear probing results and performance analysis should be put in context to previous works like [4], [2], and [5].\n\n- The parameter size of the final model seems quite low considering the dataset size and number of participants. The paper could justify the word \"foundation model\" in its title by scaling up the experiments. For example, it should have been very exciting to find the Chinchilla-optimal parameter size for this sort of data. I would appreciate any discussions here around this topic, are there overfitting issues with bigger models? What about different architectures like Transformers?\n\n[1] Yuan, H., Chan, S., Creagh, A. P., Tong, C., Clifton, D. A., & Doherty, A. (2022). Self-supervised learning for human activity recognition using 700,000 person-days of wearable data. arXiv preprint arXiv:2206.02909.\n\n[2] Spathis, D., Perez-Pozuelo, I., Brage, S., Wareham, N. J., & Mascolo, C. (2021, April). Self-supervised transfer learning of physiological representations from free-living wearable data. In Proceedings of the Conference on Health, Inference, and Learning (pp. 69-78).\n\n[3] Tang, C. I., Perez-Pozuelo, I., Spathis, D., & Mascolo, C. (2020). Exploring contrastive learning in human activity recognition for healthcare. arXiv preprint arXiv:2011.11542.\n\n[4] Wu, X., Huang, C., Robles-Granda, P., & Chawla, N. V. (2022). Representation learning on variable length and incomplete wearable-sensory time series. ACM Transactions on Intelligent Systems and Technology (TIST), 13(6), 1-21.\n\n[5] Hallgrímsson, H. T., Jankovic, F., Althoff, T., & Foschini, L. (2018). Learning individualized cardiovascular responses from large-scale wearable sensors data. arXiv preprint arXiv:1812.01696.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Important domain with very limited available models \n- Solid results across an impressive array of downstream tasks\n- Careful tuning and experimentation considering the idiosyncrasies of the data", "weaknesses": "- Some missing references and links to previous works\n- Lack of discussion around scaling up the proposed models \n- No discussion around model/data release", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JNidp3azEW", "reviewer": "Reviewer_UQ5r", "summary": "This paper employs self-supervised learning (SSL) on a large dataset comprising two types of data: photoplethysmography (PPG) and electrocardiogram (ECG) collected from the Apple Watch. The self-supervised learning is carried out on a comprehensive dataset, encompassing a vast span of 141K subjects.\nSubsequently, the authors investigate various modules within the SSL framework to glean practical insights. They also elaborate on the effects of certain design choices, such as the selection of positive-negative pairs and data augmentation strategies.\nThe authors conducted a thorough evaluation and ablation studies for the foundational model they developed. The learned embeddings demonstrate predictive power across a wide array of downstream tasks, such as predicting demographic features and survey questions.", "questions": "- Even if the authors intend to display only aggregated information, it would be beneficial for them to include a representative visualization of both PPG and ECG signals. This would provide readers, especially those without a healthcare background, with a clearer understanding.\n\n- In Figure 2, both PPG and ECG embeddings demonstrate good separability based on subject IDs. However, I'm curious if two subjects with similar demographic attributes should be distinctly separated.\n\n- While ECG and PPG are pivotal for evaluating physiological status, there are also numerous other parameters to consider, such as HRV, heart rate, and possibly activity levels. The authors might wish to discuss this aspect further.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is well-articulated. The authors clearly presented the proposed method, experimental setup, and analysis.\n\n- To the best of my knowledge, this represents the first attempt at training a self-supervised learning (SSL) foundational model for PPG and ECG data on such a grand scale. The results from this process offer valuable scientific insights.\n\n- The authors conducted an exhaustive evaluation of the pretrained models. These pretrained embeddings were assessed against more than 50 diseases.", "weaknesses": "- There are potential concerns on the technical methodology front. While this study is the product of extensive training and evaluation, much of the methodology draws from pre-existing studies. Although there are several SSL studies tailored for time-series data, particularly in the realm of biosignals in healthcare, the authors did not extensively compare different model architectures. Readers might be keen to discern whether biosignal SSL performance is more contingent upon scale or the model architecture itself.\n\n- While this study encompasses two modalities, it seems that the authors have considered them in isolation. Pretrained models have shown effectiveness across varied modalities like images and language. It would be beneficial for the authors to delve deeper into this aspect.\n\n- The pretrained embedding for PPG appears to encapsulate more information than its ECG counterpart. This raises a question: given that PPG is passively sampled and ECG is actively collected by users, could this disparity in data collection methods influence such an outcome? Additionally, conventional clinical diagnoses often rely on 12-lead ECG or periodic information like HRV derived from ECG. It would be valuable if the authors could elucidate more why the ECG embedding doesn't seem as informative as the PPG.\n\n- Lastly, the authors note that positive pairs are drawn from the same individual. However, a person's PPG and ECG patterns can vary based on different conditions or circumstances. It might be more insightful for the authors to determine positive pairs by taking additional attributes into account.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BGAFnpicFB", "reviewer": "Reviewer_FYDu", "summary": "The paper presents a significant advancement in the domain of health monitoring using wearable devices, specifically focusing on the tracking of biosignals, namely photoplethysmography (PPG) and electrocardiogram (ECG). Recognizing the potential of these biosignals, the authors highlight a major challenge: the lack of large, curated datasets with medically annotated labels for developing neural network-based biomarkers. To circumvent this, they leverage the Apple Heart and Movement Study (AHMS) and harness a self-supervised learning framework for training foundation models on PPG and ECG data from approximately 141,207 participants over a span of three years. This self-supervised approach integrates a participant level positive pair selection, stochastic augmentation, and a regularized contrastive loss optimized through momentum training. The authors demonstrate that these pre-trained models encode substantial information related to participant demographics and health conditions. Notably, this work distinguishes itself as the pioneering effort in building foundation models for PPG and ECG using large-scale data sourced from consumer wearables, as opposed to traditionally smaller datasets from clinical settings. The potential applications of these models are vast, with implications for enhancing wearable device capabilities, reducing reliance on labeled data, and ultimately benefiting users' health.", "questions": "**Concern Regarding Dataset Availability**:\nThe dataset from the Apple Heart and Movement Study (AHMS) is central to this research, offering immense value to the broader scientific community. However, the paper doesn't clarify if the dataset will be open-sourced upon acceptance. The release of this dataset, along with the associated models and code, is pivotal for reproducibility and further research.\n\nIf the authors don't have ownership of the dataset, detailed instructions on sourcing it would be essential. I kindly request clarity on this matter, as it will influence my final assessment of the paper.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strengths:\n1. **First Work on Foundation Models for Wearable Biosignals**:\nThe research stands out as the pioneering effort to develop foundation models specifically for biosignals—photoplethysmography (PPG) and electrocardiogram (ECG)—collected via wearable devices. Such biosignals offer a treasure trove of biological and cardiac information, which can be instrumental in monitoring users' overall health. The convenience of wearable devices combined with the potential of these foundation models paves the way for continuous health tracking without disrupting daily routines, potentially leading to the early detection of health issues.\n\n2. **Extensive Evaluation Using Linear Probing**:\nThe authors conducted a comprehensive evaluation of the trained models using linear probing for a plethora of tasks. This includes gender classification, age prediction, and both classification and regression for Body Mass Index (BMI). Furthermore, they delved into predicting variables extracted from questionnaires, providing a holistic understanding of the models' capabilities. The utilization of smooth effective rank as an evaluative measure underscores the robustness of the evaluation.\n\n3. **Participant Level vs. Segment Level Positive Pairs**:\nAn intriguing facet of the research was the comparison between participant level and segment level positive pairs. This distinction is crucial because the granularity at which positive pairs are chosen can significantly impact the model's performance. The exploration of this dimension offers valuable insights into the optimal approaches for training foundation models on biosignals.\n\n4. **Benchmarking with Other SSL Methods**:\nTo validate the efficacy of their approach, the authors benchmarked their models against established self-supervised learning (SSL) methods, such as SimCLR and BYOL. Such a comparative analysis not only situates the research within the broader landscape of SSL but also provides tangible metrics to gauge the relative performance of their models.\n\n5. **Curation of a Large Dataset**:\nOne of the paper's major contributions is the meticulous curation of a vast dataset from the Apple Heart and Movement Study (AHMS) for the training of foundation models. With data spanning approximately 141,207 participants over three years, the effort required to curate, clean, and prepare such a dataset for effective training cannot be understated. This endeavor not only underscores the thoroughness of the research but also sets a precedent for future studies aiming to leverage large-scale datasets for health applications.", "weaknesses": "Areas for improvement:\n\n1. **Exploration of KoLeo Regularization**:\nAn area of potential exploration is the specific impact of the KoLeo regularization on the model's performance. Ablation studies that incrementally remove or vary the strength of KoLeo regularization could provide clarity on its role and efficacy. Such an analysis would help in understanding whether the regularization is crucial for the model's success, and to what extent it contributes to the overall performance. This is particularly important as the unique characteristics of biosignals may interact with regularization techniques differently compared to other domains.\n\n2. **Inclusion of All Results**:\nTransparency and completeness in research reporting are essential for reproducibility and peer review. Therefore, it is recommended that the authors include all results, particularly those referenced as \"results not shown\" within the main body of the paper or the appendix. Having access to these results would allow the scientific community to fully evaluate the findings, methodologies, and claims made within the paper. It would also enhance the credibility and utility of the work for those looking to build upon it.\n\n3. **Augmentation Impact Analysis**:\nThe paper would benefit from a deeper dive into the effects of various augmentation techniques on the performance of the biosignal models. Unlike image data, where the impact of different augmentations is well-studied, the domain of biosignals remains relatively unexplored in this aspect. An appendix providing detailed results and analysis of how different augmentations influence the learning process would be invaluable. This could include which augmentations contribute most to model robustness or performance, and any augmentation-specific phenomena observed with biosignals. Given the novelty of the field, such insights could be highly influential for future research in biosignal analysis.\n\n4. **Demographic details missing**: The paper would benefit greatly from a more detailed presentation of demographic information related to the participants whose data underpin the foundation models. Such information is essential for assessing the diversity and representativeness of the dataset, which in turn, influences the generalizability of the model across various populations. The current omission of granular demographic details leaves a gap in understanding the scope of the model's applicability. It is recommended that the authors include statistics on age, gender, ethnicity, and other pertinent demographic factors. This would not only enhance the transparency of the research but also allow for a more nuanced evaluation of the model's performance across different demographic groups.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8izZ4TM5k2", "reviewer": "Reviewer_EBBD", "summary": "The proposed work presents a self-supervised learning (SSL) method for foundation models training using a large, longitudinal, multi-year dataset of unlabelled PPG and ECG samples recorded on Apple Watch devices. The performed experiments showed that the pre-trained models can readily encode participant demographics, conditions and medication. The introduced SSL framework incorporates various techniques, such as stochastic augmentation module or participant level positive pair selection proved to behave better than segment level selection. The work flows logically with a comprehensive analysis of how well PPG and ECG embeddings encode participants' information and ablation study of various parameters used in the work. The motivation of the work is clear and the method was compared to other existing techniques proving its robustness.", "questions": "1) Would you be able to include results for PPG model trained on a smaller dataset (similar number of segments as ECG)? I believe this would be useful for comparing the behavior of the method given similar amount of data.\n2) Have you thought about using both modalities in one model? Do you think that modulating, e.g., ECG with PPG embeddings would improve accuracy? Such approaches are useful when one modality is less descriptive than the other, so I was interested in learning more about your thought of how this would apply to this use case.\n3) How easily will it be to improve the current labels given that they were acquired using self-reported metrics and tracing it back seems impossible?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The work proposes pre-training of foundation models to a new domain using real data, acquired in uncontrolled environment, acquired over a  long period of time, what is a good indicator of how robust the method is.\n2) Comprehensive analysis that includes comparison with other methods, linear probing to analyze how well both embeddings encode participants' information and a detailed evaluation of which one is more predictive, ablation study including analysis of visual representations after dimensionality reduction, validation loss and dispersion ratio. \n3) Presented results support claims made in the work, showing robustness of the introduced method and ability to encode participants' information.\n4) Overall the soundness and completeness of the work is good in my opinion.", "weaknesses": "1) It would be great to include a figure representing the model showing encoder, MLP head and other implementation details.\n2) What was the reason for choosing the InfoNCE loss vs. e.g., the normalized temperature-scaled cross entropy loss (NT-Xent) from SimCLR?\n3) Usually large batch sizes and more learning steps are beneficial in SSL, have you experimented with even bigger batch sizes than 256?\n4) Could you clarify the reason for selecting this specific encoder and specific embedding sizes, have you experimented with other models?\n5) Are you going to make the dataset public?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)