HiMAE: Hierarchical Masked Autoencoders Discover Resolution-Specific Structure in Wearable Time Series
Authors: Simon A. Lee
Cyrus Tanade
Hao Zhou
Juhyeon Lee
Megha Thukral
Md Sazzad Hissain Khan
Keum San Chun
Baiying Lu
Migyeong Gwak
Mehrab Bin Morshed
Viswam Nathan
Md Mahbubur Rahman
Li Zhu
Subramaniam Venkatraman
Sharanya Arcot Desai
Venue: ICLR 2026 Poster (2026)
Keywords: SSL
Wearables
Interpretability
Inductive Bias
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: We propose a lightweight SSL objective that competes with much larger transformer Foundation models that also serve as an interpretability tool.

Abstract: Wearable sensors provide abundant physiological time series observations, yet the resolution at which we should extract features for downstream tasks remain unclear. We hypothesize that temporal resolution is a fundamental axis of representation learning, with different clinical and behavioral outcomes relying on features at distinct scales. To test this resolution hypothesis, we introduce HiMAE (Hierarchical Masked Autoencoder), a self-supervised framework that combines masked autoencoding with a hierarchical convolutional encoder–decoder. HiMAE produces multi-resolution embeddings across its intermediate layers that enable systematic evaluation of which temporal scales carry predictive signal, transforming resolution from a hyperparameter into a probe for interpretability. Across classification and generative benchmarks, HiMAE consistently outperforms state-of-the-art foundation models that collapse scale, while being orders of magnitude smaller. Due to the convolution based design choices behind HiMAE, the model is also compact enough to run entirely on-device, achieving sub-millisecond inference on smartwatch-class CPUs for true edge inference. Together, these contributions position HiMAE as both an efficient self supervised learning method and a discovery tool for understanding how time resolution contributes to downstream task alignment.

Published as a conference paper at ICLR 2026
HIMAE: HIERARCHICAL MASKED AUTOENCODERS DISCOVER RESOLUTION-SPECHEG STRUCTURE IN WEARABLE TIME SERIES
Simon A. Lee1.2.*, Cyrus Tanade', Hao Juhyeon Lee', Megha Thukral', Md Sazzad, Hissain Khan Keum San Chun Baiying Lu Migyeong Gwak Mehrab Morshed Viswam N Md Mahbubur Rahman' Li Zhu', Subramaniam Venkatraman',
Sharanya Arcot Desai'.
DDigital Health Team, Samsung Research America Department of Computational Medicine, UCLA Work completed during Al Residency s simonlee 7118g. ucla.edu
ABSTRACT
Wearable sensors provide abundant physiological time series observations, yet the resolution at which we should extract features for downstream tasks remain unclear. We hypothesize that temporal resolution fundamental axis of rep- resentation learning, with different clinical and behavioral outcomes relying on features at distinct scales. To test this resolution hypothesis, we introduce HiMAE (Hierarchical Masked Autoencoder), self-supervised framework that combines masked autoencoding with hierarchical convolutional encoder-decoder. HiMAE produces multi-resolutie embeddings across its intermediate layers that enable systematic evaluation of which temporal scales carry predictive signal, transform- ing resolution from hyperparameter into probe for interpretability. Across classification and generative benchmarks, HiMAE consistently outperforms state- of-the-art art foundation models that collapse scale, while being orders of magnitude smaller. Due to the convolution based design choices behind HiMAE, the model is also compact enough to entirely on-device, achieving aub-milissccon in- ference on smartwatch-clee butions position HiMAE as both an efficient self supervised learning method and a discovery tool for understanding how time resolution contributes to downstream task alignment.
CPUs for true edge inference. Together, these contri-
INTRODUCTION
Wearable sensors have emerged as a primary modality for continuous health monitoring, provid- ing access to rich physiological and behavioral signals in free-living settings (Erturk et al. 2025). Despite their ubiquity, the utility of wearable signals for machine learning in healthcare remains
poorly understood. Unlike images (Dosovisskiy et al. 2021 [Simonyan al.] [014]] Zhou et al.
2015 Petsiuk et al. 2018) or text (Brown et al. 2020 Li et al.] 2016: Sundararajan et al.) 2017
Arras al., 2017), physiological time series rarely admit obvious visual cues that map cleanly to clinical outcomes, leaving open fundamental questions about which features carry predictive value. particularly unresolved issue concerns temporal resolution: should models operate at single uni- versal resolution, or do different health outcomes depend on resolution -ssolution-specific structure? Clinically actionable events can arise on second-level timescales, requiring representations that both capture fine-grained temporal patterns and support real-time inference under the computational constraints of wearable devices. We hypothesize that resolution is not nuisance parameter but a fundamental axis of physiological representation learning. We refer to this as the resolution hypothesis, which posits that temporal granularity governs predictive performance in clinical and behavioral tasks. In this framing, "resolution" denotes the effective temporal context which representations formed from fine-scale waveform morphology to coarse-scale dynamics spanning the whole se- quence.

---

Published as a conference paper at ICLR 2026
Input
Masking
Reconstruction
Encedori
000000
Docoder
Generative
Evalluation
Downstream
Resolution
Embeddings
Linear Probes
Metrics
Staging
Different Receptive Scales
HiMAE
Classification
M
Abnormal
Cardiovascular Conditions
MM
Regression Task
Trains different Inser probas different shapa
ombedding vectors.
Figure 1: HiMAE pre-training and evaluation pipeline. (1) Physiological sequences are split into
temporal patches. (2) Selected patches masked randomly or contiguously. (3) A U-Net-style CNN encoder-decoder reconstructs missing values, with loss applied only to masked regions. (4) Multi-resolution embeddings feed linear probes for classification and regression benchmarking. (5) Three categorized fask-lists are evaluated.
From algorithmic perspective, much of the field defaults transformer-haset architec- tures (Vaswani et al.] (017). implicitly assuming that flexibility and capacity outweigh inductive bias. Yet wearable signals, while long in sequence length, are often generated by few latent pro- cesses driven by biological mechanisms and captured through only a handful of sensor modalities. In this sense they are wiwdimmnsional and highly structured. This raises the possibility that trans- formers may not only overfit but also obscure resolution-sper speciffc structure, rather than expose it. By contrast, hierarchical convolutional biases offer a natural mechanism for aligning architectures with the resolution hypothesis, capturing both local detail and long-range dependencies in principled way. This motivates a re-examination of architectural design choices for self-supervised learning (SSL) raw physiological time series.
In this work, we address these challenges by introducing HiMAE (Hicrarchical Masked Autoen- coder), self-supervised pretraining framework for wearable time series that directly operational- izes the resolution hypothesis (Figure T.. HiMAE combines the masked autoencoding paradigm with 1D physiological signals by coupling patch- patch-masking objectives (Wang al. 2023) with U-Net-inspired encoder-decoder (Ronneberger et al., (015). Crucially, HiMAE produces multi- resolution embeddings, with each level of the hierarchy corresponding ularity. This design enables systematic interrogation of which resolutions carry predictive signal, while simultancously yielding lightweight, efficient representations. Beyond its architectural advan- tages, HiMAE allows us to benchmark the resolution hypothesis across 14 classification. Our results reveal resolution speciffc structure in wearable signals that not readily identifiable by human ex- perts, offering new insights into both representation learning and the interpretability of physiological time series time domain.
distinct temporal gran-
RELATED WORK
Self-Supervised Pretraining Objectives for Wearable Signals Wearable devices equipped with mhotoplethysmoorappe (PPG), electrocardiograply (ECG), and accelerometry generate long, multi- channel time series encoding diverse physiological and behavioral phenomena, including cardiovas- cular dynamics (Castaneda al. 2018). activity patterns (Yuan al. 2024: Xu et al. 2025), ) sleep cycles (Li et al. data streams abundant, and predominantly unlabeled, making them well suited for large-scale
et al. 2024; Logacjov et al.) 2025). and other latent processes. These
2

---

Published as a conference paper at ICLR 2026
self supervupe learning (Kaplan et al. 2020 Bommasani et al. 2024).
2021 Zhou et al. 2024 Liang et al.
SSL has become the dominant paradigm for wearable time-series representation learning, given the scarcity of labeled data and the ubiquity of unlabeled signals in free-living settings. Among SSL
strategies, masked autoencoding has emerged as central approach, inspired by its success in vi- sion (He ct al., [222] Vaid et al. (023) and language modeling (Devlin et al. 2019). The method randomly occludes patches of the signal and tasks the model with reconstructing them, encouraging representations that capture latent physiological structure and temporal regularities (Zhang et al., 2022a Kong et al. [023] Thukral et al.] 2026 Lee et al.). Recent large-scale efforts, most notably Google's LSM series Narayaaswamy et al., 2024| Xu et al.) 2025), rely heavily masked au- toencoding, establishing it as pretraining standard for multi-modal wearable datasets. Yet despite its effectiveness for local pattern recovery, vanilla masked autoencoding often struggles to capture multi-resolution features unless coupled with explicitly hierarchical architectures.
In parallel, contrastive learning enforces invariance by pulling semantically similar samples together in latent space while pushing dissimilar ones apart (Schmitt & Kuljanin 2008: Jaiswal et al. 2020). The central challenge for wearables is defining positive and negative pairs without labels. One so- lution is participant-level contrastive training, where samples from the same individual are positives and samples from different individuals are negatives, an approach adopted in Apple's ECG and PPG foundation models (Abbaspourazad et al., (203)) and closely related to the SimCLR frame- work (Chen et al.] 2020b). Other domain-specific innovations define pairs through physiologi- cal priors: PaPaGei leverages PPG morphology (Pillai et al.) 2024), while SleepFM extends the paradigm across EEG, ECG, and EMG to enforce cross cross-modes modal consistency (Thapa et al. Additional embedding- regularizers, such differential entropy constraints (Jing et al. 2021 Abhaspourazad et al. 2023), further enrich learned representations. However, contrastive methods highly sensitive to augmentation heuristics (which are rarely physilogically meaningful), com- putationally intensive, and limited in interpretability, providing little insight into which temporal structures are preserved.
2024).
HiMAE departs from both flat masked and contrastive approaches in two ways. First, instead of relying on a single-scale reconstruction or augmentation heuristics, HiMAE couples masked autoen- coding with hierarchical encoder-decoder that integrates information across resolutions, treating temporal scale as an explicit dimension of representation. Second, by extracting embeddings at multiple scales and probing them independently, HiMAE transforms SSL from a pretraining mech- anism into a discovery tool: it directly tests which temporal resolutions carry predictive signal for downstream tasks. In doing so, HiMAE preserves the efficiency of masked autoencoding while introducing interpretability absent in contrastive or flat masked objectives.
Multi-scale Learning The emphasis on resolution awareness connects naturally multi-scale learning, where modeling temporal signals across multiple granularities has emerged as power- ful inductive bias. In vision, multi-scale architectures such as pyramidal CNNs and hierarchical attention enable models to integrate fine-scale edges with coarse semantic structures, substantially improving recognition and generation in 2D (Wang et al. 2016 Yang et al. 20166] Liu et al. 202 a Kusupati et al. 2024 Liu et al. 2024) and 3D (He et al. 2022b).
2017 Ghadai et al. 2019
Zhang et al.
In time series, multi-scale methods are fewer but increasingly influential. N-HiTS (Challu et al. 2022] improves long-horizon forecasting by allocating capacity across frequencies via hierarchical interpolation. Pyraformer (Liu al. 2022) leverages pyramidal attention capture dependencies over tree of scales, while Scaleformer (Shabani et al. (2023) introduces iterative refinement across resolutions. Pathformer (Chen et al. 2024) further adapts pathways dynamically match input- specific temporal dynamics.
Prior multi-scale methods typically rely on fixed hierarchies or task-specific refinement stages (c.g., for forecasting), which constrains their generality. While HiMAE also inherits inductive biases from convolutional design choices (e.g.. step size, padding, kernel width), these parameters define recep- tive fields rather than dictate which scales salient. By coupling self-supervised reconstruction with these fields, HiMAE induces a hierarchy of temporal embeddings that can be probed indepen- dently.
3

---

Published as a conference paper at ICLR 2026
3 METHODS
Hierarchical Masked Autoencoders (HiMAE) HiMAE combines masked autoencoding (Baldi) 2012 He et al. (2022) with I-D physiological time series by coupling patch-masking objective with a U-Net-style convolutional encoder-decoder (Ronneberger et al. 2015). Given an input sequence IR we partition it into N L P non-overlapping patches of length P. A binary mask m 0,1 is sampled from Bernoulli distribution with parameter r, indicating the masking ratio. Masked indices are selected uniformly at random without replacement, expanded to match temporal resolution m' E 0, 1", and applied to the sequence, yielding ã procedure removes substantial context, forcing the model to infer higher-order dependencies. In addition to random masking, we also employ contiguous masking, which adjacent patches are removed to mimic sensor dropout similar to recent protocols showing benefits (Xu et al., (025). Both regimes are interleaved during pretraining to promote robustness across reconstruction settings.
(1 mn'. This masking
Architecture The encoder fe is hierarchical ID CNN composed of residual convolutional blocks with stride-2 convolutions that downsample the temporal resolution by half at each stage, expanding the receptive field that deeper layers capture long-range dependencies while shallow layers retain local detail. Each residual block consists of two convolutions with kernel size 5, batch normaliza- tion (loffe & Szegedy 2015), and GELU activations (Hendryeks & Gimpell 2023), along with projection shortcut when input and output dimensions differ. The decoder mirrors this structure with transposed convolutions for upsampling and incorporates skip connections from encoder lay- ers, concatenating intermediate features to restore fine- one-grrined temporal structure. All convolutions are standard 1D operations defined over temporal windows, and striding handles subsampling di- rectly. Intermediate activations use GELU, while the final layer applies tanh nonlinearity so that outputs C are bounded 1-, 1]. matching the normalized input range.
We deliberately adopt convolutional U-Net backhone rather than transformer- baased encoder for two reasons. First, physiological signals exhibit strong local dependencies governed by morphol- ogy (e.g., PPG waveform shape, ECG peaks), which are naturally modeled by finite receptive fields. Convolutions O''Shea& & Nash 2015) encode this locality directly, whereas transformers must simu- structure is intrinsic to physiology (e.g., heartbeats unfold over milliseconds, rhythms span seconds). A hi- erarchical CNN with skip connections provides an architectural bias toward such nested timescales, aligning directly with the resolution hypothesis and being orders of magnitude smaller than other proposed foundation models this space (See Figure for comparison). In contrast, transformers emphasize global mixing, which may obscure resolution-specific structure while consuming sub- stantially more compute (Table 5). This rationale motivates HiMAE's design as not only efficient but also inductively aligned with the temporal statistics of wearable signals.
late it through restricted attention, often at higher parameter cost. Second,
Resolution Probes Multi-resolution em- beddings extracted from different levels of the hierarchy are probed independently. with distinct linear classifiers trained per resolution (Alain & Bengio 2018). This design enables us to systematically evalu- ate which temporal granularity carries pre- dictive signal for downstream tasks, rather than collapsing embeddings into single latent space. Finally, choices of patch length P and kernel size were guided by ablations (Appendix Section F.I). which confirmed that P 5 and kernel size 5 yield the best balance between local fi-
Wearable Foundation Models
Multi-modal
LSM-Small (19)
LSM-Base (TO+)
SiMGUR
HIMAE Small (KiF)
Unimodal
(129)
100k
100M
params
Figure 2: HiMAE is lightweight
delity and receptive field expansion when all other hyperparameters were fixed.
Training minimizes masked reconstruction loss restricted to occluded regions: (M,E(0,
( (1m=m1)mmll where m' ensures that gradients are only computed on masked segments. This objec- Ets mi tive estimates p(z with M and O denoting masked and observed indices, preventing trivial copying of visible inputs and promoting temporally coherent, multi-scale representations.
4

---

Published as a conference paper at ICLR 2026
Effective Global Context via Receptive Field Expansion While Transformers achieve global dependency modeling via (L2) self-attention, the U-Net architecture in HiMAE approximates this behavior at O(L) complexity through hierarchical spatial contraction. In D-layer encoder, the ef- fective receptive field (ERF) at layer d grows exponentially as Ra = R ()- -)- 8;, where k is the kernel size and is the stride. By the bottleneck, the ERF encompasses a significant por- tion of the input sequence L, allowing the model to capture "global" context without the quadratic memory overhead of an attention matrix. This hierarchical aggregation acts as a multi-scale proxy for global attention: deep layers integrate coarse, long-range context, while skip connections inject high-resolution local features back into the decoder. Consequently, HiMAE simulates the commu- nicative benefits of attention through a series of local-to-global inductive biases, achieving compet- itive representation power at a fraction of the FLOPs required by vanilla VIT or Transformer-based autoencoders.
Pretraining and Evaluation Protocol PPG Sequences were sampled at f. = 100 Hz over fixed windows of T 10s (T. 1000 timesteps). 10 second windows were selected due to clinically actionable events occurring in these time scales (ECG is collected at 10s intervals in clinical settings (Shuai et al. 2016 Elgendi 2012)) and due to our interest real-time monitoring on edge devices. Each signal was divided into non-overlapping patches of length P = 5 (200 patches total), and a masking ratio r 0.8 was applied with patterns resampled per sequence and iteration to mitigate overfitting (we empirically tested this masking ratio in Appendix Section F.I with similar observa- tions made in (Narayanswamy et al. 2024)). The encoder architecture employed channel widths [16,32, 64. 128 mirrored in the decoder. Optimization was performed with Adam (Loshchilov) & Hutter 2019) (Ir = 10 weight decay = 10 10-3) 3) using a warmup-cosine schedule (10% linear warmup steps followed by cosine decay). Models trained up to 100k steps with batch size 2048 and early stopping triggered after 3 epochs without improvement similar to the protocols found in Naaaaaaswamy al.). Data splits followed 90/10 (train/validation) protocol across subjects, suring no identity overlap between pretraining and validation. Pretraining converged within 12 hours when distributing training across 4 Tesla T4 GPUs using PyTorch lightning (Paszke et al. 2019).
Pretraining datasets. We construct our pretraining corpus from approximately 80,000 hours of wearable green PPG signals, drawn from seven large-scale free world studies conducted at Samsung Research, These datasets include recordings from 47, 47,644 participants across seven distinct wearable devices, capturing broad demographic, behavioral, and hardware variability in noisy environment (See Appendix Section for ethics considerations). Although our modeling framework is modality- agnostic and can extend to other physiological signals such as clectrocardiograms (see Appendix F.2). we focus here on PPG due to its prevalence and the scale of available data (we lack the same order of magnitude of ECG compared to PPG because ECG is not passively collected). To ensure reliability, we apply a standardized preprocessing pipeline that retains only high-quality segments, filtering by Signal Quality Index (SQI). The retained signals are further refined using a bandpass filter of 0.5-8 Hz (Christiano & Fitzgerald 2003), consistent across all pretraining and evaluation studies, isolate physiologically relevant dynamics. Finally, signals are normalized to the range [- to match the output range of the tanh activation function used in our models.
4 EXPERIMENTAL DESIGN
We follow the evaluation protocol of Narayaswamy et al. (2024) and extend it into unified bench- mark suite spanning generative, and classification, along with ablations to quantify how key archi- tectural components interact with scaling. Across all experiments, our goal is not only to assess HiMAE's efficiency and transferability, but also to test the resolution hypothesis: whether predictive signal concentrates at specific levels of the hierarchical embeddings. Further analysis and results are displayed in full in Appendix Section
Model scaling and generative reconstruction. We first study HiMAE's scaling properties by mea- suring how reconstruction performance varies as a function of dataset size, number of participants, model capacity, and training compute capacity (batch size). For each axis, we systematically sub- sample or expand the relevant resource while holding others fixed, enabling us to isolate its contri- bution to representation quality. Scaling is assessed through mean squared error on masked recon- struction held out validation set, which provides direct measure of how model capacity and data availability govern loss reduction. We also squeeze in ablations in this experiment to assess how removing skip connections, and removing the hierarchal design affect scaling.
5

---

Published as a conference paper at ICLR 2026
Model Scaling Performance Across Different Axes
MAE-1D (VTT)
CNN
HiMAE (no skip)
HIMAE
Scaling with Data
Scaling with Participants
Scaling with Model Size
Scaling with Compute
0.07
0.4
0.06
Mean Squared Error
0.20
0.20
0.05
0.3
0.15
0.15
0.04
0.2
0.10
0.10
0.03
0.02
0.1
0.05
0.05
0.01
0.0
0.00
0.00
0.00
102
10
10
105
10a
10a
104
105
104
10
10
10?
10
256
1024
2048
Training Hours
Participants
Model Parameters
Batch Size
Figure 3: HiMAE exhibits superior scaling across axes. Mean squared error decreases most rapidly for HiMAE as data, participants, model size, and compute scale on a held out validation set. Ablations without skip connections confirm that both the hierarchical design and skip pathways are helpful for generative performance. Grey lines indicate multiple runs whereas colored lines are average performance.
To complement this aggregate view, we also evaluate generative performance under three increas- ingly challenging reconstruction regimes defined the LSM papers (Narayanswamy al.) Xu al. 2025): (i) random imputation, where patches are masked at random uniformly: (ii) temporal inter- polation, where contiguous spans are removed to simulate sensor dropout; and (iii) temporal ex- trapolation, where future spans are occluded and predictions must rely solely past context. We compute the mean squared error (MSE) for these evaluations.
Classification To assess downstream transferability and adaptability, we benchmark HiMAE on 12 binary classification tasks drawn from labeled datasets fully disjoint from our pretraining sources. We organize these into three groups: cardiovascular outcomes, sleep staging, and abnormal labo- ratory prediction. Cardiovascular outcomes, provide the most established benchmarks, with well- documented links between PPG and clinical endpoints (Shabaan et al., 2020). These include hyper- tension detection, and arrhythmia-related events such as Premature Ventricular Contractions (PVCs) detections, typically identified via electrocardiograms (ECGs). Sleep staging is another task we in- clude which is of high interest, given the demand for wearables to track fine-grained sleep states despite the temporal and physiological complexity of the task (Imtiaz) 202T Thapa al., 2024] Birrer et al.) 2024). Laboratory predictions, on the other hand, serves as a discovery setting, testing whether PPG contains sufficient biomarker information separate abnormal from healthy labs- an open question compared to patient-record benchmarks where such signals are more explicit (Kolo) (ct al. 2024; McDermott et al., 2025). Together, these canonical and exploratory tasks form a spec- trum that enables comprehensive evaluation of representation quality across diverse digital health applications. All tasks are described in greater detail in Appendix Section D
We evaluate HiMAE against two complementary classes of baselines. The first comprises estab- lished self-supervised representation learning methods for time series, including SimCLR (Chen et al. 2020b), DINO (Caron et al.) 2021). Masked Siamese Networks (MSN) (Assran et al. 2022), and a ViT-based ID masked autoencoder that follows the LSM training protocol of (Narayanswamy let al.)) The second class consists of state-of-the-an time-series and wearable foundation models. This includes PaPaGei, a leading foundation model for PPG (Pillai et al.) 2024), evaluated both using its publicly released Bell Labs checkpoint (PaPaGei-BL and variant retrained training corpus (PaPaGei-SRae -RA). We additionally benchmark aganst Chronos (Ansari et al. 2024), large-scale time-series foundation model, and the hierarchical Swin Transformer (Liu et al. 2021b), configured to match the LSM setting for controlled comparison. Further implementation details for all baselines are provided in Appendix F All models are evaluated using a standard linear probing protocol, in which the pretrained encoder is frozen and a linear classifier is trained on top of the learned representations. Performance is reported using AUROC as the primary metric of discrimi- native ability. For every architecture, we expose the full sequence of embeddings along the temporal
our pre-
htte:s//aandd..ogg/eerrr/////11

---

Published as a conference paper at ICLR 2026
Table 1: Performance on generative benchmarks. Mean squared error and for random imputa- tion, temporal interpolation, and temporal extrapolation at varying missingness levels. Bold outline indicates best performing model.
Random Imputation (MSE)
Temporal Interpolation (MSE)
(c) Temporal Extrapolation (MSE)
Method
50%
80%
Method
30%
50%
Method
Mcan Kill
0.224
0.225
01.225
Mcan Hill
0.225
0.2234
0.225
Mcan Fill
0.225
0.225
0.225
NN Fill
0.023
0.137
0.188
NN Fill
0.151
0.466
0.491
NN Fill
0.450
0.193
0.529
0.045
0.183
0.153
Lincar
0.377
0.385
Lincar
0.526
MAE-ID (VIT)
0.017
0.029
0.041
MAE-ID
0.173
0.203
0.299
MAE-ID (VE))
0.255
0.282
0.356
CNN
0.017
0.030
0.040
CNN
0.165
0.199
0.278
CNN
0.248
0.269
0.343
HIMAE
0.012
0.017
0.026
0.126
0.173
HIMAE
0.194
0.211
(d) Rundom Impotation (R')
(e) Temporal Interpolation (R")
Temporal Extrapolation (R))
Meihod
30%
50%
80%
Method
31%
50%
80%
Method
319
50%
805
Fill
Mcan Fill
Miaan
Fill
NN
0.807
(1399
0.164
NN
-1,022
-1.080
-1.182
NN
-1.000
-1.191
Linear Int.
0.799
0.631
0.320
Linear lnt.
-0.691
-0.719
-0.791
Linear lnt.
-0.996
-1.164
-1.338
MAE- (VIT)
0.924
0.571
0.818
MAE-IDE (VIT)
0.224
0.094
-0.329
MAE- ID (VIT)
-0.133
-0.253
-0.582
CNN
0.924
0.567
0.822
CNN
0.260
0.112
-0.236
CNN
-0.102
-0.195
-0.524
HIMAB
0.946
0.924
0.884
0.228
0.102
dimension, rather than collapsing representations to a single summary token, ensuring that down- stream probes retain access to resolution -ppuciff information. This evaluation protocol allows us to assess whether pretraining yields representations that are both discriminative and transferable across tasks.
Resolution Hypothesis HiMAE produces embeddings at multiple temporal scales, and we probe each scale independently with linear classifiers. This allows to test whether predictive information concentrated at fine, intermediate, or coarse resolutions depending on the clinical endpoint. this way, the classification tasks serve not only as benchmarks for transfer learning, but also controlled tests of the resolution hypothesis (Receptive field lengths are described in Section C.T
5 RESULTS
5.1 SCALING AND GENERATIVE BENCHMARK
Scaling: We first examine the scaling behavior in Figure of HiMAE relative baselines across data, participants, model parameters, and compute capacity (batch size). The overall scaling trends follow conventional expectations, error decreases monotonically with additional data, participants, or compute. However, scaling with model parameters reveals a interesting insight. HiMAE achieves substantially lower loss at smaller parameter capacities, while transformers only begin to close the gap once scaled to orders of magnitude more parameters (we chose transformer parameter count based on LSM's original paper (Narayanswamy et al. ).24)). This difference reflects an inductive bias. Transformer which assume global receptive fields, appear to require considerably larger model capacity before capturing the local dynamics of the data. In contrast, HiMAE's hierarchical con- volutional structure exploits spatial and temporal locality efficiently, yielding superior performance at modest scales. This observation reinforces the importance of architectural priors in low-capacity regimes.
Turning to generative benchmarks, HiMAE consistently outperforms all baselines across random imputation, temporal interpolation, and temporal extrapolation tasks (Table I In terms of mean squared error, HiMAE achieves the lowest reconstruction eтror in every setting, in- cluding cases with heavy missingness. This advantage persists when evaluated with R², where the mean-fill baseline serves as the reference. By achieving positive R² scores even in challenging extrapolation scenarios, HiMAE demonstrates reconstruction ability beyond naive heuristies (e.g.. mean fill, nearest neighbor, linear interpolation). Together, these results establish HiMAE as a strong generative model for missing data problems, with advantages that persist across scaling regimes and input corruption patterns.
Generative:
Ablation in Table I and Figure further highlights the contributions of hierarchical design and skip connections in HiMAE. Removing either component results in increased error, in- dicating that both are crucial for effective representation learning. Nevertheless, even without these architectural elements, HiMAE variants remain competitive with larger transformer based models,
Ablations:
7

---

Published as a conference paper at ICLR 2026
Table 2: Linear probing classification performance comparison against baselines on different tasks. AUROC is reported in percent with 95% confidence intervals. The best performance is bold, the second best model is underscored. denotes p < 0.05, "* denotes p < 0.01 from two-sided z-test comparing HiMAE with the second-best model.
Model
Pyocus (l0
Canforncator Corditons
Abtomaa Lacs
Skep Staging
Dtyee
Hopte (hie-lnitg)
Hernegooeee
Sodtem
Light
Deep
5.0
53.4 saae
5322 14
53.7 i
011 1n33)
64.4 Lond
55. i4t
523 (at2)
5320
DINO
51.7 144
52.2
41201440
408-
429ru
55.1 (an8
092
MAE (WT)
180.6
43.2 4.6
796 182
48.
00.3
59.3
HIMAR
1.3
66.8 05155
45.5
81.2 006
2011 12/6
503
603 131
(31
(68
19
18.5
Table 3: Linear probing classification performance comparison against state-of-the-art wearable and time-series foundation models. AUROC reported in percent with 95% confidence intervals. The best performance is bold, the second best model is underscored. denotes p < 0.05, ** denotes p < 0.01 from a two-sided z-test comparing HiMAE with the second- beesnd-bestt model.
Cuziioaaca Crecitono
Alrecral Lab
Slop Stazio
Tyyne (U8)
PVC
ATC
Skreogionn
Plasas
Seciurs
Prtasiaa
Wake
Light
Deep
HBM
RaParie S
5-
43 1a
24.3 m014
01.3
(27
13.5 1040
56.8 mm
17A cm
85 25
18.5 (22))
Swis Imaatomm
110.
20.5 uss
56.7 m45
HRMAR
1.5
65.8 1e
882:214
78. 1276
463
64.8"
633
81.1
6688 201
12
723 m314
56.5 255
underscoring the robustness of the approach. More importantly, the full model exhibits improved generalization across scaling axes (Appendix Section [.3), suggesting that the combination of hier- archy and skip connections facilitates better transfer as data and compute grow.
5.2 CLASSIFICATION BENCHMARKING
Classification In Tables 2 and 3 HiMAE consistently secures the majority of wins, frequently outperforming matching models that are considerably larger. This is particularly striking given that prior work has typically relied on heavy architectures to reach similar levels of performance, highlighting HiMAE's ability to capture broad spectrum of physiological features with a compact design. These outcomes emphasize the model's 's robustness when applied to structured, temporally dependent problems that demand sensitivity to subtle variations in wearable signals.
Taken together, these results position HiMAE as the most consistently strong performer across the benchmark suite. In cases where HiMAE does not place first is only ~1-2% behind the winning model. Crucially, this level of performance is achieved with substantially smaller model than competing approaches, demonstrating a favorable tradeoff between efficiency and predictive power. Rather than excelling only in isolated cases, HiMAE delivers broad, cross-domant competitiveness, suggesting that compact models, when designed with the right inductive biases, can rival or even surpass far larger architectures.
5.3 RESOLUTION SPECIFIC CLINICAL INTERPRETABILITY
The resolution hypothesis predicts that different health outcomes depend on distinct temporal gran- ularities. To test this, we analyze performance across HiMAE layers, where each layer corresponds to progressively coarser resolution. Figure reveals clear resolution-speciff structure: individual downstream tasks achieve maximal AUROC at different layers, highlighted by the red boundaries.
This layer-task alignment underscores two key insights. First, temporal resolution is not nuisance parameter but an axis of predictive structure: different outcomes are best represented at different scales (we show that collapsing an encoder decoder still has concordant results showing that our hierarchal model is not an artifact in Appendix Section [:4). Second, HiMAE naturally exposes this heterogeneity, functioning as a discovery tool for identifying the most informative resolution per task. This complements conventional interpretability methods (Amann et al., 2022] Xu et al. 2023) Lee et al.) 2025) by shifting the focus from which features drive predictions to which resolutions matter. In doing so, HiMAE operationalizes the resolution hypothesis and provides insights to tasks where the resolution needed is not entirely clear.
Clinical Interpretation
The iroolution-specifii structure discovered by HiMAE carries clinical implications that resonate with existing literature or provide new clinical insights from scientific discovery perspective. Car-
8

---

Published as a conference paper at ICLR 2026
HIMAE Layers Discover Resolution-Specific Structure Across Downstream Tasks
Corresponding Recaptive
Cardiovascuarr
Blood-related labs
Electrolytes
Sleep Staging
Field Resolutions
m
HIMAE-L1
0.38
0.73
0.52
0.25
0.70
043
0.56
0.36
Model & Layer
HIMAE-L2
0.55
0.59
0.58
0.57
HIMAE-L3
0.60
0.64
0.77
0.66
0.53
0.6
0.59
0.82
0.65
0.59
0.72
0.88
HIMAE-L4
0.63
HIMAE-L5
0.62
0.55
0.00
0.70
0.52
0.66
0.59
0.82
0.65
0.59
0.70
0.58
Hypeswooe
Free
PVC
Downstream Taska
Figure HiMAE discovers task-specific structures for downstream tasks. AUROC across layers shows that tasks rely on distinct temporal scales, highlighting HiMAE tool for discovering the most informative resolution in clinical machine learning.
diovascular and sleep-staging outcomes achieve maximal performance in later layers, which aligns with physiological understanding: cardiovascular (Zhou et al., 202T Tang al., 2025) and sleep dynamics (Patanaik et al.) 2018). which evolve gradually over longer time horizons. Capturing these slower patterns requires less temporal granularity, consistent with the notion that general trends, not transient spikes, dominate predictive structure in chronic cyclic physiological processes.
In contrast, tasks involving laboratory measurements, are a more exploratory and scientific discovery setting where there isn't much intuition on what resolution should reveal the most predictive signal. When looking at Figure particularly the blood-related biomarkers, it exhibit opti- mal performance in earlier layers corre- sponding finer temporal scales. These outcomes reflect inherently volatile phys- iological processes, where shifts in mor- phology can signal meaningful physiolog- ical change on lab measurements.
Processing
m HIMAE
Memory
HIMAE
1,21
0,0647 JFLOPs
4.8 MB
1159
Effecent Net
7.8M
0.70 gLO0%s
51,1
1.42
Swin-Tramformor
110.6
11.89 gFLOPS
423 MB
2.55
MAR-ID
11066
15.94 15.94 GFLOPS
441.3
3.16
5.4 CASE STUDIES
Figure Model efficiency and on-device inference: Sample on-device detections on Samsung Galaxy de- vice. Size, compute cost, memory footprint, and CPU latency (ms per sample, batch size 2048) measured over a 10s sequence at 100Hz.
Case Study 1: On-Device Benchmark- ing A central novelty of HiMAE that knowledge, the first SSL method compact enough to run entirely on-watch, rather than on phone-class hard- ware. We evaluate on-device PVC detec- tion on smartwatch-class CPUs sampled 100 Hz (Figure 5). HiMAE is exceptionally lightweight (1.2M parameters. 0.0647 gFLOPs, 4.8 MB) and achieves 0.99 ms latency per sample, equivalent to processing =1,010 samples/s or x22.8 hours of signal per minute of wall time. By contrast shows massive performance gains against transformer baselines, Swin-Trane Transformff (100M param- eters, 11.9 gFLOPs, 423 MB) and a MAE-ID (ViT) (1 L0M, 15.9 LFOPs, 441 MB). HiMAE also outperforms optimized models like Efficient-Net B1 (Tan & Le 2020) providing context to the latency and compactness of our model. HiMAE thus ~3 4x more efficient compared to trans- formers while fitting fully on-watch (without quantization (Jacob et al., (017)), enabling continuous, private inference at the point of signal collection. This prototype is strictly for research and is not deployed commercially.
it

---

Published as a conference paper at ICLR 2026
Case Study 2: HiMAE is adaptable in few shot settings
A central challenge in the wearable domain is that labels are scarce across tasks. Models that can adapt quickly from generic pretraining to specific detection tasks with limited supervi- sion are therefore essential. Figure 6 illustrates this setting: HiMAE pro- vides strong representations that can be adapted to diverse tasks such as PVC detection or hypertension mon- itoring with only a handful of labeled examples reflected by the shape of the learning curves on the few- shot learning experiments. By reduc- ing the supervision required to reach high performance, HiMAE enables new tasks to be supported on-device without the prohibitive cost of large curated datasets which help bolster its practical utility.
Few Shot Learning Performance
Hypertension (Lab)
0.90
0.85
0.75
0.70
0
256
512
1024 2048 Shots
all
0
256
512 1024 2048
Shots
Figure 6: Few-shot adaptation. HiMAE adapts efficiently to new wearable tasks under sparse labels indicated by curve shape over transformer baselines.
DISCUSSION
Summary. HiMAE advances wearable self supervised methods along three dimensions: (i) its flex- ible architecture is expressly designed for mustiiresolution mapping, enabling seamless adaptation across heterogeneous tasks, (ii) by aligning task-dependent resolutions with model representations, it not only optimizes predictive performance but also offers window into the temporal organization of physiological biomarkers, and (iii) by design of the compactness, it achieves the first demonstra- tion of true on-watch inference, running entirely within smartwatch-class constraints while matching or surpassing performance on far larger models. These results position HiMAE as an efficient repre- sentation learner but also framework for interrogating which temporal resolutions carry signal.
Resolution structural prior. Our findings validate the resolution hypothesis and suggest shift in how representation learning on wearables should be conceptualized. This reframing implies that representation learning for physiological signals should expose, rather than collapse, scale-specific embeddings. The layer-wise AUROC profiles in Figure 4 show that predictive performance peaks different levels of the hierarchy depending on the task, with fine-scale embeddings capturing short-lived physiological events and coarse-scale embeddings capturing slower behavioral phenom- ena. By revealing this heterogeneity, HiMAE provides empirical evidence that resolution-specife representations are essential for wearable health modeling.
From "on-device" to "on-watch." HiMAE demonstrates that convolutional hierarchies duce model size by two orders of magnitude relative to transformer-based models, enabling the first instance of true on-watch inference. This moves the deployment frontier from phone-class watch- class processors, where inference occurs exactly at the point of sensing. Beyond efficiency, this shift has consequences for privacy (data never leave the device) and for clinical viability (continuous real-time monitoring becomes feasible).
Limitations and Future Works While we focus on PPG, the principles underlying HiMAE gen- eralize to multimodal settings. Physiological signals are inherently multi-scale across modalities (e.g., ECG beats, accelerometer motion cycles, EEG rhythms), and resolution-aware architectures could expose complementary temporal signatures across them. Another limitation our work is we don't handle sequences beyond 10 second windows which could unlock another breadth of tasks. Future works also warrants a clinical validation to the discoveries made by HiMAE which could be of significant interest to the health community.
10

---

Published as a conference paper at ICLR 2026
LLM USAGE
large language model (LLM) was used to assist in refining the phrasing and structure of the manuscript. Its role was limited to improving clarity, coherence, and readability of the text based on cuuthor-provider drafts. All scientific content, experimental design, and analysis were conceived, implemented, and verified by the authors.
ACKNOWLEDGMENTS
We thank Minji Han and Rachel Choi for their expertise in UX/UT design and for crafting the spe- cialized visualizations not supported by standard Python libraries; their design contributions were essential to this work. We also thank Praveen Raja, Matthew Wiggins, Yilin Shen, and Mike Freed- man for their invaluable feedback and insightful discussions throughout the project.
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. Tensorflow: a system for large- scale machine learning. In Proceedingy of the 12th USENIX Conference Operating Systems Design and Implementation, OSDI' 16, pp. 265-283, USA, 2016. USENIX Association. ISBN 9781931971331.
Salar Abbaspourazad, Oussama Elachqar, Andrew C Miller, Saba Emrani, Udhyakumar Nallasamy, and Ian Shapiro. Lcrge-scale training of foundation models for wearable biosignals. arXiv preprint 23iv:2312.05409, 2023.
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes, 2018. http::/:: wwwpriiwww...raaeooooooe wwtgg/aa/////lr.ee 1110. 01 0164
Julia Amann, Dennis Vetter, Stig Nikolaj Blomberg, Helle Collatz Christensen, Megan Coffee, Sara Gerke, Thomas K Gilbert, Thilo Hagendorff, Sune Holm, Michelle Livne, al. To explain or not to explain? artificial Parfffiaartiiffcialll intelligence explainability in clinical decision support systems. PLOS Digital Health, 1())x0000016, 2022.
Ulzee An, Moonseong Jeong, Simon A Lee, Aditya Gorla, Yuzhe Yang, and Sriram Sankarara- man. Raptor: Scalable train-free embeddings for 3d medical volumes leveraging pretrained 2d foundation models. arXiv preprint arXiv:2507.08254, 2025.
Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, Jasper Zschiegner, Danielle C. Maddix, Hao Wang, Michael W. Mahoney, Kari Torkkola, An- drew Gordon Wilson, Michael Bohlke-Schneider, and Yuyang Wang. Chronos: Learning the language of time series, 2024. htt.http:://// /attts://arxiv/ http//aavvvvo/r//e/eee has///a///aa 2403. 07815
Leila Arras, Grégoire Montavon, Robert Müller, and Wojciech Samek. Explaining recurrent neural network predictions in sentiment analysis, 2017. URL https:://axxv..ogg//
1706.07209
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Mike Rabbat, and Nicolas Ballas. Masked siamese networks for label-efficient learning. In European conference computer vision, pp. 456-473. Springer, 2022.
Pierre Baldi. Autoencoders, unsupervised learning, and deep architectures. In Proceedings of ICML workshop on unsupervised and transfer learning, pp. 37-49. JMLR Workshop and Conference Proceedings, 2012.
Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the
dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference
New Fairness, Accountability, and Transparency. FAceT '21, pp. doi: 10. York, NY. USA, 2021. Association for Computing Machinery. ISBN 978 9781450383097,
8844499988.4442..... 1145/3442188.36 3445922. URL htt.ttttttt....... http://.....rr1...1111.....
/ttp///////////hoooo///////////0 34459922

---

Published as a conference paper at ICLR 2026
Vera Birrer, Mohamed Elgendi, Olivier Lambercy, and Carlo Menon. Evaluating reliability in wear- able devices for sleep staging. NPJ Digital Medicine, 7(1):74, 2024.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportu- nities and risks of foundation models. arXiv preprint arXiv:2108.07255 2021.
Lucía Bouza, Aurélie Bugeau, and Loic Lannelongue. How to estimate carbon footprint when training deep learning models? a guide and review. Environmental Research Communications, doi: 00.1088/5115-722/accb33 /RLhttp:
(11):115014, November 2023. ISSN 2515-7620.
7/dx. dolx.doi..om www.:oo...ggg.... 1088
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman- and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http:
//github. com/
jax-m1/
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1900 2020.
2020.
Ebubekir Buber and DIRI Banu. Performance analysis and epu vs gpu comparison for deep learning. In 2018 6th International Conference on Control Engineering & Information Technology (CEIT), pp. 1-6. IEEE, 2018.
Stephen Butterworth et al. On the theory of filter amplifiers. Wireless Engineer, 7(6))536-541) -41,, 1930.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers, 2021. URL Intps:
arxiv. wrxiv.org/abs..eo.ooooooooo wtw.vv..rggabs//11.... 14424 14294
Denisse Castaneda, Aibhlin Esparza, Mohammad Ghamari, Cinna Soltanpur, and Homer Nazeran.
A review wearable pholoplethysmography sensors and their potential future applications in health care. International journal of biosensors & bioelectronics, 4(4):195, 2018
Yong-Mei Cha, Glenn K Lee, Kyle W Klarich, and Martha Grogan. Premature ventricular contraction-induced cardiomyopathy: a treatable condition. Circulation: Arrhythmia and Elec- trophysiology, 5(1):229-236. 2012.
Cristian Challu, Kin G. Olivares, Boris N. Oreshkin, Federico Garza, Max Mergenthaler-Canseeo,, and Artur Dubrawski. N-hits: Neural hierarchical interpolation for time series forecasting, 2022.
URL https: /ttp://aaxvvvvvvv wwtp:/ggg//abo.... 2200 12886
Irene Y Chen, Emma Pierson, Sherri Rose, Shalmali Joshi, Kadija Ferryman, and Marzyeh Ghas- semi. Ethical machine learning in healthcare. Annual review biomedical data science, 4(1):
123-144, 2021.
Peng Chen, Yingying Zhang, Yunyao Cheng, Yang Shu, Yihang Wang, Qingsong Wen, Bin Yang, and Chenjuan Guo. Pathformer: Multi-scale transformers with adaptive pathways for time series forecasting, arXiv preprint arXiv:2402.05556. 2024.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PmLR, 2020a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, 2020b. URL
2002. 05709
Lawrence Christiano and Terry J Fitzgerald. The band pass filter. International economic review, 44(2):335-44 2003.
12

---

Published as a conference paper at ICLR 2026
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 4171-418 Minneapolis, Minnesota, June 2019. Association for Com- putational Linguistics. doi: 10. 10.18653/1//N19114 1423, URL https::tt//att attts::/aclanttoloog. org
9423/
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby, An image is worth 16x16 words: Transformers for image recognition at scale, 2021. URL httpt:::////m /taps:///axiii /ttpa//////aa....raaa///////////
Mohamed Elgendi. On the analysis of fingertip photoplethysmogram signals. Current cardiology reviews, 8(1):14-25, 2012.
Eray Erturk, Fahad Kamran, Salar Abbaspourazad, Sean Jewell, Harsh Sharma, Yujie Li, Sinead Williamson, Nicholas J Foti, and Joseph Futoma. Beyond sensor data: Foundation models of behavioral data from wearables improve health predictions. arXiv preprint arXiv:2507.00191, 2025.
Hany Ferdinando, Matti Huotari, and Teemu Myllylä. Photoplethysmography signal analysis to assess obesity, age group and hypertension. In 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 5572-5575. IEEE, 2019.
Chloë FitzGerald and Samia Hurst. Implicit bias in healthcare professionals: a systematic review. BMC medical ethics, 18(1):19, 2017.
Sambit Ghadai, Xian Yeow Lee, Aditya Balu, Soumik Sarkar, and Adarsh Krishnamurthy, Multi- level 3d cnn for learning multi-scale spatial features. In Proceedings of the IEEE/CVF conference conferccce computer vision and pattern recognition workshops, pp. 0 0, 2019.
Thomas D Giles, Bradford C Berk, Henry Black, Jay N Cohn, John B Kostis, Joseph L. Izzo Jr, and Michael A Weber. Expanding the definition and classification of hypertension. The Journal of Clinical Hypertension, 709):505-512, 2005.
Thomas D Giles, Barry J Materson, Jay N Cohn, and John B Kostis. Definition and classification of hypertension: an update. The journal of clinical hypertension, 11(11)661--14 2009.
Ary L Goldberger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph Mietus, George Moody, Chung- Peng, and Eugene Stanley. Physiobank, physiotoolkit, and physionet: components of new research resource for complex physiologic signals. circulation, 101(23)=e21--222) 2000.
Awni Hannun, Jagrit Digani, Angelos Katharopoulos. and Ronan Collobert. MLX: Efficient
and flexible machine learning on apple silicon. 2023. URL https://gi thub. com/
m1 explore
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition, 2015. URL/tttps://axxiv. wwtp:///wwrr.aa//1111//// wat.a.///111.1112..
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEE/CV conference on computer vision and patlern recognition, pp. 16000-16009, 2022.
Mingyi He, Bo Li, and Huahui Chen. Multi-scale 3d deep convolutional neural network for hy- perspectral image classification. In 2017 IEEE International Conference on Image Processing (ICIP), pp. 3904-3908. IEEE, 2017.
Yuting He, Fuxiang Huang, Xinrui Jiang, Yuxiang Nie, Minghao Wang, Jiguang Wang, and Hao Chen. Foundation model for advancing healtheare: challenges, opportunities and future direc- tions. IEEE Reviews in Biomedical Engineering, 2024.
13

---

Published as a conference paper at ICLR 2026
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus), 2023. URL https://
wtx.v.org/ab//166.000 08415
Syed Anas Imtiaz. A systematic review of sensing technologies for wearable sleep staging. Sensors, 2165):152, 2021.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift, 2015. URL
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient ontger-arithmetic-only inference, 2017. URL stttps://arxiv.or//aaaa 05877]
Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Dehapriya Banerjee, and Fillia Makedon. A survey on contrastive self-supervised learning. Technologies, 9(1::2, 2020.
Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. arXiv, 2021. doi: 10448555Warxxx22110009444 URL https:://arxiv..rr////
09344
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836/ 2020.
Yasin Kaya and Hüseyin Pehlivan. Classification of premature ventricular contraction in ecg, Inter- national Journal of Advanced Computer Science and Applications. 6(7). 2015.
Aleksia Kolo, Chao Pang, Edward Choi, Ethan Steinberg, Hyewon Jeong, Jack Gallifant, Jason A Fries, Jeffrey N Chiang, Jungwoo Oh, Justin Xu, et al. Meds decentralized, extensible validation (meds-dev) benchmark: Establishing reproducibility and comparability in ml for health. Machine Learning For Health Confernce, 2024.
Lingjing Kong, Martin Q Ma, Guangyi Chen, Eric P Xing, Yuejie Chi, Louis-Philippe Morency, and Kun Zhang. Understanding masked autoencoders via hierarchical latent variable models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7918-7928, 2023.
Aditya Kusupati. Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ra- manujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, and Ali Farhadi. Matryoshka representation learning, 2024. URL http::://aax...orrgg/p//o
220...225521122 13147
Simon Lee, Cyrus Tanade, Hao Zhou, Juhyeon Lee, Megha Thukral, Baiying Lu, and Sha- ranya Arcot Desai. Towards on-device foundation models for raw wearable signals. In NeurlPS 2025 Workshop on Learning from Time Series for Health.
Simon A Lee, Sujay Jain, Alex Chen, Kyoka Ono, Arabdha Biswas, Ákos Rudas, Jennifer Fang, and Jeffrey N Chiang. Clinical decision support using pseudo-notes from multiple streams of ehr data. npj Digital Medicine, 8(1):394, 2025.
Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. Visualizing and understanding neural models in NLP. In Kevin Knight, Ani Nenkova, and Owen Rambow (eds.), Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 681-691, San Diego, California, June 2016. Association for Computational Linguistics. doi: 10.18653/v 1/N55/3//1/111111 1082. URL https:///cclatthologg
Brg/N16-1033
Qiao Li, Qichen Li, Ayse S Cakmak, Giulia Da Poian. Donald L Bliwise, Viola Vaccarino, Amit J Shah, and Gari Clifford. Transfer learning from eeg to ppg for improved sleep staging from wrist-worn wearables. Physiological measurement, 42(4):044004, 2021. 2021.
14

---

Published as a conference paper at ICLR 2026
Yuxuan Liang, Haomin Wen, Yuqi Nie, Yushan Jiang, Ming Jin, Dongjin Song, Shirui Pan, and Qingsong Wen. Foundation models for time series analysis: A tutorial and survey. In Proceedings of the 30th ACM SIGKDD conference on knowledge discovery and data mining, pp. 6555-6565, 2024.
Anne-Laure Ligozat, Julien Lefevre, Aurélie Bugeau, and Jacques Combaz. Unraveling the hid- den environmental impacts of ai solutions for environment life cycle assessment of ai solutions. Sustainability, 14(9):5172, April 2022. ISSN 2071-1050. doi: 10.3390/su1409517. URL
http:// dx. doi. www.oogg11.... 3390 / su1495117
Yihan Lin, Zhirong Bella Yu, and Simon Lee. A case study exploring the current landscape of synthetic medical record generation with commercial llms. arXiv preprint 1tti:2504.146557 2025.
Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X. Liu, and Schahram Dust- dar. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In International Conference on Learning Representations, 2022. URL hctps:
//openzeview wwwp:nrevieew.et/oooroeeee wet///ormmm..d-0Ea..... dExmm I
Yun Liu, Yu-Huan Wu, Guolei Sun, Le Zhang, Ajad Chhatkuli, and Lue Van Gool. Vision trans- formers with hierarchical attention. Machine intelligence research, 21(():670-68) 2024.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021a.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows, 2021b. URL https:
//aziiv.rrg/ab//2100000 14030
Aleksej Logacjov, Kerstin Bach, and Paul Jarle Mork. Long-term self-supervised learning for accelerometer-bon based sleep-wake recognition. Engineering Applications of Artificial Intelligecce 141:109758, 2025.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2017. URL https: //axxiv.rrgaab///1..
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. URL https:
Jianwen Luo, Kui Ying, and Jing Bai. Savitzkygollay smoothing and differentiation filter for even number data. Signal processing, 85(7):1429-14333 2005.
Connor MacIsaac, Macros Nguyen, Alexander Uy. Tianmin Kong, and Ava Hedayatipour. A pro- grammable gain calibration method to mitigate skin tone bias in ppg sensors. Biosensors, 15(7): 423, 2025.
Melissa D McCradden, Shalmali Joshi, James A Anderson, Mjaye Mazwi, Anna Goldenberg, and Randi Zlotnik Shaul. Patient safety and quality improvement: Ethical principles for a regulatory approach to bias in healthcare machine learning. Journal of the American Medical Informatics Association, 27(12):202-22) 2027, 2020.
Matthew BA McDermott, Justin Xu, Teya S Bergamaschi, Hyewon Jeong, Simon A Lee, Nas- sim Oufattole, Patrick Rockenschaub, Kamilé Stankevičiūte, Ethan Steinberg. Jimeng Sun, et al. Meds: Building models and tools in a reproducible health ai ecosystem. Proceedings of the s1 ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, pp. 6243-6244, 2025.
Andrew Miller, Joseph Futoma, Salar Abbaspourazad, Christina Heinze- -emml Saba Emrani, Ian Shapiro, and Guillermo Sapiro. A wearable-basee aging clock associates with disease and behav- ior. Nature communications, 16(1):9264, 2025.
15

---

Published as a conference paper at ICLR 2026
Girish Narayanswamy, Xin Liu, Kumar Ayush, Yuzhe Yang, Xuhai Xu, Jake Garrison, Shyam A Tailor, Jacob Sunshine, Yun Liu, Tim Althoff, et al. Scaling wearable foundation models. The Thirteenth International Conference on Learning Representations.
Girish Narayanswamy, Xin Liu, Kumar Ayush, Yuzhe Yang, Xuhai Xu, Shun Liao, Jake Garrison, Shyam Tailor, Jake Sunshine, Yun Liu, et al. Scaling wearable foundation models. arXiv preprint arXiv:2410. 13638, 2024.
Keiron O'Shea and Ryan Nash. An introduction to convolutional neural networks, 2015, URL https://arxiv. http::/://axxi.or/////oo. wtt../orrr/////11..000000
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Ed- ward Yang. Zach Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library, 2019. http:::/: wwtp:/iwwwww....///////aaoaaaa/a
C1703
Amiya Patanaik, Lynn Ong, Joshua Gooley, Sonia Ancoli-Israel, and Michael WI. Chee. end-to-end framework for real-time automatic sleep stage classification. Sleep, 41(5):2sy041, 2018.
Ignacio Perez-Pozuelo, Dimitris Spathis, Jordan
Jessica Morley, and Josh Cowls. Digital phenotyping and sensitive health data: Implications for data governance. Journal of the American Medicul Informatics Association, 2809):2002-2088, 2021.
Vitali Petsiuk, Abir Das. and Kate Saenko. Rise: Randomized input sampling for explanation of black-box models, 2018. URL ptetps://arxive wwtpx/wwwwgggaa////////eea /10/6/88660000 07421 07421
Arvind Pillai, Dimitris Spathis, Fahim Kawsar, and Mohammad Malekzadeh. Papagei: Open foun- dation models for optical physiological signals. arXiv preprint arXiv:2410.20544 2024.
Arvind Pillai, Dimitris Spathis, Fahim Kawsar, and Mohammad Malekzadeh. Papagei: Open foun- dation models for optical physiological signals, 2025. URL https://acxivv.org/ee
2050.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234-241. Springer. 2015.
Neal Schmitt and Goran Kuljanin. Measurement invariance: Review of practice and implications. Human resource management review, 18(4):210-222, 2008
Muhammad Shabaan, Kaleem Arshid, Muhammad Yaqub, Feng Jinchao, Sultan Zia, Girid- har Reddy Bojja, Muazzam Iftikhar, Usman Ghani, Loknath Sai Ambati, and Rizwan Munir. Survey: smartphone-bart based assessment of cardiovascular diseases using ecg and ppg analysis. BMC medical informatics and decision making, 20(1):177, 2020.
Mohammad Amin Shabani, Amir H. Abdi, Lili Meng. and Tristan Sylvain. Scaleformer: Iterative multi-scale refining transformers for time series forecasting. In The Eleventh International Con- ference on Learning Representations, 2023. URL. https: ptttp:://openrevieee net/forum
id=sCrnllct
Wei Shuai, Xi-xing Wang. Kui Hong, Qiang Peng, Ju-xiang Li, Ping Li, Jing Chen, Xiao-shu Cheng, and Hai Su. Is 10-second electrocardiogram recording enough for accurately estimating heart rate in atrial fibrillation. International journaa of cardiology, 215:175-1777 2016.
Gerald Simonneau, Nazzareno Galiè, Lewis Rubin, David Langleben, Werner Seeger, Guido Domenighetti, Simon Gibbs, Didier Lebree, Rudolf Speich, Maurice Beghetti, al. Clinical classification of pulmonary hypertension. Journal of the American College of Cardiology, 43 (125):85-S12, 2004.
16

---

Published as a conference paper at ICLR 2026
Gérald Simonneau, Ivan M Robbins, Maurice Beghetti, Richard N Channick, Marion Delcroix, Christopher Denton, C Gregory Elliott, Sean P Gaine, Mark Gladwin, Zhi-Cheng Jing, et al. Updated clinical classification of pulmonary hypertension. Journal of the American college of cardiology, 54(1 Supplement S):S43-554, 2009.
Gerald Simonneau, Michael A Gatzoulis, Ian Adatia, David Celermajer, Chris Denton, Ardeschir Ghofrani, Miguel Angel Gomez Sanchez, R Krishna Kumar, Michael Landzberg. Roberto Machado, et al. Updated clinical classification of pulmonary hypertension. Journal of the Ameri- can College of Cardiology, 62(255):D34-DAA 2013.
Gérald Simonneau, David Montani, David Celermajer, Christopher P Denton, Michael A Gat- zoulis, Michael Krowka, Paul G Williams, and Rogerio Souza. Haemodynamic definitions and updated clinical classification of pulmonary hypertension. European respiratory journal, 53(1), 2019.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps, 2014. URL https://arxiv.
orq/ arq/abs/Du at//abbs/1111 6034
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning. pp. 3319-3328. PMLR, 2017.
Mingxing Tan and Quoe V. Le. Efficienttet: Rethinking model scaling for convolutional neural 1tt.//995..1999/
networks, 2020. URL
Rui Tang, Jaejin An, Brandon K Bellows, Andrew E Moran, and Yiyi Zhang. Trends in hypertension prevalence, awareness, treatment, and control among us young adults, 2003-2023, American 2025.
Journal of Hypertension, pp. hpaf044, 2025.
Benjamin A Teplitzky, Michael McRoberts, and Hamid Ghanbari. Deep learning for comprehensive ecg annotation. Heart rhythm, 17(5):881-88) 2020.
Rahul Thapa, Bryan He, Magnus Ruud Kjaer, Hyatt Moore Iv, Gauri Ganjoo, Emmanuel Mignot, and James Zou. Sleepfm: Multi-modal representation learning for sleep across brain activity, ecg and respiratory signals. In International Conference on Machine Learning, pp. 48039-48037. PMLR, 2024.
Anja Thieme, Aditya Nori, Marzyeh Ghassemi, Rishi Bommasani, Tariq Osman Andersen, and Ewa Luger. Foundation models in healthcare: Opportunities, risks strategies forward. In Extended abstracts of the 2023 CHI conference on human factors in computing systems, pp. 1-4, 2023.
Megha Thukral, Cyrus Tanade, Simon A Lee, Juhyeon Lee, Hao Zhou, Keum San Chun, Migyeong Gwak, Viswam Nathan, Md Mahbubur Rahman, Li Zhu, et Wavelet-driven masked multiscale reconstruction for ppg foundation models. arXiv preprint arXiv:2601.12211 2026.
Akhil Vaid, Joy Jiang, Ashwin Sawant, Stamatios Lerakis, Edgar Argulian, Yuri Ahuja, Joshua Lam- pert, Alexander Charney, Hayit Greenspan, Jagat Narula, al. A foundational vision transformer improves diagnostic performance for electrocardiograms. NPJ Digital Medicine, 6(1):108, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
Haochen Wang. Kaiyou Song, Junsong Fan, Yuxi Wang, Jin Xie, and Zhaoxiang Zhang. Hard patches mining for masked image modeling, 2023. URL httl:tttaa/aaxxv..ooo org. http://awxw....o//////222//
05919
Ke Wang. Jiamu Yang, Ayush Shetty, and Jessilyn Dunn. Dreamt: Dataset for real-time sleep stage cstimation using multisensor wearable technology. PhysioNet https://doi. org/10. 13026/62AN. CB28, 2024.
Peng Wang, Yuanzhouhan Cao, Chunhua Shen, Lingqiao Liu, and Heng Tao Shen. Temporal pyra- mid pooling- baseed convolutional neural network for action recognition. IEEE Transactions on Circuits and Systems for Video Technology, 27(12):2113-2222, 2016.
17

---

Published as a conference paper at ICLR 2026
Lukasz Wesolowski, Bilge Acun, Valentin Andrei, Adnan Aziz, Gisle Dankel, Christopher Gregg, Xiaoqiao Meng. Cyril Meurillon, Denis Sheahan, Lei Tian, et al. Datacenter-scale analysis and optimization of gpu machine learning workloads. IEEE Micro, 41(5):101-112, 2021.
Michael Wornow, Yizhe Xu, Rahul Thapa, Birju Patel, Ethan Steinberg, Scott Fleming, Michael A Pfeffer, Jason Fries, and Nigam Н Shah. The shaky foundations of large language models and foundation models for electronic health records. upj digital medicine. 6(1):135, 2023.
Maxwell Xu, Girish Narayanswany, Kumar Ayush, Dimitris Spathis, Shun Liao, Shyam A Tailor, Ahmed Metwally, A Ali Heydari, Yuwei Zhang, Jake Garrison, et al. Lsm-2: Learning from incomplete wearable sensor data. arXiv preprint arXiv:2506.05321, 2025.
Qian Xu, Wenzhao Xie, Bolin Liao, Chao Hu, Lu Qin, Zhengzijin Yang, Huan Xiong, Yi Lyu, Yue Zhou, and Aijing Luo. Interpretability of clinical decision support systems based arti- ficial intelligence from technological and medical perspective: A systematic review. Journal of healthcare engineering, (023(1):991925) 2023.
Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. Hierarchical attention networks for document classification. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies, pp. 1480-1489, 2016.
Hang Yuan, Shing Chan, Andrew P Creagh, Catherine Tong, Aidan Acquah, David A Clifton, and Aiden Doherty. Self-supervised learning for human activity recognition using 700,000 person- days of wearable data. NPJ digital medicine, 7(1):11, 2024.
Qi Zhang, Yifei Wang, and Yisen Wang. How mask matters: Towards theoretical understandings of masked autoencoders. Advances in Neural Information Processing Systems, 127 127-27139, 2022a.
Renrui Zhang, Ziyu Guo, Peng Gao, Rongyao Fang, Bin Zhao, Dong Wang. Yu Qiao, and Hong- sheng Li. Point-m2ae: multi-scale masked autoencoders for hierarchical point cloud pre-training. Advances in neural information processing systems, 35:27061-27074, 2022b.
Bin Zhou, Rodrigo M Carrillo-Larcoo Goodarz Danaei, Leanne M Riley, Christopher Paciorek, Gretchen Stevens, Edward W Gregg, James Bennett, Bethlehem Solomon, Rosic K Single- ton, et al. Worldwide trends in hypertension prevalence and progress in treatment and control from 1990 2019: pooled analysis of 1201 population-repreeentatvee studies with 104 million participants. The lancet, 398(10304):957-999) 980, 2021.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep fea- tures for discriminative localization, 2015. URL tttt:p::/arxxv.oog//aaa
04150
Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang, Kai Zhang, Cheng Ji, Qiben Yan, Lifang He, et al. A comprehensive survey pretrained foundation models: A history from bert to chatgpt. International Journal of Machine Learning and Cybernetics, pp. 1-65, 2024.
18

---

Published as a conference paper at ICLR 2026
APPENDIX
A AUTHOR CONTRIBUTION
We attribute proper credit the following authors for the development of this project
Table 4: Overview of author contributions.
Author
Concept Experiment Design
Coding Analysis
Writing
Visualization
Project Mgmt.
Discussion
Resources
Simon Loc
Cyras Tanade
Han Zhoe
Juhyeon Lee
Megha Thurkal
Minji Han
Boiying Liu
Keum Clum
Migyeok Gwak
Mchrab Bin Mondaed
Viewam Nathan
Mahbubuur Rahman
Li Zhn
Shumnnyn Desai
B ETHICS CONSIDERATIONS
B. DATA PRIVACY AND CONSENT
Wcarable signals capture sensitive physiological and behavioral information (Erturk ct al. 2025
Our study relies on both publicly available and proprictary (company-ownedt datasets that have been carefully vetted. These datasets include transparent disclosure of data usage, explicit opt-in mechanisms, and the option for participants to withdraw (Perez-Pozuelo et al. 2021). Across the seven datasets used in this study, obtained written consent (via paper digital waivers) that clearly informed participants that their data may be used for commercial research purposes.
B.2 BIAS AND REPRESENTATIVENER
Physiological signals vary across age, gender, ethnicity, health status, and socioeconomic context, yet most existing datasets underrepresent key populations (FitzGerald & Hurst 2017 McCradden et al. 2020 Chen et al. 202I). Such underrepresentation risks embedding biases into foundation models, leading inequitable performance downstream applications. Mitigation requires deliber- ate corpus curation, bias auditing, and systematic evaluation across diverse cohorts. In this study, we sought to mitigate bias by incorporating pre-training corpus drawn from wide range of wearable devices, collected across multiple regions of the world and over many years.
B.3 CLINICAL IMPLICATIONS
Wearable foundation models are not substitutes for medical judgment. Their predictions require regulatory approval and clinical validation before integration into healthcare practice. Without safe- guards, model misinterpretation could lead to misdiagnosis or inappropriate treatment. Development should involve clinical collaborators, real-world evaluations, and explicit positioning of models as decision-support rather than diagnostic systems. In our group, ongoing collaborations aim to eval- uate where our foundation model performs well and how may assist in forming clinical insights. We emphasize that no definitive clinical conclusions should be drawn from this work.
B.4 ENVIRONMENTAL IMPACT
Training generative models entails substantial computational and environmental costs (Ligozat et al., 2022 Bender al. 2021 Bouza et al. 2023). To minimize our footprint, we limited redundant runs, and reused checkpoints to avoid unnecessary GPU usage. All experiments were conducted on data- center GPUs with efficient cooling systems and renewable energy credits reduce carbon intensity. We emphasize that transparent reporting of compute usage and bounding resource allocation are necessary steps toward sustainable machine learning research.
19

---

Published as a conference paper at ICLR 2026
C
REPRODUCIBILITY STATEMENT
Table 5: HiMAE architecture components.
Encoder-Decoder
Layer
Output Shape
EncoderConvBlooc
Input
[B. 1'
DecoderškipBlock
BooodderCon/Bloc11111100
[B. 16, 772]
Layer
BroodderCoomBBloc((16-333
[B. 32, 774]
BicoderCCom/Blocc(22----6
[B. 64, T78]
Convld (A 5, s=2, p=2)
Come Tringpooa Id (2 = 1.522.pz, opel) Csncar skip connection
BatchNorm
BnooderCommBlocc(44411113
[B. 128, 7/16
GELU
Com id 5,5=], p=2)
BnoodeCCom-BBock(188--23
[B. 256, T/32]
BacchNom
Conv -5,, p=2)
DecodesSkipBlocc//55-1112
[B. 128, T/16]
BatchNorm
Cam (i 5,m1l.p2)) pm2)
DecodeSSkipBlinc<(12---1
[B. 64, 778]
BaccNoon
Convld 1=2) BN
DecodeSkkipBliocc(44--33
[B, 32, T741
GELU
DecodeSkkipBlocc(322-11
[B, 16, T/21
Final Decony (16-+1) Tanh
[B, I,
[B. L.
Due to restrictions around data licensing and industry policies, we are unable to release the full source code associated with HiMAE. However, To mitigate this limitation, provide a simplified code base in this
as well as complete details of the model architecture, layer configurations, and hyperparameters in Table 5 This includes all encoder, de- coder, and skip connection blocks, along with kernel sizes, strides, padding, activation functions, and normalization layers. Together, these descriptions and codebases
sufficient to re-implement
the model faithfully in any modern deep learning framework (Paszke et al. 2019 Abadi et al.) 2016 Bradbury et al. 2018 Hannun al. 2023 In addition, we report all training settings (e.g., optimizer, learning rate schedule, and batch size) in the Appendix Section EE further support repro- ducibility. Our goal
ensure that, while the exact implementation cannot be shared, independent researchers can replicate the methodology and validate the findings presented in this work.
C.1 TEMPORAL RESOLUTION AS AN EXPLICIT SCALE AXIS
HiMAE's encoder implements a structured mapping from depth to temporal scale. Each encoder block halves the temporal resolution while increasing the span of input samples contributing each feature, yielding a hierarchy representations indexed by effective temporal support. This makes temporal resolution an explicit axis of representation, rather than an emergent byproduct of depth.
Concretely, the encoder is composed of convolutional blocks, each reducing the sequence length by factor of two. As result, the representation depth b operates At the same time, each block aggregates information over an increasingly large window of the input signal. Because kernel size fixed across layers, the cumulative temporal support of encoder features grows exponentially with depth, scaling as
grid of resolution T/2b.
Rs O(2").
up to rehiitecture-dependee constants. Thus, encoder depth simultancously controls both the gran- ularity at which the signal represented and the temporal extent over which features are computed.
Table instantiates this scale hierarchy for the HiMAE encoder. Shallow layers operate at high temporal resolution with receptive fields spanning only few of samples, capturing fine-scale waveform morphology. Intermediate layers aggregate information over 10'-10² samples, corre- sponding to sub- sub-second temporal structure such as beat-to- variability. The deepest layers inte- grate over several hundred samples, encoding longer-range physiological dynamics across multiple cardiac cycles.
This explicit scale stratification is central to masked autoencoding physiological signals. Because masking removes contiguous temporal regions, successful reconstruction requires contextual infor- mation scale comparable to the masked interval. Features whose receptive fields too small lack sufficient context, while features whose receptive fields are too large oversmooth across distinct physiological events. HiMAE's exponential scale ladder ensures that intermediate encoder depths naturally align with the characteristic temporal extent of masked regions, concentrating learning signal at those resolutions.
20

---

Published as a conference paper at ICLR 2026
Table 6: Temporal resolution and cumulative receptive field through the encoder. T denotes the input length in samples. Re is the receptive field after layer &
Layer
Kernel k
Stride
Output length
R
Encl-conv
5
2
T/2
5
Encl-conv2
T/2
Enc2-conv
5
2
T/4
21
Enc2-conv2
5
T/4
37
Enc3-convl
5
2
T/8
53
Enc3-conv2
5
T/8
85
Enc4-convl
5
2
T/16
117
Enc4-conv2
5
T/16
181
Enc5-convl
5
T/32
245
Enc5-conv2
5
T/32
373
Viewed this way, encoder depth in HiMAE should not be interpreted as a measure of abstraction alone, but an index over temporal scales. This perspective explains why linear probes trained on intermediate layers often outperform both shallower and deeper representations: they correspond to resolutions which physiological structure most predictive.
D DATASETS
D.1 AQUISTION AND APPROVAL
All data analyzed in this study were collected under informed consent, with participants explic- itly agreeing for their wearable-derived signals to be used in health-related research. The consent language stated that data could be used for developing new health features and algorithms and for inclusion in scientific publications. In particular, participants were informed that health and wellness data such as steps, heart rate, sleep, and photoplethyssooraphh (PPG) signals could contribute to findings aimed at advancing general knowledge of health and science. No data used in this study included personally identifying information such as names or email addresses. attach portion of the protocols defined in user data agreements below:
The use of these de-ide de-identified data for data usage reviewed and classified exempt. In addition, because the supporting records constitute case histories and document exposure to devices, we com- plied with the recordkeeping requirements in 21 CFR (3))))0())33),, including obtaining written digital consent and dated information. Participants could withdraw at any time; such withdrawals documented in the case history, and data collected up to the point of withdrawal were retained and used for the investigation in accordance with the consent and applicable regulations.
For downstream evaluations, we relied on a combination of institutional review board (IRB). approved datasets and publicly available resources. For instance, the PVC detection task used paired PPG and ECG recordings to derive annotations of premature ventricular contractions, with ECG- based labels verified both algorithmically and manually. The hypertension classification tasks were drawn from the My Heart Lab Study collected in lab Setting (ID NCT043 14947) and My BP Lab (Clinical Trials ID 19-27169) studies collected in free-world settting. both of which collected wrist-based PPG alongside reference blood pressure measurements under IRB-approved protocols. Sleep staging was evaluated using the DREAMT dataset, which combines PPG with gold-standard polysomnggraphy annotations in individuals with and without diagnosed sleep disorders. Finally, a range abnormal lab test prediction tasks were derived from the Tulane University dataset (ID 20242033), linking PPG from Samsung devices with clinical laboratory values for biomarkers (More details in Appendix Section D).
Across all studies, participants consented to data collection through mobile platforms that supported eligibility screening and enrollment, provided full informed consent, and enabled seamless integra- tion of Samsung devices for continuous signal acquisition. Where appropriate, participants also reported medical histories completed questionnaires through these platforms. All data were de- identified and stored accordance with the approved study protocols, ensuring compliance with ethical and regulatory standards.
21

---

Published as a conference paper at ICLR 2026
This layered consent and governance framework ensures that the data underpinning pretraining and evaluation tasks are both ethically sourced and scientifically robust, supporting the broader goal of advancing health monitoring through consumer wearables.
D.2 PRE-TRAINING DATASETS
Table 7: Demographic Characteristics of the Study Population. Distributions are shown by biological sex, age group, racial identity, and BMI category (N 644).
Category
Subgroup
N
% of total
Sex
Male
36,990
77.6
Female Another gender identity
10,532
22.1
122
0.3
Age
18-29
12,019
25.2
30-49
27.207
57.2
50-64
7,067
14.8
65+
1,351
2.8
Race
White
31,029
65.2
Asian or Pacific Islander
7,630
16.0
Black or African American
3,414
7.2
American Indian Alaskan Native
592
1.2
Another race
4,979
10.4
BMI
Underweight (< 18.5)
823
Normal weight (18.5-24.9)
13,626
28.5
Overweight (25-29.9)
16,634
34.9
Obese (30-34.9)
8,745
18.5
Obese II and IIII (2 35)
7,816
16.4
D.2.1 DEVICE DISTRIBUTION
The distribution of participants and data availability highlights both the diversity of collection de- vices and the heterogeneity of study contributions (Figure 7 At the device level, participation is primarily sourced from Watch Active 2, Watch 3, Watch Active each contributing a lot of par- ticipants, while older models such the Galaxy Gear S3 are represented by fewer users. This heterogeneity in devices provide us with realistic and diverse set of raw wearable signals that can help build generalizable foundation models. The presence of entries labeled "NA" further reflects the mixture of collection devices and the occasional incompleteness of metadata. We note that the devices used our study are provided by two distributors limiting its generalizability and causing potential biases due to not having access to other consumer wearable devices.
D.2.2 PARTICIPANT COUNTS
In terms of study based segmentation, the dataset contains handful of large-scale cohort studies, leading to diverse representation (Figure Efforts were made to ensure representation across stud- ies of varying sizes. This underscores the necessity of leveraging the vast scale of high-volume cohorts while simultaneously preserving the heterogeneity introduced by smaller studies, since both dimensions are essential for building foundation models that truly capture the variability and com- plexity one-dimensional PPG signal modeling. Our data was collected across countries (USA, South Korea, Brazil, Bangladesh) and the demographics are highlighted in Table 7 Note that miss- ing demographics were imputed via KNN based on average PPG segments which have shown to recover these people specific attributes (Miller et al.) 2025 Maclsaac et al. 2025 Ferdinando al. 2019).
D.2.3 PRE-PROCESSING PIPELINE
We segment raw PPG signals into fixed-length 10 s windows and apply lightweight quality-control pipeline to remove motion artifacts and non-physiologic segments. Each window is first standard- ized to remove scale and offset differences across devices and recording conditions. Windows with extreme amplitude fluctuations, indicative of motion bursts sensor saturation, identified using a simple distributional check and either trimmed to remove outliers or discarded the signal remains unstable. This step prioritizes precision over recall ensure high-quality pretraining data.
22

---

Published as a conference paper at ICLR 2026
Figure 7: Segment Count by Study. This bar chart shows the number of data segments collected for each study, with the y-axis on a logarithmic scale to account for the large differences segment counts.
For windows that pass amplitude screening. autocorrelation. Physiologically plausible PPG signals exhibit quasi-perion structure; windows with highly irregular unstable periodicity are rejected, as these patterns typically arise from tion or sensor decoupling. We additionally enforce a minimum number of cycles to eliminate de- generate truncated traces.
temporal regularity by measuring short-range
Surviving windows are band-pass filtered to the cardiac frequency range to remove baseline drift and high-frequency noise while preserving pulse morphology. Signal quality is then evaluated via template matching against canonical PPG waveform. We compute per-window quality score that jointly reflects the fraction the signal that matches the template and the strength of that match, penalizing cases where apparent agreement is driven by only small portion of the window. Windows that fail this final morphology check are excluded.
This filtering is applied at scale across the corpus, retaining only windows that clean, periodic, and morphologically consistent. The resulting pretraining set emphasizes physiologically meaning- ful PPG signals across devices and sampling rates, substantially reducing motion artifacts without relying on labels, heuristics tied to specific hardware, or subject-level metadata.
D.3 DOWNSTREAM EVALUATION DATA
We evaluate HiMAE across diverse downstream tasks to assess the generality of wearable PPG representations. Rather than assuming fixed mapping between PPG and outcomes, we exploit HiMAE's ability learn hierarchical temporal features and adaptively resolve signal segments scales most informative for prediction. This design allows us to probe the representational value of optical physiological signals across clinically and behaviorally relevant applications.
D.3.1
PVC DETECTION
Table Stratified 80/20 Train/Test splits for PVC tasks (with per-task totals).
Task
Split
Negative
Positive
Total
train
369987 (91.8%)
32832 (8.2%)
402819
PVC Detection
test
69880 (89.7%)
8019 (10.3%
77899
totals
439767 (91.4%)
40950 (8.6%)
480717
Premature Ventricular Contractions (PVCs) (Number Breakdowns in Table
abnormal beats arising in the ventricles (Cha et al. 2012 Kaya & Pehlivan (013). We use paired PPG-ECG data, with ECG annotations generated using Beatloggie (Teplitzky et al. 2020) and manually verified.
23

---

Published as a conference paper at ICLR 2026
PPG imputs 10s non-overlapping wrist segments, pre-processed with Savitzky- Golay filter (Luo et al. 2005), a 0.5-4.0 Hz bandpass, normalization to |-1, 1], and exclusion of segments with motion artifacts disruptions s. This task evaluates whether ubiquitous PPG can approximate arrhythmia detection typically restricted to ECG.
D.3.2 HYPERTENSION CLASSIFICATION
Distribution of Blood Pressure (lab) PPG Segments across 135 Subjects
PPG Segments
Systolic Blood Pressure
300
Diastolic Blood Pressure
200
100
25
50
75
100
125
150
175
200
mmHg
Distribution of Blood Pressure (free-living) PPG Segments across 9427 Subjects
PPG Segments
2000
Systolic Blood Pressure
Diastolic Blood
1000
125
150
175
200
mmHg
Figure 8: Blood Pressure Distribution: The distribution of Blood Pressure Values (mmHG) across the lab and free-living studies. We define hypertension as systolic over 130 and diastolic over 80 to generate binary outcomes.
Hypertension classification (Number Breakdowns in Figure g relies on cuff-based references (ii- monneau et al. 2004 Giles al. 2005 2009 [Simonneau al. 2009 2013 2019). Subjects within 18 mmHg of the diagnostic cutoff are excluded to reduce label noise, with remaining individuals la- beled hypertensive normotensive. Each 10s PPG segment undergoes Savitzky-Gole smoothing, 0.5-4.0 Hz bandpass filtering, normalization to [-1, 1]. and artifact removal. Unlike PVC detection, which event-based, this task leverages PPG morphology and temporal dynamics reflect vas- cular state. These evaluations contain both hypertension data collected in a naturalistic free world environment and within a controlled lab environment for both the hypertensive and blood pressure regression tasks.
D.3.3 SLEEP STAGING
Table 9: Stratified 80/20 Train/Test splits for Sleep Staging,
Task
Split
Wake
Light
Deep
RFM
Total
train
44829 (23.9%)
115932 (61.8%)
6696 (3.6%)
20214 (10.8%)
187671
Sleep Staging (4-class)
test
11298 (23.6%)
30153 (63.1% (%)
1416 (3.0%)
4881 (10.2%)
47748
totals
56127 (23.8%)
146085 (61.9%)
8112 (3.4%)
25095 (10.6%)
235419
Sleep staging (Number Breakdowns in Tables 9) is evaluated on the DREAMT dataset (Wang al.) 2024) hosted on PhysioNet (Goldberger et al. 2000), which includes overnight wristband data with simultaneous PSG. Annotations follow AASM standards into wake, REM, NREMI, NREM2, and NREM3, excluding missing and preparation segments. PPG bandpass filtered (0.5-12 12 Hz) (But- terworth et al. 1930), segmented into 10s windows, and normalized Performance is measured with five-fold subject-independer cross-vart validation. This task examines
mean and unit variance.
whether PPG encodes temporal patterns sufficient for sleep stage classification. We note that sleep
24

---

Published as a conference paper at ICLR 2026
Number of Abnormal ATC PPG Segments across 19 Subjects (52.79% Positive Rate)
HEMOGLOBIN Segments Subjects
PPG Seyments
1000
Normal Positive
Segments
1500
Normal
Positive
500
1000
500
2
6
10
12
0.0
17.5
Lab Value
Value
Number
Segments across Subjects
Number Abnormal PLATELETS PPG Segments across Subjects
(50.74%
Rate)
(50.92% Positive Rate)
Segments
1000
Normal
2000
Postive
Posttive
500
PPS
1000
0
100
200
300
Lab Value
Lab Value
Number of Abnormal SODIUM PPG Segments across 36 Subjects (67.59% Positive Rato)
PPG Sagmants
Normal
Positive
2000
1000
Lab Value
Figure 9: Abnormal Labs Distribution: The number of PPG segments for abnormal labs seperated based on lab-specific cutoffs. We define an abnormal lab as falling above the 75th percentile of values and normal lab as falling within the 25th percentile.
staging has canonically been designed by leveraging the whole sleep cycle but we are assessing the ability to monitor real time sleep staging from much shorter PPG segments.
D.3.4 ABNORMAL LAB TESTS
For abnormal lab test prediction, paired with clinical laboratory results. Each test is framed as binary classification task: outcomes labeled negative if within the 25th percentile of lab values and the positive labels are anything above the 75th percentile (Figure D.. All other labels are excluded. Preprocessing matches other tasks. Targets include AIC, hemoglobin, platelets, potassium, and sodium, each selected for estab- lished clinical relevance. This task extends evaluation beyond cardiovascular and behavioral end- points systemic markers of metabolic, and hematologic health. We note that it unclear whether PPG can predict abnormal from healthy lab values bused the PPG alone. Despite this, Tulane univeristy presents us with an opportunity to discover if PPG signal can provide digital signatures making this exploratory task in benchmark.
Samsung Galaxy Watch PPG collected Tulane University
25

---

Published as a conference paper at ICLR 2026
E BASELINES AND MODEL CONFIGURATION
Self Supervised Pre-trained methods have become dominanat paradigm for health and wellness to study a variety of applications (Wornow et al. 2023 Thieme et al. 2023] He et al.] 2024] [m al. 2025 Lin et al., 2025). Foundation models for mieddimesioonal signals predominantly
purposed from architectures designed for vision, with adaptations that reinterpret temporal structure as flattened analogue of spatial correlation. In this section highlight our baseline models and model configurations
E. BASELINES
MAE-ID We introduce a Masked Auto ecnoder that mimics the protocol of LSM (Narayanswamy et al. 2024). This model introduces large-scale foundation model trained multimodal wearable sensor data but we adapt it to ID PPG Signal. Specifically, it adopts a vision transformer archi- tecture trained via masked autoencoding with random masking. In work, we do replicate the full multimodal design; instead, we adapt and constrain the model to unimodal setting for fair comparison and due lack of open source code.
Tein-Trannformer (Liu et al. 2021a) is a hierarchical Transformer that forms multi-scale repre- sentations by restricting self-attention to non-overlapping windows and alternating partitions with a shifted-window scheme, which enables cross-window communication while keeping computation near-linear sequence length. We use this baseline this direct comparison and counterpart to our proposed hierarchical HiMAE model. For wearable sensing, we adopt ID adaptation that tokenizes temporal patches and applies windowed attention along time, capturing both fine-graing iineigrieedd waveform morphology and longer-range dependencies.
Masked Siamese Networks (MSN) (Assran et al. 2022 learn label-efficient representations by combining masked signal modeling with Siamese-style contrastive objectives. Instead of relying on class labels, MSN masks portions of the input and enforces consistency between augmented views. Architecturally, it employs Vision Transformer encoder shared across views, while leveraging a predictor network to stabilize training. The key idea to couple self-distillation with masked reconstruction to reduce sample complexity.
DINO (Caron al. 2021) is a sulf-supervised framework that leverages knowledge distillation without labels. Using a teacher-studen setup, the student network is trained to match the output distribution of the teacher under different data augmentations. Both networks are ID-ViTs, and the method induces cluster-like emergent properties in the learned embedding space, enabling strong transfer performance without explicit contrastive pairs or handcrafted pretext tasks.
SimCLR (Chen et al., 2020b) establishes contrastive learning as competitive self-supervised paradigm. The core idea to maximize agreement between augmented views of the same sig- nal in latent space while pushing apart representations of different images. This is implemented using a ResNET encoder (He et al. 2015). projection head, and a contrastive loss (NT-Xent (Chen al. 2020a)).
PaPaGei (Pillai et al. 2024) is a domain-specific foundation model designed for optical physiolog- ical sensing, particularly photoplethysmographh (PPG). It adapts ResNET-style CNN architectures to learn robust, generalizable representations from large-scale optical physiological datasets. Pa- PaGei releases both model weights and datasets to support reproducibility and broader adoption in physiological signal analysis. In our work, we used their source code to benchmark their method by pre-training volume of data to ensure fair comparison.
E.2 HYPERPARAMETERS FOR HIMAE AND BASELINES
To ensure fair comparison across models, we aligned the training setup closely as possible to the original implementations while maintaining consistency in optimizer choice and scheduling. All the methods trained from scratch (HiMAE, MAE-ID, were trained under identical optimization regimes, while PaPaGei follows its released open source training protocol. Table 10 summarizes the key hyperparameters for all models.
MSN. DINO, SimCLR)
26

---

Published as a conference paper at ICLR 2026
Table 10: Hyperparameter Configurations for Different Models
Contiguration
HIMAE
MAE-ID
Swin- Trunsformser
MSN
DINO
SimCLR
PaPuGei
Training Steps
50000
15000
Warmup Steps
2500
Optimizer
AdunW
Lodichioo & Hamer
[017]
Opt. momentum (5.
[09, 0.95]
[0.9, 0,95
[0.9. 0.95
1000 1999]
[0590099))
[09, 0.99]
Buse learning rute
0,001
0.005
0.000
0.001
0,004
0.001
0.001
Weight
Grudicat clipping
Dropout
Denry
Function
Mian Squaaal Frace
Carec Estrap
Contrasive
resolation
Sampling
Augincatation
Time- Tens-Wapiag,
E.3 LAYER WISE ANALYSIS
Layer Wise Analysis
def layorwise_prols (model, dataloader, labels, device):
For all architectures we use the full sequence embedding across the temporal dimension, without collapsing single summary token, to ensure that downstream probes have access to resolution-specff. information.
model. to (device) eval () hocks, acts [1, (}
9
10
# capture encoder layer outputs
for i, layer in enumerate (model. layers)) nooks.appents (layer (wwycr..rogistrrree....ooo.. rog.ttrr___o_oooar_..ooo.. (lambda m, x, Y, 1=1: acts .setdefault (1, y yedetach c).u))))))
Xa (i: [] for in range (len (model encoder (aaeordderraaer))) layers))
ys []
for xb in dataloader:
18
xb to (device)
with torch grad t)
model (xb. tww.ssoraesseeeeee (1, 2))
for i, a in acts.itens..
22
xs Xs [i]..ap (a.).aap preserve time
(a.
(1) numpy ()) flatten embedding to
ys. (1abells
(0))))
labels labels (ib.
26
results = +)
for 1, feats in Xs. ().items():
np. copcateenntee (feats)
y np. concateenaee (ys)
30
StaandaddSaaees () firttsateer....ttttaaaffoommmm
clf iogistickegresss (
cross_val_scoree roc auc") m) results [ laaeer auc
X, y, co=$trattifiedkro1d(5)) scoring-"
34
for h in hooks: h. remeve) return results
The layer-wise analysis examines how temporal resolution in the learned representations aligns with performance on downstream tasks. For each encoder block, we extract and flatten the full sequence embedding to preserve temporal detail, allowing probes access features different levels of abstraction. By training logistic regression classifiers on these embeddings, we can assess which layers best capture task-relevant temporal patterns.
27

---

Published as a conference paper at ICLR 2026
F ADDITIONAL RESULTS
F.1 MODEL CONFIGURATIONS ABLATIONS
We conducted a comprehensive ablation study of HiMAE on 100 dataset comprising ten million segments (roughly 30k hours). The experiments systematically varied architecture and hyperparam- eters to understand their effect on reconstruction quality (Extrapolation task from our generative benchmark in tables where it is not explicitly stated as previously done in @Narayanswamy et al, 2024)), with multiple independent training runs averaged to reduce variance from stochastic initial- ization and data sampling. Unless otherwise noted, all training employed Adam with learning rate of 3 x 10-4, cosine decay scheduling, and batch size of 512.
Architecture. We evaluated HiMAE alongside CNN baselines across increasing network depths, defined by the sequence of hidden channel dimensions [16, 32, 64), [16, 32, 64, 128), and [16, 32, 64, 128, 256]. Table [1 lists the parameter counts, showing a modest growth for HiMAE compared to CNN baselines, with the skip-connected HiMAE exhibiting slightly higher capacity than its no-skip variant.
Table 11: Model Parameters (in K M)
Model Depth
3-layer
4-layer [16,32,64,12 128]
5-layer
[16,32,64]
[16,32,64,128,550
CNN
26.2 K
108 K
437 K
HiMAE-no skip
66.
271
1.10
HiMAE
75.3 K
309 K
1.25 M
The impact of network depth on mean absolute error (MAE) and mean squared error (MSE) is sum- Increasing depth consistently reduced both MAE and MSE for HiMAE, with
marized in Table T2 the deepest configuration yielding the lowest reconstruction error. Skip connections were critical, as HiMAE consistently outperformed its no-skip variant across all depths.
Table 12: MAE and MSE for Different Network Depths
Model
3-layer
4-layer
5-layer
[16,32,64]
(66,2,64,1288
[16,32,64,128.256]
Depth
MAE 4
MSE +
MAE 4
MSE
MAE
MSE
CNN
0.405
0.234
0.417
0.249
0.400
0.231
HiMAE-noskip
0.403
0.236
0.400
0.246
0.397
0.233
HiMAE 0.400
0.230
0.389
0.223
0.382
0.221
Patch Size. We varied the spatial-temporal patch sizes over 1, 5. 10, and 20. The results in Table 14 indicate that 5 provided the best trade-off between local resolution and generative performance. Smaller patches increased flexibility but slightly degraded performance due to reduced receptive field per token, while overly large patches caused loss of fine-grained structure.
Table 14: Model Performance for Different Patch Sizes
1
5
10
20
Model
MAE
MSE 4
MAE L
MSE +
MAE 4
MSE
MAE
MSE
CNN
0.414
0.239
0.400
0.231
0.412
0.244
0.427
0.261
HiM no-okkip
0.406
0.239
0.397
0.233
0.403
0.246
0.419
0.262
HiMAE
0.389
0.226
0.382
0.221
0.386
0.231
0.403
0.247
Convolution Kernel Size. Kernel size was varied over (1,5, 10, 20). Table 15 shows that yielded the lowest errors across all models, suggesting moderate receptive fields match the temporal and
28

---

Published as a conference paper at ICLR 2026
spatial scales of our data. Very small kernels restricted context aggregation, while very large kernels oversmoothed latent features.
Table 15: Model Performance Across Convolution Kernel Sizes
1
5
10
20
Model
MAE
MSE
MAE
MSE |
MAE 4
MSE
MAE
MSE
CNN
0.416
0.241
0.401
0.230
0,410
0.241
0.424
0.257
nissE-noskip
0.409
0.242
0.395
0.233
0.403
0.244
0.420
0.259
HiMAE
0.392
0.228
0.382
0.220
0.388
0.231
0.404
0.248
Smaller strides yielded the best performance, particularly for HiMAE, by preserving high temporal resolution in early feature maps. Performance degraded monotonically with stride increases.
Stride. We evaluated stride values of 2, 4, and 8 (Table 16..
Table 16: Model Performance Across Stride Values
2
Model
MAE
MSE
MAE
MSE 1
MAE 1
MSE
CNN 0.401
0.231
0.413
0.244
0.431
0.267
HiMAE-noskip
0.397
0.233
0.409
0.247
0.427
0.270
HiMAE 0.382
0.220
0.392
0.232
0.410
0.250
Masking Ratio. Finally, we explored the effect varying the latent masking ratio in the masked autoencoding objective for generative tasks, with ratios from 0.5 to 0.9. As shown in Table [7 interpolation and extrapolation both improved when increasing the ratio up to 0.8, after which per- formance degraded for interpolation and collapsed for extrapolation.
Table 17: MAE and MSE for HiMAE Across Different Masking Ratios Evaluated on Generative Tasks
Temporal Interpolation
Temporal Extrapolation
HiMAE Masking Ratio
MAE |
MSE
MAE L
MSE
0.5
0.397
0.229
0.407
0.251
0.6
0.388
0.222
0.397
0.229
0.7
0.384
0.220
0.396
0.227
0.8
0.379
0.218
0.387
0.221
0.9
0.381
0.221
0.288
0.221
Final Selection. These controlled experiments informed the final HiMAE configuration: the deepest: architecture [16, 32, 64, 128, 256] with skip connections, patch size 5, kernel size 5, stride 2, and a masking ratio of 0.8, which jointly achieved the best tradc-off between reconstruction fidelity and parameter efficiency.
F.2 ECG
attains the lowest masked- reconstruction error on ECG (Table [8]. indicating that its hierarchical masking and reconstruction inductive biases capture re- construction capacity beyond PPG. MAE-ID (ViT) is close second, while the ablated HiMAE and CNN trail, reinforcing that the full HiMAE design transfers effectively to the ECG domain.
HiMAE
Table 18:
Masked-reconstruucie reconstruction loss on ECG masked auto encoding task.
Model
MSE (L)
HÎMAE
0.148
MAE-ID (ViT)
0.162
HÎMAE (no skip)
0.184
CNN
0.207
29

---

Published as a conference paper at ICLR 2026
Scaling Number Hours Generative Tasks
MAE 1D (wi)
CNV
HIMA (ne sep)
HRAL
0.45
Random Imputation
0.45
Temporal Interpolation (Sansor Off)
Temparal Extrapolation (Forecasting)
0.40
0.40
0.40
0.35
Salee Eree
0.30
0.25
0.30
0.30
0.20
0.15
0.25
0.10
0.20
0,20
0.05
0.00
0.15
0.15
10*
10a
104
Hours
Number Hours
Number Hours
Scaling Number of Participants Generative Tasks
MAE 1D (V7)
CMV
HMME (mo wie))
HRAME
0.35
Random Imputation
0.40
Temporal Interpolation (Sensor Off)
0.40
Temporal Extrapolation (Forecasting)
0.30
0.35
Meen Squared Eroo
0.25
0.0
0.30
0.30
0.15
0.25
0.20
0.05
0.00
0.15
0.15
10a
10a
10
10F
102
10a
10*
103
10a
10a
10
10F
Participants
Participants
Participanss
Scaling Model Size on Generative Tasks
MAE-1D (VT))
CNN
(ro sep)
Random Imputation
Temporal Interpolation (Sensor Off)
Temparal Extropolation (Forecasting)
0.30
0.50
0.50
0.25
0.45
Meen Sqaared Enor
020
0.35
0.35
0.30
0.30
0.10
0.25
0.05
0.00
0.15
0.15
10+
10"
10
10*
10
10*
10
10
107
10
104
10*
10"
10'
109
Model Parameters
Model Parameters
Model Parameters
Sceling Batch Size on Generative Tasks
MAE (DEDD (i])
CNN
HMME (se seg)
HRM
0.35
Random Imputation
0.40
Temporal Interpolation (Sensor Off)
0.40
Temporal Extrapolation (Forecasting)
0.30
0.35
0.35
Meen Sqowed Eret
0.25
0.20
0.15
0.25
0.25
0.10
0.20
0.00
0.06
0.00
1024
2043
256
1024
2046
256
1024
2048
Batch Size
Batch Size
Batch
Figure 10: Scaling Experiments on Generative Tasks: Evaluation on the three generative tasks. HiMAE consistenly outperforms all model at our scale of data
30

---

Published as a conference paper at ICLR 2026
F.3 SCALING RESULTS FOR GENERATIVE TASKS
Scaling analysis. We evaluate HiMAE's reconstruction error under participant, recording hour, batch size, and model size scaling, following the regimes of Narayanswamy et al. (2024); Xu al. (2025): random imputation, temporal interpolation, and temporal extrapolation. Across all settings HiMAE follows clean scaling law trends (Kaplan et 2020) and maintains margin over MAE-ID (ViT) and CNN baselines.
The most pronounced effect model size. small capacities HiMAE achieves lower error than much larger transformer baselines, highlighting the advantage of hierarchical inductive bias over sheer parameter count. MAE- 1D only begins close the gap at orders of magnitude more parame- ters. The transformer could surpass HiMAE model when given larger capacity but this again highlights the effectiivnness of the inductive bias that we are conveying. Participant, hour, and batch size scaling follow canonical patterns. More participants and longer recordings steadily reduce ror, with HiMAE continuing to improve where baselines saturate, especially on interpolation and extrapolation.
F.4 HIERARCHAL CONCORDANCE
HiMAE layers vs. tasks
Cardorass
Boodmobird D25
Dectro
HiMAE-L1
HIMAE Layers
HiMAE-L2
HIMAE-L3
HiMAE-L4
Downstream Tasks
HiMAE layers vS. tasks
Blochroleted tlectrr
HIMAE-L1
HIMAE Model & Layer
HiMAE-L2
HIMAE-L3
HiMAE-L4
HiMAE-L5
Downstream Tasks
Figure 11: HiMAE layer concordance across encoder depths. Heatmaps compare downstream AUROC when probing HiMAE at layers (top) versus layers (bottom). Despite removal of an encoder-decodes stage, the resolution-task alignment remains highly concordant: tasks such as PVC detection and hypertension consistently peak at similar layers, while sleep staging benefits from coarser representations, Minor deviations appear in intermediate layers, but the overall hierarchy of predictive resolutions is preserved, indicating robustness the resolution hypothesis to architectural depth.
31

---

Published as a conference paper at ICLR 2026
Layer concordance across depths. We further assess the stability of the resolution hypothesis by comparing HiMAE trained with four versus five mncoder-decodee stages (Figure [T]. The resulting heatmaps reveal that the alignment between downstream tasks and temporal resolutions is largely preserved across depths. Cardiovascular endpoints such as PVC detection and hypertension consis- tently achieve their best performance finer layers, while blood related labs benefits from coarser layers. Although minor fluctuations appear in intermediate levels, the overall hierarchy of predictive resolutions concordant. This suggests that the resolution-task mapping uncovered by HiMAE not an artifact of architectural depth, but robust property of the representations themselves.
F.5 TRANSFORMER LAYER INTERPRETABILITY
Transformer Layers Discover Resolution Specific Str acture Across Downstream Tasks "Blitad
Electolytte
Slepp stag
0.74
0.56
0.48
0.62
0.54
0,54
MAE--D-L2
0.42
0.62
0.71
0.72
0.58
0.58
0.48
0.85
0.62
0.54
0.66
0.56
MAE-1D-LA
0.43
0.63
0.72
0.75
3.56
0.50
0:48
0.59
0.62
0.55
0666
0.56
wwW. 100
PIG Deneenttioo
A0C
Manogooo
Sodum
Light
200
Figure 12: Transformer based Layer Wise Analysis: Unlike HiMAE, MAE-ID exhibits non- monotonic trends and lacks concordance with HiMAE's internal representation hierarchy. We ob- serve that performance typically peaks in intermediate and task-specific later layers, though this pattern carries important nuance.
outline deep conceptual contrast between HiMAE's layer wise interpretability from Figure 12
Figure and Appendix Section F.4 and Transformer based encoders like MAE-ID (ViT) how their internal representations evolve and why their layerwise "probing" behaviors differ.
In a U-Net with convolution operators, each layer has a localized receptive field that gradually ex- pands with depth. Convolution is local operator, so early layers capture fine-grained spatial details (edges, textures), mid layers combine local motifs into parts, and deeper layers encode semantic abstractions or whole objects. Skip connections reintroduce lost resolution but don't globalize in- formation. This creates hierarchical representation: the notion of "scale" is physically encoded in the architecture, with clear separations between low-level and high-level representations. When you probe features layer by layer, you observe clean transitions.
Transformers, on the other hand, start with global receptive fields from the very first layer because self-attention mixes information across positions in one step. Every token can, in principle, in- teract with every other token regardless of spatial proximity. Depth therefore does not correspond to expanding spatial scale but instead refines representations through repeated global mixing and spe- cialization. Each layer develops different attention patterns and specialized circuits (like induction heads or copy-suppression heads), rather than encoding progressively larger spatial features. Depth adds precision and compression, not hierarchical abstraction.
This makes transformer probing results very different. Since there is no strict bottom-to- to- feature pyramid, probes do not show a monotonic increase in semantic abstraction. Instead, they show nonmonotonic behavior: mid layers often peak in semantic information, and late layers compress
32

---

Published as a conference paper at ICLR 2026
features toward task-specific prediction spaces. Representations distributed, overlapping, and non-hierarchical each layer contributes globally but in subtly different ways.
Mathematically, this difference arises because convolution enforces spatial locality and translation cquivariance via weight sharing and limited receptive fields, while self-attention defines global kernel = softmax (KT /Vd) that mixes all spatial positions. Hence, the "hierarchy" in CNNs is emergent geometric property of the convolution operator, whereas in transformers the represen- tational geometry is depth-wise iterative refinement within a globally connected graph.
F.6 T-SNE VISUALIZATION AND REPRESENTATION INTERPRETABILITY
Layer HIMAE (PVC)
Layer 2- HiMME (PVC)
Layer (PVC)
50
PVC
50
dimentennn
PVC
Class
Cleesr
Cleoss
-50
-50
-50
0
50
-50
50
-50
50
1-SNE dimension1
SNE dimension
I-NNE dimension 1
Layer 4 HIMAE (PVC)
Layer 5 HIMAE (PVC)
1-SHE dimension 2
E:SNE dimesronn
PNC
Class
-25
Normal
Noraal
dimeniion1
dimersii
Figure 13: t-SNE Visualization and Representation Interpretabillit. We explore how the layer wise TSNE evolve on the PVC Classification task to help us understand how the representations are organized as they traverse the multiple encoders.
To characterize how HiMAE's hierarchical representations evolve across depth, we visualize t-SNE projections of encoder embeddings from each layer the PVC detection task in Figure 13 The corresponding AUROC scores (L1: 0.73, L2: 0.72, L3: 0.77, L4: 0.80, L5: 0.80; Figure 4 closely mirror the progressive emergence of class separability observed across layers.
Representations from early layers (L1-L2) form diffuse clusters with limited separation between normal and PVC segments, indicating that these layers predominantly encode local waveform mor- phology. At intermediate depth (L3), distinct PVC clusters begin to emerge, coinciding with the first substantial increase in AUROC and marking transition toward rhythm-level abstraction. Deeper layers (L4-L5) exhibit compact, well-separated clusters, consistent with representations that inte- grate longer temporal context and capture higher-level cardiac dynamics relevant for arrhythmia discrimination.
Together, these visualizations provide an interpretable view of HiMAE's hierarchical inductive bias: representations progressively abstract temporal information from fine-grained morphology to broader physiological context. The laser-wise evolution of t-SNE structure offers empirical sup- port for the resolution hypothesis, suggesting that higher layers encode slower, more discriminative temporal processes that ultimately drive clinical performance.
33

---

Published as a conference paper at ICLR 2026
G ON-DEVICE EXPERIMENTS
G.1 EXPERIMENTAL PROTOCOL
We evaluated the on-device performance HiMAE using a Samsung Galaxy Watch running Wear OS. All experiments were performed natively on-device to capture realistic latency and through- put characteristics under wearable hardware constraints. The model was deployed using PyTorch Mobile with TorchScript conversion to minimize runtime overhead and ensure compatibility with ARM-based computation. The device is powered by the Exynos W W1000 chipset. featuring 5-core CPU (Ix Cortex-A78, 3x Cortex-A55, 1x Cortex-M55) fabricated on 3 nm GAA process, and equipped with GB LPDDR5 memory.
Inference was performed at fixed batch size of corresponding to 10-second physiological signal window sampled at 100 Hz. To ensure measurement stability, we used 20 warm-up runs followed by 100 timed inference passes. Latency was defined as the mean per-sample forward-pass time, with additional reporting of median and 95th percentile values to capture tail latencies. Throughput was defined as the total number samples processed per second over a 10-second rolling interval.
For timing measurements, we used CUDA event synchronization on GPU and Python's high- resolution wall-clock timers on CPU. All computations were executed using float32 precision. The benchmarking routines are provided in the accompanying code listings, which include throughput and latency measurement functions.
As a point of reference, we additionally report datacenter-grade inference metrics obtained using an NVIDIA T4 GPU contextualize the mobile device performance. Although the T4 operates at higher power, modern mobile GPUs (e.g.. Qualcomm Adreno 750) demonstrate comparable infer- ence efficiency per watt (Buber & Banu 2018 Wesolowski al.) (20IT) validating the relevance of on-device inference as proxy for real-world deployment on consumer hardware.
Throughput Code
def measure mhasure_throughput (model, dummy_inpate input, device, sam_seconda-10)
"""Measures inference throughout (samples/sec) with fixed batch size of amn
3
model. eval ()
batch_size ze
dummy_input dunmy_input. (device)
model. (device)
with torch. moaddoddgrd)()) for in range (10)
model (durmy_input)
12
if device. type == 'cuda':
13
torch. cuda www..cu.a...cccm....aamm synchronize
num_inferences start time time. time with torch. no torchh.o_graad)):
17
while time. time. start time num sum_nec model (durmy_input)
device. type 'cuda';
21
torch.cuda, synchronize()
nun_inferences +=
total_time time. start throughout num num_infeeee iun_inferences batch batch_size / total_tinn return ("Throughput. wttpgghptttssaape_ppee__e_e throughput)
26
34

---

Published as a conference paper at ICLR 2026
Latency Code (Batch Size 1)
def measure_int inference_ting satrice___ttt__________ siste_bslll (model, danmy_input, device)
"*Masurres inference latency (mean, median, 95th percentile) with fixed baten size of 1. "** Hommy_inpiln shape10]
raise Valuetrrer (f" Input batch size must be 1 for this function. Got whwpiintttthhppe1111 [)])")
model.evall
dummy_ingg input dunmy_input. (device)
model.to (device)
with torch.no_grad
for in range (varmup_runs) model (dummy_input)
13
if serice.type -- 'cuda':
torch. cuda haw.uddasyncoooiiio..... ze
timings 1|
17
with torch.no ()
18
for in range (num_runs):
if device, type 'cuda':
start_event torch. cuda Event (enablee_iiiing-tsse))
event torch, cuda. Event cenable_timing--eee
22
Seart_evvet..ecorc
= model (dummy_inpur)
End_eveent.reorre
torch, cuda synchronize())
26
elapsed_time_ start_event wwtp:eeett.eepppeettteeeeee (end_event)
27
else:
start_time time.time
model (dummy_input
end time time.time ()
elapsed time ms (end time start_times 1000
timings aaa.ings..ppendeeeeeeeee time ms)
mean_latency np. mean (timings)
35
median_latency = np (ww.iaanttiiinnngegge
36
std_latency sp.std(tim
p95_latency
ieerreenillettimnngee 95)
return {
"Mean_iatency_nt" mean_
40
Leccian_Latency_rat mss: median_ latency, "P95 "P95_Latency__s" 195__attncyy
35

---

Published as a conference paper at ICLR 2026
G.2 INFERENCE EFFICENCY
We benchmarked the inference efficiency of our proposed HiMAE against the transformer baseline (MAE-ID), measuring three aspects: model footprint and computational complexity in terms of parameters, memory, and FLOPs per 10-second input window at 100 Hz (Table [9: latency, defined mean per-sample forward-pass time at batch size I; and throughput, defined the max- imum number of samples processed per second (Table 20)
Model
Params
FLOPs
Memory
HiMAE
1.2M
0.0647 gFLOPS
4.8 MB
Efficient-Ne:
7.8M
0.70 gFLOPS
31.1 MB
Swin- Transformer
110.6M
11.89 gFLOPS
423.8 MB
MAE-ID ID
110.6M
15.94 gFLOPS
441.3 MB
Table 19: HiMAE is lightweight and efficient: Model size and compute cost comparison between HiMAE and MAE- ID. FLOPs measured per forward pass 10s sequence at 100Hz.
Results Despite being more than two orders of magni- tude smaller in parameter count, the HiMAE consistently out- performs the transformer base- line across all efficiency metrics. Between Efficient-Net (Tan & Le, 2020), remains marginally better which is encouraging due to the optimizations designed in this model.
Model
GPU Lat.
GPU Thr.
CPU Lat.
CPU Thr.
HiMAE
0.039 ms
25.8k/s
0.99 ms
1.2k/s
Efficient-Net
0.082 ms
12.2k/s
1.42 ms
0.704k/s
Swin-Transformer
0.704
1.42k/s
2.95
0.456k/s
MAF-ID
0.80 ms
1.24k/s
3.36 ms
0.298k/s
Table 20: Inference Performance: Latency (ms per sample, batch size 2048) and throughput (samples/sec) measured over 10s windows.
Model footprint: HiMAE re-
duces parameters from 110M to 0.31M (~ 355x fewer), FLOPs from 15.94G 0.0647G (~ 246x fewer), and memory from 441.3MB to 3.6MB (~ 123x smaller). These reductions highlight that computational savings scale with the compactness of the model, without loss of representational capacity for the task.
Latency: HiMAE achieves substantially faster per-sample inference. On GPU, latency drops from 0.80ms to 0.039ms (~~ 20x faster), while on CPU it falls from 3.93ms to 0.99ms (~ 4x faster). The reduction in latency follows directly from the smaller computational footprint, reflecting a consistent efficiency advantage.
These improvements translate into higher throughput across hardware. On GPU, throughput increases from 1.24k to 25.8k samples/s (~ 21x higher), while CPU throughput rises from 0.255k to 1. 1.2k samples/s (~~ 5x higher). These results confirm that computational gains extend beyond memory and FLOPs, yielding end-to-end speedups at inference time.
Throughput:
In summary, HiMAE achieves favorable tradeoff between compactness and efficiency, providing lower FLOPs, smaller memory footprint, and faster inference despite its reduced model size. It also outperforms Efficient-Net B1 which was specially designed and optimized for performance and compactness giving comparison and context to our models performance.
36

---

Published as a conference paper at ICLR 2026
H FREQUENTLY ASKED QUESTIONS
What are the main conclusions from this work? We demonstrate that convolutional architectures benefit from inductive biases that remain advantageous for PPG signals. On our pre-training data, our model consistently outperforms alternative baselines. Furthermore, scaling experiments across model sizes reveal that brute-force scaling of generic architectures is possible, but less effective: our model achieves stronger performance and scales more gracefully due to better initialization and inductive structure relative to other models. In addition to this inductive bias and compact design, our contributions are two fold in the sense that our model demonstrates the first on-device model which does not require phone level processors to run inference.
Is your pre-training dataset large enough? Our pre-training corpus was collected internally and of comparable scale to recent public benchmarks such as PaPaGei and Apple's datasets. In terms of magnitude,
dataset PaPaGei (Pillai et al. 2025) < Ours < Apple (Abbaspourazad et al.] (023)) < Google (Narayanswamy et al. 2024). Thus, while not the largest available, our dataset size is sufficiently large validate the approach and lies within the range of accepted practice for self supervised learning wearable models.
position
Why do you model at 10-second windows? We deliberately adopt 10s windows sampled at 100Hz to balance physiological coverage with on-device feasibility. Many clinically actionable events, such as arrhythmic beats or premature ventricular contractions, unfold the order of seconds and require rapid detection to enable continuous monitoring and real-time feedback. Shorter windows would impair the model's ability to capture meaningful temporal context, while much longer windows would hinder low-latency inference on watch-class hardware. By constraining the receptive field to 10s, HiMAE preserves second-level resolution while remaining efficient enough to process signals continuously under the hardware limits of edge devices. Additionally, 10-second window are standard protocol that are adopted in the clinical setting where ECG for example is collected and interpreted at 10 second segments (Shuai et al.) 2016).
What are the advantages of smaller models? From a research perspective, smaller models foster inclusivity by reducing reliance on brute-force scaling of transformer-base architectures that only industry- scale labs can realistically afford. From a deployment standpoint, compact models enable on-device inference constrained hardware such wearables. This dual benefit- research barriers and wider deployment potential-underscomn the importance of investigating architectures that remain competitive at modest scale.
How large is too large to deploy on a smart watch? In principle, models up to approximately 50MB can be stored and executed on modern smart watches or larger models can be quantized (Jacob et al. 2017). In practice, however, latency and energy considerations suggest that models exceeding roughly 10MB may already hinder real-time inference and limit commercial viability. Additionally quantization does not do due dilligence to the original model and some level of the model's ' performance lost. While smartphones relax these constraints, our contribution highlights that the proposed model remains sufficiently compact to fit within the computational and storage budgets of wearable devices such as watches, thereby supporting direct on-device deployment.
Can PPG predict abnormal laboratory results? We frame this as binary classification task, test- ing whether photoplethysmmoraphy signal encodes biomarkers that separate "normal" from "abnor- mal" lab classes. Our investigation probes whether learned PPG representations capture biomarker signatures correlated with out-of-range labs, using lightweight classifiers on frozen embeddings with strict temporal alignment. Preliminary evidence suggests discriminative signal above chance, but these findings are designed to be exploratory and not clinically actionable.
37


Reviews:
[{"review_id": "dv0eiayg80", "reviewer": "Reviewer_5CPC", "summary": "This paper introduces HiMAE (Hierarchical Masked Autoencoder), a self-supervised learning (SSL) framework for wearable time-series data, primarily focusing on photoplethysmography (PPG) signals. The central hypothesis, termed the \"resolution hypothesis,\" is that different clinical and behavioral outcomes depend on predictive structures at distinct temporal scales. The key feature of the proposed architecture is that it produces multi-resolution embeddings from its different hierarchical layers. These embeddings can be independently evaluated (e.g., via linear probing) to determine which temporal scale (i.e., which receptive field) carries the most predictive signal for a specific downstream task.", "questions": "1. Could you please regenerate the main results figures (3, 4, 5, 15) using a high-contrast, colorblind-friendly color palette? It is currently very hard to distinguish the performance of the different models.\n\n2. Could you please clarify the discrepancy in the on-device benchmark? Why does the text state 0.99ms latency ($\\approx 1.01$k samples/s) while Table 18 reports a throughput of 1.2k/s? Which number is correct?\n\n3. The paper focuses on PPG, with a brief but promising result on ECG in the appendix (Table 16). Given that other physiological signals (e.g., accelerometry, electrodermal activity) also possess rich, multi-scale structures, do you have any intuition or preliminary results on how the HiMAE framework would perform on these other common wearable modalities? Or considering adding results on those public datasets for better reproducibility and verification?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "1. High Practical Significance (On-Watch Inference): The paper's most compelling strength is its focus on and successful demonstration of a \"true on-watch\" SSL model. By creating a model that is ~99% smaller (1.2M vs 110M params) and significantly faster (0.99ms CPU latency) than transformer baselines, the authors present a practical path toward real-time, continuous health monitoring while preserving user privacy (since data does not need to leave the device).\n\n2. Novel Interpretability Framework: The \"resolution hypothesis\" is an elegant conceptual framework. HiMAE's design, which \"transforms resolution from a hyperparameter into a probe for interpretability\", is a key strength. The resulting analysis in Figure 6, which maps tasks like \"PVC Detection\" to fine-grained layers and \"Sleep Staging\" to coarser layers, is a genuinely novel discovery tool that provides insight into the temporal structure of physiological signals.\n\n3. Experimental Rigor and Breadth: The paper is experimentally dense. The authors evaluate HiMAE on 14 different tasks against a strong set of SSL baselines (SimCLR, DINO, MSN, LSM, etc.). The inclusion of scaling law analysis (Figure 3), multiple generative tasks (Figure 4), few-shot learning (Figure 8), and extensive ablations (Appendix F.1) demonstrates a high-quality, thorough evaluation (assuming the results are verifiable).\n\n4. Strong Inductive Bias: The paper makes a strong case for not defaulting to transformers. It argues that the hierarchical convolutional biases of a U-Net are better \"inductively aligned with the temporal statistics of wearable signals\", which exhibit strong local dependencies. The results, showing HiMAE achieves lower error at much smaller model sizes, provide compelling evidence for this architectural choice.", "weaknesses": "1. Poor Presentation and Figure Quality: The paper suffers from a clear lack of polish. For instance, the choice of color palette is a significant problem. In Figures 3, 4, 5, and 15, the colors for competing models are nearly indistinguishable (e.g., \"PaPaGel-S\", \"SimCLR\", \"DINO\", \"MSN\", \"LSM\", \"HIMAE\" are all shades of blue/teal/green). This makes it extremely difficult to review the paper's core performance claims. This is particularly problematic for presenting clear comparisons against baselines, and maybe unreadable for colorblind readers. Additionally, multiple figures (e.g., Figures 3, 4, 5, 6, 14 and 15) needs either larger or smaller fonts for clearer readability.\n\n2. Unverifiable On-Device Benchmarks: The on-device latency (0.99 ms) and throughput (1.2k/s) figures in the text and Table 18 are inconsistent. The text calculation of $\\approx 1,010$ samples/s (which is $1/0.00099s$) seems correct, but Table 18 reports 1.2k/s. This discrepancy undermines confidence in the on-device benchmark.\n\n3. Proprietary Dataset: The pre-training corpus of ~80,000 hours is proprietary. While this is often unavoidable, when combined with the lack of model source code, it means the paper's results are entirely unverifiable from start to finish.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CHaWIYCHKr", "reviewer": "Reviewer_CnvE", "summary": "The paper proposes a Hierarchical Masked Autoencoder with a U-Net architecture for self-supervised pre-training on wearable signals. The model divides input PPG signals into patches and randomly masks partial signals for reconstruction. The stride-2 convolutional blocks downsample the input signal by a factor of 2 at each stage to enable multi-scale learning. The model is pre-trained on 80,000 hours of PPG signals collected from 47,755 subjects. Comprehensive downstream tasks, such as generative and classification, are included. Further case studies are conducted on on-device benchmarking to demonstrate real-world application ability.", "questions": "For the existing wearable PPG foundation model, PaPaGei, how did you compare with it? Did you load their pre-trained model and fine-tune it directly, or self-supervise and pre-train it on your dataset? If you pre-trained their model on your dataset from the beginning, please also report the linear probing result on their provided pre-trained weights.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The training set is large, with substantial hours and participants, which is good for alleviating subject-specific noise. The experiment is comprehensive, including different classification tasks, such as cardiovascular and sleep stages, which are the functionalities widely required for wearable devices. The model scale and total training time show the efficiency and applicability for wearable devices.", "weaknesses": "The method lacks novelty. Multi-scale learning is commonly used in medical/wearable time series representation learning, either through model-based or manual data preprocessing. Multi-scale learning using convolutional networks is more common in past research on time series analysis. Besides, it is better to have a comprehensive results table for comparison with baseline methods on different tasks. The results in Figure 5 are not straightforward enough.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pCsj2fu9l7", "reviewer": "Reviewer_uicj", "summary": "This work introduces a hierarchical masked autoencoder termed HiMAE to capture multi-resolution representations in PPG signals. The authors hypothesize that predictive information in physiological signals resides at distinct temporal scales and such multi-resolution information can be extracted by CNN-based encoder-decoder architecture relying on U-NET, and trained with a masked reconstruction loss. The model extracts embeddings at several temporal granularities, which are then evaluated across a distinct downstream tasks. HiMAE is shown to outperform or match large transformer-based baselines such as LSM with significantly fewer parameters and faster inference, even achieving on-watch inference capabilities.", "questions": "Please refer to the Weaknesses section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The idea that different downstream tasks may depend on representations at distinct temporal resolutions is both intuitive and well-motivated. Designing the architecture explicitly around this principle provides a clear and reasonable inductive bias for modeling physiological time series.\n- The proposed method is thoroughly evaluated across several downstream tasks, showing consistent improvements over large-scale baselines such as LSM while using substantially fewer parameters.\n- The emphasis on on-device inference is important and relevant for wearable applications. The authors convincingly show that HiMAE achieves remarkably lower latency and memory requirements compared to transformer-based counterparts, showing its practical compatibility with resource-limited wearable hardware.", "weaknesses": "- While the idea of resolution as interpretability is interesting, the paper primarily focuses on downstream performance differences across embeddings obtained from different HiMAE layers. Although this evaluation setup is reasonable, it is not clear that it fully substantiates the claimed interpretability of HiMAE. Visualization analyses of embeddings or frequency-response characterizations across layers could strengthen the interpretability argument.\n- The authors could provide more intuition or physiological grounding for why certain downstream tasks (e.g., sleep staging vs. cardiovascular detection) depend on finer versus coarser embeddings. Such a discussion would enhance both the interpretability and the practical implications of the resolution hypothesis.\n- The approach multiplies patches by their corresponding binary mask values, which may inject noise or misinformation during training. Have the authors tried other alternatives, such as dropping masked patches entirely or replacing them with a learnable mask token. This can also be an important design consideration when extending this work to other wearable signals that can contain large amounts of naturally missing observations.\n- It is not clearly stated whether the masked reconstruction loss used in the scaling analysis (Fig. 3) is computed on a held-out validation/test set or directly on the training data, particularly for data- and participant-scaling experiments. Clarification here is important for assessing generalization.\n- Several figures are difficult to interpret, which significantly affects the readability: the colors in Fig. 3 are nearly indistinguishable, the markers in Figs. 4–5 are very small, and the text in the confusion-matrix style visualization of Fig. 6 is hard to read. \n- Statistical significance or performance bounds are largely missing. Except for the error bars in Fig. 4, no standard deviations or confidence intervals are provided. Given that multiple baselines exhibit similar downstream decoding performance (e.g., Fig. 5; Tables 13–15), such statistical significance tests are necessary to assess whether improvements are meaningful.\n- The paper compares HiMAE only against the original LSM model, but LSM-2 is a more recent and stronger baseline shown to outperform LSM. While HiMAE’s on-device efficiency is important and valuable, a direct comparison with LSM-2 would better establish its standing among current state-of-the-art SSL methods for wearables.\n- Figure 6 provides an insightful layer-wise analysis for HiMAE. However, even though transformer layers are globally receptive by design, it is possible that they also evolve from encoding local to increasingly global relationships with depth. A similar per-layer analysis for LSM (e.g., comparing early vs. deep layers) would help determine whether the observed resolution-specific trends are unique to HiMAE or more general.\n- The paper proposes the local sufficiency notion in Appendix H.2 (Eq. 1) as a justification for the masked prediction objective, yet the concept is neither formally defined nor supported by prior references. Moreover, if local sufficiency truly held, it is unclear how the model could reliably reconstruct up to 80 % of masked patches from only 20 % visible input. The stable training observed under such high masking ratios suggests that PPG signals—and possibly the learned representations—encode substantial global temporal information, which appears inconsistent with a strict locality assumption.\n- The inability to release the codebase remains a key limitation. Providing even a minimal reproducibility package, such as a subset of data, a lightweight training/evaluation script would greatly improve transparency and contribute to the open-source community.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)