REBAR: Retrieval-Based Reconstruction for Time-series Contrastive Learning
Authors: Maxwell Xu
Alexander Moreno
Hui Wei
Benjamin Marlin
James Matthew Rehg
Venue: ICLR 2024 poster (2024)
Keywords: time-series
contrastive learning
masked reconstruction
self-supervised learning
imputation
unsupervised learning
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We introduce a novel method of identifying positives and negatives for time-series contrastive learning with Retrieval-Based Reconstruction (REBAR)

Abstract: The success of self-supervised contrastive learning hinges on identifying positive data pairs, such that when they are pushed together in embedding space, the space encodes useful information for subsequent downstream tasks. Constructing positive pairs is non-trivial as the pairing must be similar enough to reflect a shared semantic meaning, but different enough to capture within-class variation. Classical approaches in vision use augmentations to exploit well-established invariances to construct positive pairs, but invariances in the time-series domain are much less obvious. In our work, we propose a novel method of using a learned measure for identifying positive pairs. Our Retrieval-Based Reconstruction (REBAR) measure measures the similarity between two sequences as the reconstruction error that results from reconstructing one sequence with retrieved information from the other. Then, if the two sequences have high REBAR similarity, we label them as a positive pair. Through validation experiments, we show that the REBAR error is a predictor of mutual class membership. Once integrated into a contrastive learning framework, our REBAR method learns an embedding that achieves state-of-the-art performance on downstream tasks across various modalities.

REBAR: RETRIEVAL- RECONSTRUCTION FOR TIME-SERIES CONTRASTIVE LEARNING
Maxwell A. Xu', Alexander Moreno', Hui Wei Benjamin M. Marlin', James M. Rehg' Georgia Tech," 2 UIUC, UMass Amherst
ma xxxu exxuggatech.ddu jrehgeillinois edu
ABSTRACT
The success of sulf-supe contrastive learning hinges on identifying positive data pairs, such that when they are pushed together in embedding space, the space encodes useful information for subsequent downstream tasks. Constructing pos- itive pairs is non-trivial the pairing must be similar enough to reflect shared semantic meaning, but different enough to capture within-class variation. Classi- cal approaches in vision use augmentations exploit well-established invariances to construct positive pairs, but invariances in the time-series domain are much less obvious. our work, we propose novel method of using learned measure for identifying positive pairs. Our Retrieval Baled Reconstruction (REBAR) measure measures the similarity between two sequences as the reconstruction error that re- sults from reconstructing one sequence with retrieved information from the other. Then, if the two sequences have high REBAR similarity, we label them as a pos- itive pair. Through validation experiments, predictor of mutual class membership. Once integrated into a contrastive learning framework, our REBAR method learns embedding that achieves state-of- of-th- art performance on downstream tasks across various modalities.
show that the REBAR error is a
INTRODUCTION
Self-supervised learning uses the underlying structure within dataset to learn rich and generaliz- able representations without labels, enabling fine-tuning on various downstream tasks. This reduces the need for large labeled datasets, which is attractive for many machine learning tasks, and partic- ularly useful in the analysis of time series data for health applications. Due to advances in sensor technology, it is increasingly feasible to capture a large volume of health-related time-series data (Nasiri & Khosravani, 2020), but the cost of labeling this data remains high. For example, in mobile health applications, acquiring labels requires burdensome real-time annotation (Rehg et al., 2017) by participants. Additionally, in medical applications such as ECG analysis, annotation is costly as it requires specialized medical expertise.
Contrastive learning is powerful self-supervised approach to learning cemantically-maningge representations, which is based on constructing and embedding positive and negative pairs of unla- beled samples. In order to obtain useful representations, pairs should capture important structural properties of the data. In the vision applications that have driven this approach, augmentations are used to construct positive pairs by exploiting invariances of the imaging process (e.g. transfor- mations such as flipping and rotating that change the data vector without changing its meaning). Unfortunately, general time-series do not possess large and rich set of such invariances. Shifting, which addresses translation invariance, is uidely-used, but other augmentations such shuffling or scaling can destroy the signal semantics. For example, shuffling an ECG waveform destroys the temporal structure of the QRS complex, and scaling it can change the clinical diagnosis (Nault al., 2009). Morcover, there is no consistent consensus of augmentations in the literature; methods such TF-C (Zhang et al., 2022) incorporate jittering and scaling, while TS2Vec (Yue et al., 2022) finds that these augmentations impair downstream performance.
In this work, introduce novel approach for identifying positive pairs for time-series contrastive learning. Our key idea is that instead of generating positive pairs via augmentation, we use learned similarity measure identify positive pairs that naturally occur in extended time-series recordings.

---

Acceleronneeer Time-series
Sit Subsay
Walk Subseq
Subseq 2
a) Framing Time-series as a Composition of Class-labeled Subsequences
Good Reconstruction!
Query
Retrieval
Poor Reconstruction.
Query
Retrieval
b) Query other subsequences for information to retrieve be used to reconstruct the original
Figure 1: This figure demonstrates the intuition of our Retrieval-Based Reconstruction (REBAR) approach. If we can successfully retrieve information from another subsequence to aid in recon- struction, then the two subsequences should form a positive pair in contrastive learning frame- work. We first use the context-window designated by the grey box, of Walk Subseq 1 to query for information in Walk Subseq 2 or in Sit Subseq. Upper b) shows that the context window in Walk Subseq 2 provides good match with a similar double peak motif, leading to good reconstruction. Lower b) shows that Sit Subseq has no matching motif, leading to a poor reconstruction.
In our framework, we conceptualize a time-series as a composition of sequence of subsequences, each of which has class label. This framing describes many real-world physiological signals. For example, daily record of an accelerometry signal from a wrist-worn smartwatch contains many re- peated subsequences corresponding frequent activities like walking sitting. Each subsequence from the same activity class will in turn be comprised of brief temporal patterns or "motifs" such as "swing up" hand motion during walking. Likewise, the deflections in ECG signal due to the depolarization of the heart define motifs within the QRS complex (Bouaziz et al., 2014). If two sub- sequences contain similar motifs, then they are likely to share the same class label and are therefore a good candidate to form a positive pair.
We operationalize the idea of matching motifs with our Retrieval-Based Reconstruction (REBAR)) approach. In order to avoid explicitly modeling and detecting motifs, we adopt a reconstruction- based approach in which masked samples one subsequence are reconstructed directly from values retrieved from second, candidate subsequence. as illustrated in Fig. 1. A context window is taken around each masked sample, and the REBAR cross-attention model learns to compare each context window to the windows in the candidate subsequence to be retrieved for reconstruction. When two subsequences have many motifs in common, high quality matches can be obtained that minimize reconstruction error. Therefore, the REBAR reconstruction error is learned measure that captures motif similarity, and pairs with a lower error can then form positive examples contrastive learning. Such pairs are likely to share semantic meaning, that the resulting learned embedding space is class- diisssiicciminative. We are able to demonstrate this by showing that REBAR achieves state-of-the art performance a diverse set of time-series. The full REBAR approach can be seen in Fig. 2, and our public code repository can be found here: https: / gig.hb..bb.... com ma maxxx055/ rebar.
Our main contributions in this work are:
1. This is the first work to use a similarity measure to select positive and negative pairs in time- series contrastive learning. We do with our REBAR measure, which captures motif-similarity between subsequences using a convolutional cross-attention architecture.
We demonstrate that our learned measure predicts mutual class membership in a nearest neighbor sense, which validates that our positive pairs are implicitly capturing the subtle invariances within time series signals, as required for contrastive learning.
3. Our REBAR contrastive learning approach achieves SOTA performance against a representative set of contrastive learning methods that encompass the different ways which positive and negatives pairs can be generated. Note that our contrastive training method also beats fully- supervised training approach.
Note that we will interchangeably use "REBAR" to refer to the REBAR contrastive learning approach, the REBAR cross-attention, the REBAR measure. The specific meaning will be evident from the context.
2

---

2 RELATED WORK
Augmentation-basee Contrastive Learning: Augmentation-basee methods are the most studied type of contrastive learning method in time-series research, due the success of augmentation- based strategies in computer vision (He et al., 2020: Chen et al., 2020; Chen & He. 2021; Caron et al., 2021). However, it unclear which augmentation strategies are most effective for time-series, and the findings across different works are inconsistent. VS2 (Yue et al., 2022) uses cropping and masking to create positive examples, and their ablation study found that jittering, scaling, and shuffling augmentations led to performance drops. Conversely, TF-C (Zhang et al., 2022) included jittering and scaling, along with cropping, time-shifting, and frequency augmentations. TS-TCC (Eldele al., 2023) augments the time-series with either jiteeinngssallng or jittering+shuflinnn is in spite of how shuffling breaks temporal dependencies, and scaling changes the semantic meaning of bounded signal. Other augmentation works (Woo al., 2022; Yang & Hong, 2022; Yang al., 2022b; Ozyurt et al., 2022; Lee et al., 2022) also use some combination of scaling, shifting, jittering, or masking. Empirical performance was used justify the augmentation choice, but differences in datasets, architectures, and training regimes make it difficult to draw clear conclusion on what the best set of augmentations are. Our REBAR method instead uses sampling-bai approach to identify positive instances from set of real subsequences, rather than generating positive instance from a inconsistent set of augmentations.
This
Sampping-basse Contrastive Learning: After sampling an anchor subsequence, TLoss (Franceschi al., 2019) creates the positive subsequence as crop of the anchor and the negative as crop from a different time-series. CLOCS (Kiyasseh et al., 2021) samples pairs of temporally- adjacent subsequences and pairs of subsequences across channels from the same time-series as positives. TNC (Tonekaboni et al., 2021) randomly samples positive example from the anchor subsequence's s neighborhood region and an unlabeled example from outside. The neighborhood is found via a stationarity test, resulting in TNC's run-time being 250x slower than TS2 (Yue al., 2022), and utilizes a hyperparameter to estimate the probability that the unlabeled example is a true negative. Our REBAR approach is sampling-basse baased,, but unlike previous work, our positive amples are not selected based on temporal proximity to the anchor. Instead, positive examples are selected on the basis of their similarity to the anchor, measured by retrieval-based reconstruction.
Other Self-Supervised Learning Methods: CPC is a contrastive learning method learns to con- trast future points against incorrect ones (Oord al., 2018). There have also been contrastive learn- ing methods designed for specific sensor modalities with expert knowledge, such as for EEG data (Zhang et al., 2021). Another method is the Masked Autoencoder, which involves masked recon- struction but is fundamentally different from REBAR. See Appendix 1.5 for further discussion.
Time-Series Motifs: A motif is a brief temporal shape that repeats itself approximately across the time-series and is potentially class-discimminaive.. Much work has been done in identifying motifs via works such as matrix profile (Yeh et al., 2016; 2018; Gharghabi et al., 2018), and there are many classical time-series approaches that use template-matching methods to classify motifs (Frank et al., 2012; Okawa, 2019; Niennattrakul et al., 2012). However, instead of decomposing our time- series into specific motifs as the classical literature does, our REBAR method uses cross-attention to retrieve motifs that are useful in the context of reconstruction. Then, we can utilize the reconstruction error to capture motif-similarity in novel contrastive learning context for identifying positive pairs.
3 NOTATION
The dataset is designated by A E NxUxDD with N long time-series of U temporal length and D channels. A (0) E RUXD is the ith time-series the dataset. ( E T'x a subsequence of the
A (6) with length T,, where (() A (1) for brevity when not relevant. x E R refers to a specific time-point's 's data found in X.
7", for some t E N where T < U. (i) will be omitted
Throughout the paper, subscript is used to describe specific subsequence, Xdeseription Within cross-attention. X, and X3 designate the subsequences that serve as the query or key, respectively. A bar in X designates that X has been partially masked out. In contrastive learning, Xanebor is the anchor, and X is a candidate. We then identify which of the candidates Xanns should be labeled as positive, X Xpos or negative, Xneg-
3

---

Pre-Contrastive Learning Stage
Contrastive Learning Stage
Good
REDAR
Fillat IV
M MM
MMW
MMW
Fatret I's
Excoda
1y Train REBAR model without labels
2) Obtain Positive and Negative Labels
3) Train an encoder with contrastive on the previously ussigned labels
REBAR model
Figure 2: 1) First, our REBAR cross-attention trained to retrieve information from the key to reconstruct masked-out query. 2) Next, it is frozen and utilized to identify the positive instance. After sampling subsequences from the time-series, the subsequence that reconstructed the anchor with the lowest REBAR error labeled as positive, and the others labeled as within-time-series negatives. These negatives capture how time-series dynamics can change over time. Subsequences from other time-series within data batch are labeled as between- -ee---iiee---eeeeeeeeeeeee negiieseeeativess. and these negatives capture differences among patients. 3) We use the assigned labels to train an encoder.
4 REBAR APPROACH
Self-supervised contrastive learning methods learn an embedding by constructing positive and neg- ative instance pairs and then pushing the positive pairs together and negative pairs apart. In order to construct positive and negative pairs via retrieval, we designate one subsequence as the anchor, and use our REBAR measure to quantify the similarity between the anchor and other instances from the same time-series. The most similar instance forms a positive pair with the anchor, while the other instances, including instances from other time-series, form the negative pairs. Sec. 4.1 describes how we design our REBAR cross-attention module to produce the REBAR measure, and Sec. 4.2 explains how we apply the measure for sequence comparison contrastive learning framework. Sec. 4.3 tests the hypothesis that the REBAR measure can capture semantic relationships by demon- strating that REBAR predicts mutual class membership.
REBAR REBAR(Xanchoe anchor Xeand) cross-attention reconstructs Kanchor by retrieving motifs in Xeand that match the context window. Then, REBAR error serves as a distance measure? between two sequences, shown in Eq. 1. We hypothesize that if d(Xanchur, Xcand) is small, then it predicts if X is the same class as Xanchor (i.e. mutual class membership), allowing us to identify positive pairs.
d( X ancchor; Xeand)
|REBAR X anchor; Xcuna) Xanchur 112
4.1 DESIGN OF THE REBAR CROSS-ATTENTIC
We would like to design retrieval- batsee reconstruction error to be class-discrmminative. such that pairs with better reconstruction and lower distance are more likely to share classes and thus be se- mantically related. As such, REBAR identifies the motifs from the candidate, X that best match to the context window of the anchor, Xanehor and retrieves these motifs to reconstruct the anchor. For example, take the visualizations shown in Fig. 1. The error resulting from reconstructing Walk Subsequence I from the retrieved matching motif in Walk Subsequence 2 is lower than reconstruct- ing from the retrieved motif in the Sit Subsequence. The reconstruction performance is dependent on how closely the motifs in the candidate are able to match with the anchor, which allows for the REBAR measure to be class-discriminatie.
Cross-attention learns to produce weighted averages of transformation of the key time-series and an attractive method for modeling this paradigm. This because the retrieval function, p(x; (x,) shown in Eq. 2. can be interpreted as identifying the x that best matches Xg- Cross-attention is most commonly used with text acting as a query to retrieve relevant regions in an image (Lee et al., 201 8a; Miech et al., 2021; Zheng al., 2022), but it has been used occasionally for supervised
We refer this measure because is valid distance metric, violating strinlry incquality.

---

Dilated Cone A3
REBAR Cross-Attention
Query
1) Retrieval Vx, Xx (9(0RR))
excalsim (oa (1.8,,). ((ss))) nic Fagja exp mypisim ( Selmvi))
Σ
X,
fe
Dilated +
Key X1
Cume
m
Diltted
Recunsiruction Xg
Conre
2) Reconstruction
Figure 3: REBAR Cross-attenties reconstruction of single masked-out point x,, REBAR X+). Red designates the functions used for the attention weight calculation. In 1), the attention weights identify which region in X₁ should be retrieved for reconstruction by comparing the unmasked context around x, via the f, dilated convolution, with the motifs in X via the fa dilated convolution. In 2), the attention weights are used to retrieve an transformation of the key in weighted average for reconstruction. Minor model details (e.g. norms) are omitted for brevity.
time-series tasks (Garg & Candan, 2021; Yang et al., 2022a). We describe cross-attention below for given query time-point, N, (biases, norms, scaling factor, and linear layer omitted for brevity):
exp( x Wx)
CrossAttn (x.i Xx)= Σ N,EX,
We)
Ex ex, exp((x, WR. X Wx))
Σ p(xx(xx) (xs W.)
(2)
NxEX;
After generalizing (+-)) = sim(:.) and xW = (x) and reformulating for reconstruction with x, we have REBAR and its retrieval formulation in Eq. 3 and Eq. 4. respectively:
exp(sim (x,) fx(x))) N
REBAR (R, x,)=
Σ XLEX,
EX, exp(sim (g(x))) fu fx(x))))) fo (Xx)
(3)
Σ X4EX2
p(xa (x+x
(4)
The retrieval and reconstruction steps of our REBAR X,, X₂) cross-attention is visualized in Fig. 3. We utilize stacks of dilated convolutions for REBAR'S fa/a//s similar to WaveNet (van den Oord al., 2016), instead of a linear layer. This allows for the retrieval function's similarity function, sim(. -). to compare the motifs from the context window around K, with those in the window around N; (Xu et al., 2022). Vanilla cross-attention' 's linear layer only compares individual time-points with each other. This comparison is illustrated in Fig. 4. The retrieval function, p(xx Xx)), identifies regions surrounding an X₃ E X₃ that are useful for reconstructing X,, and then the fo consolidates information from that region for reconstruction. See Appendix A.1.1 for further details.
To restate: the objective of REBAR is to learn a dinsriiisrriiinative measure that will reconstruct well when X, has matching motifs with the Xg. but will reconstruct poorly when it does not. There- fore, we would like to emphasize a good retrieval function, p(xx|E,, that effectively compares and when
retrieves motifs and avoid complex model that may achieve accurate reconstruction X, and X are dissimilar. As such, we design our model so that the query is not directly used for reconstruction: is only used to identify regions in the key subsequence to retrieve with p(xx| In other words: for some function g {ATHT x (T D),
REBAR(X Xx) = ((p((X)XX,))
(5)
where AT is the Tddimensional probability simplex: that is, the reconstruction only depends on the query through the probability weights in Xx, E (ATYT. The model cannot simply borrow information from within the query to reconstruct itself. For example, if there are time-points on upwards line and the middle-time point is missing, the model is unable to directly learn simple linear interpolation to reconstruct that point. Instead, the model forced to identify a similar upwards line in the key and use this retrieved window for reconstruction. The model can only reconstruct the query from retrieved regions of the key with fo (xx). As such, reconstruction ability is directly dependent how similar the motifs in the key are the query.
5

---

Comparison
explsiml felx.))
REBAR Cress Alin
Vanilla Cross-Att
((xeRR)) Σ
X Exjex exp(skm (6112)) (oll)t(( (elme))
sim
Ouer
sim
Figure 4: Comparison of different fa/s within (X,)) fx()))). fx Convolution allows for ceaatttiiiall-mmmnnniiii motif comparison within the retrieval function, un- like in the vanilla's f == xW, in which single time-points are compared with another.
The REBAR's f == Dilated
Now, we note that training REBAR(X, Xx) to learn how to retrieve similar motifs for reconstruction is done in the pre-contrastive learning stage, and this should be done without labels so that we can later use REBAR to identify positive pairs for the self-supervised setting. However, given a random X,, X5 pair, we do not know if they share class labels, and thus we do not know if reconstruction error should be minimized to learn a motiif-macching similarity function between them. Therefore, during training,
set X, and Xg to be the same value, that REBAR learns motif similarity function that is able to retrieve the regions from the key X5 that match the missing region from the query, X, for reconstruction. Note that use X,, and X₃. to indicate two separate variables as inputs to pross-atteention, and their values can be the same or different. As previously noted, they are the same during training, but during application, when we use REBAR to identify positive pairs for contrastive learning, X, and X3 are different instances with different values, and REBAR uses its learned motif-retrieval function to reconstruct the query from the most salient motifs in the key. See Appendix A.1.2 for further details on masking methodology during training and application.
4.2 APPLYING OUR REBAR MEASURE IN CONTRASTIVE LEARNING
In contrastive learning, the anchor and positive instance are pulled together and the anchor and negative instances are pushed apart in the embedding space. Due to the REBAR measure's afore- mentioned class-disrimmnnatve properties (which further empirically validated in Sec. 4.3), we can use REBAR label candidate instances being positive or negative relative to the anchor.
The trained REBAR cross-attention used to attempt to reconstruct annchr from X Note that the anchor subsequence X and set of candidate subsequences, S are randomly sampled from the time-series A (+) Across all of our downstream experiments and datasets, we set SSead = 20. Then, we label the candidates either to be the positive, ( to be in the within-time-serine negative set (()) withii--eg neg based on the reconstruction performance. These labels are then used in our within-time-seree loss, Lww, modeled by NT-Xent (Sohn, 2016), shown below.
( pos
= argmin di ((x x(9) (S()) anchor cand
S wilhin neg
cand
expleos ( 1/7)
L log
Ex(t
exp(cos
X texpp x(i))/))
This within-time-seriee loss, Ly captures how a time-series can change class labels over time. learns to pull the anchor and the subsequence that is most likely to be of the same class as the anchor, according to REBAR, together in the embedding, while pushing those that are less likely, apart.
Next, in order to capture the relationships
the other anchor subsequences from the other time-series in our batch, A (i) with + i, are set be the between-ting seeee---im--seiiee negatives set, Shetween-neg. Then, along with our original X pos: we have our between-time-serien loss, Lz-
between-neg
anchor
(x
12 /T)
Cb log
(1) (4)
exp(cos( (X
unboe +T) (X()) man x(1)/7)
This between-time-sere loss, Lb, captures differences in time-series, well differences in pa- rients, because commonly, each time-series originates from a different patient. This approach is most similar to augmentation--aast methods (e.g. SimCLR) that draw their negatives from the batch.
We utilize a convex combination of these two losses from Eq. and 7 to create our final loss function, f. in Eq. 8, give the flexibility between emphasizing learning differences found within-ts or

---

between-ts. For example, for an accelerometry signals, could emphasize learning how activity changes over time for a given user by decreasing a, and for an ECG signal, we could emphasize how patient's 's heart condition is different from other patients' by increasing See Appendix A.1.3 for further discussion on how a is chosen and its impact on downstream performance.
== aLs + (1 -1--00 Lw with 0 S a $
(8)
4.3 REBAR NEAREST NEIGIBOR VALIDATION EXPERIMENT
Before using REBAR in contrastive learning, we assess whether the REBAR-identifce positive pairs are meaningful, by evaluating whether the REBAR measure effectively predicts mutual class membership. This validation experiment shown in Eq. 9 and is done by borrowing the class- labels that would typically only be used in downstream experiments. The labels used in nearest neighbor classification of an anchor, where distance is measured by REBAR.
P(Cpred = c. c(Crre) = EADND Extoaat
1 Came
argmin d(x DE (E(),,,C)
X
cand. 2)]
(9)
anchor,
This gives us a conditional probability of a predicted class being c, given that the anchor is of class Cure- One trial randomly segments anchor subsequence and one candidate subsequence from each of the C' classes. This trial is repeated for the given time-series, (9) and for all A ( in our dataset to obtain each empirical expectation estimate. The specific algorithm details are in Appendix 1.4.1. The confusion matrices in Fig. 5 help visualize REBAR's ' strong results.
HAR
PPG
ECG
609
0.00
Basdin
Labd
Walk
03
11
0119
031
K15
834
024
Stank
611
6.1
0.8
6.7
031
6/6
154
614
Prodiced Labd
Prok
Figure 5: There is high concentration on the diagonals of the confusion matrices across all of our datasets. This shows that REBAR, although trained with a reconstruction task without class labels, is able to predict mutual class membership, validating our idea for using REBAR to identify positive pairs in contrastive learning.
Our three datasets are further explained in the Sec. 5 and in Appendix A.2, but what is most im- portant to note is that each of them represent distinctively different sensor modalities. Given this, across all three distinctive domains, each true label's 's highest prediction label is still always itself, such that Crue = argmax, P(Cpeed c/Caue). This implies that match the anchor with its correct class more often than any of the other individual classes, and so us- ing REBAR to train our contrastive learning framework will encourage mutual classes to be pushed together in the embedding space over time. This validates our usage of REBAR the unsupervised setting. As previously noted in Sec. 4.2, we use REBAR to compare an anchor subsequence with set of randomly sampled candidate subsequences order to identify the positive candidate.
positive pairs will
5 DOWNSTREAM EXPERIMENTS AND RESULTS
In this section, we detail our experimental design for evaluating REBAR against other contrastive learning methods, and the results with ablation study are shown in Sec 5.1.
We aim to assess how differing methods perform based upon their contrastive ob- jective, so the encoder architecture (Yue al., 2022) is kept constant across all benchmarks. Each benchmark represents a specific time-series contrastive learning paradigm, and they are listed below. Further implementation details are found in Appendix A.3.
Benchmarks:
7

---

VS2
INC
CPC
SimCLR
SlidingMMSE
REBAR (ours)
Walk
Walk Up
Walk Dowa
Si
Stand
Lay
Figure 6: Qualitative Clusterability results with I-SNE Visualizations of each benchmark's 's encod- ings on the HAR dataset. Most methods are able encode "Lay" into its own cluster, and also "Sit" and "Stand" into nearby but generally distinct clusters. REBAR is the only method able to successfully separate out the "Walk", "Walk Up". and "Walk Down" labels into disjoint clusters.
TS2 Vee a strong augmentation based method with time-stamp representations (Yue al., 2022).
TNC is a sampling- sumpling-based method that samples nearby subsequence as positive and utilizes a hyperparameter estimate whether a distant sample is negative (Tonckaboni al., 2021).
CPC contrasts based on future timepoint predictions (Oord et al., 2018).
SimCLR is simple augmentation-base baaed method (Chen et al., 2020), which we have adapted for time-series with the most common augmentations (i.e. shifting, scaling, jittering).
Sliding-MSE a simplified REBAR that instead uses sliding-MSE comparison as a measure.
Data: We utilize 3 datasets from 3 different sensor domains with time-series that have their classi- fication labels change over time: Human Activity Recognition (HAR) with accelerometer and gyro- scopic sensors to measure activity (Reyes-Ortee
al., 2015), PPG to measure stress (Schmidt al., 2018), and ECG to measure heart condition (Moody, 1983). Each of these modalities have drasti- cally different structures, and the class-specific temporal patterns vary differently within modality. Please find further dataset descriptions and visualizations in Appendix A.2.
Downstream Evaluation: To evaluate their diassddierimminaiiee strengths, learn linear probe (i.e. logistic regression) on each model's frozen encoding for downstream classification and use the Accuracy, AUROC, and AUPRC metrics quantify the results. Additionally, fully supervised model composed of an encoder, identical to that used by the baselines, with linear classification layer is benchmarked. This matches our baselines' linear probe evaluation, only trained end-to-end. We then also assess cluster agreement for further corroboration. After k-means clustering of the frozen encoding, with as the number of classes, we assess the similarity of these clusters with the true labels with the Adjusted Rand Index and Normalized Mutual Information metrics.
5.1 RESULTS
Linear Probe Classification: Tbl. 1 shows that the linear probe trained on our REBAR representa- tion consistently achieved the strongest results, even beating the fully supervised model in PPG and HAR, achieving the same accuracies, but higher AUROC and AUPRC. For ECG, REBAR achieves better accuracy, but lower AUROC and AUPRC. This demonstrates our REBAR's methods strength in learning representation that is better at handling class imbalance than the fully-supervised model.
HAR
Model
Accuracy
AUROC
1 AUPRC
Accuracy
1 AUROC
AUPRC
AAceracy
AUROC
AUPRC
Fully Supervised
0.9535
0.9835
0.9531
0.4138
0.6241
0,3689
0.7814
0.9329
0.9260
VC2
0.9324
0.9931
0.9766
0.4023
0.6428
0.3959
0.7612
0,8656
0.8516
TNC
0.9437
0.9937
0.9788
0.2989
0.6253
0.3730
0.7340
0.8405
0.8195
CPC
0.8662
0.9867
0.9438
0.3448
0.5843
0.3642
0.7775
0,8377
0.8223
SimCLR
0.9465
0.9938
0.9763
0.3448
0.6119
0,3608
0.6992
0.8254
0.8063
MiSd
0.9352
0.9931
0.9767
0.3335
0.6456
0,3831
0.7751
0,8755
0.8574
REBAR (ours)
0.9535
0.9965
0.9891
0.41.3
0.6977
0.4457
0.8154
0.9146
0.8985
Table 1: Linear Probe Classification Results with Accuracy, AUROC, and AUPRC
Our REBAR method demonstrates a much stronger performance than Sliding-MSE, showing the necessity of learning a tify positive pairs. Our improved performance compared to TNC highlights the value of identifying positives that are not necessarily near the anchor. SimCLR's 's subpar results demonstrate that even if common time-series augmentations are used, this does not guarantee strong performance and the set augmentations should be tuned. Although TS2Vec a state-of-thee oftt-----hh-an art method, it is unable consistently achieve the strongest performance among the other benchmarks. We suspect that
distance rather than using a simple measure to iden-
8

---

this is because VS2 evaluated on short class-labeled time-series rather than time-series with class-labeled subsequences. REBAR's ' sampling- aapliin-bassed approach successfully exploits this structure to sample positive pairs from subsequences across the time to achieve strong performance.
Clusterability Evaluation: Tbl. 2 shows that when measuring the cluster agreement with the true class labels, REBAR continues achieve the best ARI and NMI, corroborating the strong classification results. This is unlike other methods, such as TS2vec in PPG, that achieve strong linear probe results, but low cluster agreement. Additionally, the t-SNE visualizations shown in Fig. 6 for HAR and in Appendix A.4.2 for remaining datasets show that REBAR's s encodings push mutual classes together and distinet classes apart, even distinct classes that are semantically similar, such as "Walk", "Walk Down", and "Walk Up". These results support the idea that REBAR'S embedding space particularly glass-disrimminaive.
HAR
PPG
ECG
Model
ART
NMT
1 ART
1 NMT
1 ARI
NMT
TS2
0,4654
0.6115
-0.0353
0.1582
0.2087
0.1701
TNC
0.4517
0.5872
0.0958
0.1666
0.2186
0.1753
CPC
0.1603
0.22117
0.1110
0.1867
0.0532
0.0724
SinCLR
0.5805
0.6801
0.1535
0,3081
0.2182
0.1751
Miding-MMS:
0.5985
0.7019
0.1083
0.2141
-0.001
0.0180
REBAR (ours)
0.6258
0.7721
0.1830
0.3422
0.2260
0.1796
Table 2: Clusterability Results with Adjusted Rand Index and Normalized Mutual Information
REBAR approach analysis: Fig. 7 visualizes the positive pairing that was identified by our REBAR measure from a randomly sampled set of candidates for given anchor for the HAR dataset. We that even when there is no exact match of the anchor within the candidates, REBAR's motif- comparison retrieval and reconstruction is able to identify a positive example that shares the same class as the anchor. Please find large gallery of positive pairing visualizations in Appendix A.4.4.
Candidates
Walk Anchor
Positive Cand
Anchor
Positive Cand
Am
(tadd)
(Will))
(Unahaedd)
(Wallt
Walk Up Ancher
Dositive - Cand 4
Stand Anchor
Positive - Cand 16
WWm
(Wall Town) 12
(hill)
(Wil))
10
13
Fundmng
(Walk Up)
(Sland)
(bak))
Fay 16
(aa))
(.g)
Walk Down Anchor
Positive Cand 1
Lay Auchor
Positive
Cand 12
15
17
18
19
MMM
(er))
(laakeed))
(Stanh
(haal
(Walk Down)
(las)
Figure 7: Positive pairings identified by REBAR from the candidates, for an anchor of each class.
The key component of our REBAR cross-attention design is the dilated convolutions used for motif comparison and retrieval. We find that removing and replacing these convolutions with the vanilla linear layer, results in performance drops of 6.4% decrease in accuracy of the linear probe, which is worse than all but one of the benchmarks, and a 34.1% NMI decrease in the clusterability evalu- ation. Additionally, we find that REBAR is fairly robust against hyperparameter tuning. When we modify the size of the dilated conv's 's receptive field, the size of masks, or REBAR cross-attention reconstruction training epochs, performance remains consistent. See Appendix A. for further details and additional specific model ablation results.
6 CONCLUSION
In this paper we introduced REBAR, novel approach to time-series contrastive learning. By using cross-attention to retrieve class-specific motifs in one subsequence to reconstruct another subse- quence, we can predict mutual class membership. Then, if we use this REBAR measure to identify positive pairs, we are able to achieve of results in learning a class discriminative em- bedding space. Our REBAR method offers new perspective in time-series self-supervised learning with our measure- measure-focused approach, and we hope that this work will drive future research into how to best capture and encode semantic relationships between time-series.

---

7 ACKNOWLEDGEMENTS
We would like to thank Catherine Liu for her help and support on this work. This work sup- ported in part by NIH P41-EB002242-01AA1 NIH 1-R00-CA224537-011 and the National Science Foundation Graduate Research Fellowship under Grant No. DGE-2039655. Any opinion, findings, and conclusions recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
ETHICS STATEMENT
Our paper works on creating models for health-related signals, and it has the potential to improve health outcomes, but at the same time could lead to loss of privacy and could possibly increase health-related disparities by allowing providers to characterize patients more fine-grained ways. In the absence of effective legislation and regulation, patients may lack control over of their data, leading to questions whether autonomy, a key pillar of medical ethics, are being upheld. Overall though, hope that work leads to personalized health recommendations, allowing patients to receive improved care and achieve better health outcomes, directly contributing patient safety and overall well-being.
positive it helps further the field towards creating
REPRODUCIBILITY STATEMENT
Our Methods section in Section 4 details the way in which we set-up our method, and our Experi- ments section in Section 5 details our experimental design. Additionally, the Appendix A.3, we itemize each of the hyperparameters we used to tune each of our benchmarks. Upon acceptance, we will release our GitHub code publicly, which will have the set seeds and exact code we used to experiments. We will also make model checkpoints downloadable. The datasets used are publicly available, and we describe how we curate each of them for our task in Appendix A.2. Additionally, code can be found at https: http//gitttt /ggttbb.. com coom/maxeee maxxu05 rebar.
REFERENCES
Fatíha Bouaziz, Daoud Boutana, and Messaoud Benidir. Multiresolution wavelet-based qrs complex detection algorithm suited several abnormal morphologies. IET Signal Processing, 8(7):774- 782, 2014.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. Proceedings of the IEFF/CVF international conference on computer vision, pp. 9650-9660, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning. pp. 1597-1607. PMLR, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 15750-15758 2021.
Mingyue Cheng, Qi Liu, Zhiding Liu, Hao Zhang, Rujiao Zhang, and Enhong Chen. Timemae: Self- supervised representations time series with decoupled masked autoencoders. arXiv preprint arXiv:2303.00320 2023.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee-Keong Kwoh, Xiaoli Li, and Cuntai Guan. Self-supervised contrastive representation learning for semi-supervie time-series classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Jean- Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. Unsupervised scalable representation learning for multivariate time series. Advances in neural information processing systems, 32, 2019.
10

---

Jordan Frank, Shie Mannor, Joelle Pineau, and Doina Precup. Time series analysis using geometric template matching. IEEE transactions on pattern analysis and machine intelligence, 35(3):740- 754. 2012.
Yash Garg and K Selçuk Candan. Sdma: Saliency-driven mutual cross attention for multi-variat time series. In 2020 25th International Conference on Pattern Recognition (ICPR), pp. 7242- 7249. IEEE, 2021.
Shaghayegh Gharghabi, Shima Imani, Anthony Bagnall, Amirali Darvishzadeh, and Eamonn Keogh. Matrix profile xii: Mpdist: novel time series distance measure to allow data mining in more challenging scenarios. In 2018 IEEE International Conference Data Mining (ICDM), pp. 965-970. IEFE, 2018.
Priya Goyal, Dhruv Mahajan, Abhinav Gupta, and Ishan Misra. Scaling and benchmarking self-supervised visual representation learning. In Proceedings of the IERE/CVF International Conference on Computer Vision (ICCV), October 2019.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 9729-9738, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Seongsil Heo, Sunyoung Kwon, and Jaekoo Lee. Stress detection with single ppg sensor by orches- trating multiple denoising and peak-detecting methods. IEEE Access, 9:47777-47785, 2021.
Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ho Choi, and Jacgul Choo. Re- versible instance normalization for accurate time-series forecasting against distribution shift. In International Conference on Learning Representations, 2021.
Dani Kiyasseh, Tingting Zhu, and David A Clifton. Clocs: Contrastive learning of cardiac signals across space, time, and patients. In International Conference on Machine Learning, pp. 5606- 5615. PMLR, 2021.
Harim Lee, Eunseon Seong, and Dong-Kyu Chae. Self-supervised learning with attention- latent signal augmentation for sleep staging with limited labeled data. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, UCCA1-22, LD Raedt, Ed. International Joint Conferences Artificial Intelligence Organization, volume 7, pp. 3868-3876, 2022.
Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. Stacked cross attention for image-text matching. In Proceedings of the European conference on computer vision (ECCV), pp. 201-216, 2018a.
Wonki Lee, Seulgee Kim, and Daeeun Kim. Individual biometric identification using multi-cycle electrocardiographic waveform patterns. Sensors, 18(4):1005, 2018b.
Zhe Li, Zhongwen Rao, Lujia Pan, Pengyun Wang, and Zenglin Xu. Ti-mae: Self-supervised masked time series autoencoders. arXiv preprint arXiv:2301 008871, 2023.
Guilin Liu, Fitsum Reda, Kevin Shih, Ting- Cing-Chun Wang, Andrew Tao, and Bryan Catanzaro. Image inpainting for irregular holes using partial convolutions. In Proceedings of the European conference on computer vision (ECCV), pp. 85-100, 2018.
Antoine Miech, Jean- Jean-Baptiste Alayrac, Ivan Laptev, Josef Sivic, and Andrew Zisserman. Thinking fast and slow: Efficient text-to-visual retrieval with transformers. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pp. 9826-9836, 2021.
George Moody. A new method for detecting atrial fibrillation using IT intervals. Proc. Comput. Cardiol., 10:227-230, 1983.
11

---

Sara Nasiri and Mohammad Reza Khosravani. Progress and challenges in fabrication of wearable sensors for health monitoring. Sensors and Actuators A: Physical, 312:112105, 2020.
Isabelle Nault, Nicolas Lellouche, Seiichiro Matsuo, Sébastien Knecht, Matthew Wright, Kang- Teng Lim, Frederic Sacher, Pyotr Platonov, Antoine Deplagne, Pierre Bordachar, et al. Clinical value of fibrillatory wave amplitude on surface ecg in patients with persistent atrial fibrillation. Journal of interventional cardiac electrophysiology. 26:11-19, 2009.
Vit Niennattrakul, Dararat Srisai, and Chotirat Ann Ratanamahatana, Shape-based template match- ing for time series data. Knowledge-Basde Based Systems, 26:1-8, 2012.
Manabu Okawa. Template matching using time-series averaging and dtw with dependent warping for online signature verification. IEFE Access, 7:8 7:81010-81019, 2019.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807 007488, 2018.
Yilmazcan Ozyurt, Stefan Feuerriegel, and Ce Zhang. Contrastive learning for unsupervised domain adaptation of time series. arXiv preprint arXiv:2206.06243, 2022.
James
M Rehg. Susan A Murphy, Mobile Health: Sensors, Analytic Methods, and Applications. 10.10077997-333119-5134444
and Santosh Springer, 2017.
Kumar. doi:
Jorge Reyes-Ortiz, Davide Anguita, Luca Oneto, and Xavier Parra. Smartphone- -Based Based Recognition of Human Activities and Postural Transitions. UCI Machine Learning Repository, 2015. DOI:
Patrick Schäfer and Ulf Leser. Motiflets: Simple and accurate detection of motifs in time series. Proceedings of the VLDB Endowment, 16(4):725-737, 2022.
Philip Schmidt, Attila Reiss, Robert Duerichen, Claus Marberger, and Kristof Van Laerhoven. In- troducing wesad, a multimodal dataset for wearable stress and affect detection. In Proceedings of the 20th ACM international conference multimodal interaction, pp. 400-408, 2018.
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. Advances in neural information processing systems, 29, 2016.
Sana Tonekaboni, Danny Eytan, and Anna Goldenberg. for time series with temporal neighborhood coding. International Conference of Learning Representations, 2021.
Unsupervised representation learning
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing gredient for fast stylization. arXiv preprint arXiv: (tl:::1607000222. 2016.
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio, 2016
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Patrick Wagner, Nils Strodthoff, Ralf-Dieter Bousseljot, Dieter Kreiseler, Fatima Lunze, Wojciech Samek, and Tobias Schaeffter. Ptb-xl, large publicly available electrocardiography dataset. Scientific data, 7(1):154, 2020.
Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. Cost: Contrastive learning of disentangled seasonal-trend representations for time series forecasting. arXiv preprint arXiv:2202.01573 2022.
Maxwell Xu, Alexander Moreno, Supriya Nagesh, Varol Aydemir, David Wetter, Santosh Kumar, and James M Rehg. Pulseimpute: A novel benchmark task for pulsative physiological signal imputation. Advances in Neural Information Processing Systems Dataset and Benchmarks Track, 35:26874-26886 26888, 2022.
12

---

Ling Yang and Shenda Hong. Unsupervised time-series representation learning with iterative bi- linear temporal-spere -peectall fusion. In International Conference on Machine Learning, pp. 25038- 25054. PMLR, 2022.
Sitan Yang, Carson Eisenach, and Dhruv Madeka. Mqremnn: Multi-horizon time series forecasting with retrieval augmentation. arXiv preprint 1XXx:2207.10511,, 2022a.
Xinyu Yang. Zhenguo Zhang, and Rongyi Cui. Timeclr: A self-supervised contrastive learning framework for univariate time series representation. Knowledge- Based Systems, 245:108606, 2022b.
Chin-Chia Michael Yeh, Yan Zhu, Liudmila Ulanova, Nurjahan Begum, Yifei Ding, Hoang Anh Dau, Diego Furtado Silva, Abdullah Mueen, and Eamonn Keogh. Matrix profile i: all pairs similarity joins for time series: unifying view that includes motifs, discords and shapelets. In 2016 IEEE 16th international conference on data mining (ICDM), pp. 1317-1322. Ieee, 2016.
Chin-Chia Michael Ych, Yan Zhu, Liudmila Ulanova, Nurjahan Begum, Yifei Ding, Hoang Anh Dau, Zachary Zimmerman, Diego Furtado Silva, Abdullah Mueen, and Eamonn Keogh. Time series joins, motifs, discords and shapelets: a unifying view that exploits the matrix profile. Data Mining and Knowledge Discovery, 32:83-123, 2018.
Zhihan Yue, Yujing Wang. Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu. Ts2vec: Towards universal representation of time series. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8980-8987, 2022.
Hongjun Zhang, Jing Wang, Qinfeng Xiao, Jiaoxue Deng, and Youfang Lin. Sleeppriorcl: Con- trastive representation learning with prior knowledge-based positive mining and adaptive temper- ature for sleep staging, arXiv preprint arXiv:21 10. 00099666, 2021.
Xiang Zhang, Ziyuan Zhao, Theodoros Tsiligkaridis, and Marinka Zitnik. Self-supervised con- trastive pre-training for time series via fime-frequency consistency. Advances Neural Information Processing Systems, 35:3988-4003, 2022.
Fuzhong Zheng, Weipeng Li, Xu Wang, Luyao Wang, Xiong Zhang, and Haisu Zhang. cross- attention mechanism based on regional-level semantic features of images for cross-modal text- image retrieval in remote sensing. Applied Sciences, 12(23): 12221, 2022.
13

---

A APPENDIX
A.1 REBAR APPROACH DETAILS
We use this section of the appendix to explain and justify the architectural and implementation details of the full REBAR approach. In Sec. A.I.I, we further explain the specific REBAR cross- attention architecture details, including the dilated convolution block used for motif comparison and retrieval. In Sec. A.1.2, we explain the masking procedure we use during training and application of our REBAR aooss-attention. Sec. A.1.3, we explain how we choose an a in our combined loss function, emphasizing within-time-seriee interactions and/or setween-time-setee its effects on downstream performance. Sec. A.1.4, we detail our ablation study and parameter robustness analysis.
interactions, and
A.1.1 REBAR CROSSIATTENTION ARCHITECTURE DETAILS
REBAR Cross-Attention
fask/v Dilated Conv
WiW
pilaad
Cruos
WWWh
Dilled
Diiind Caews
a) Block diagram version of our REBAR Cross-Altn
b) Individuul companents of the fark/u Dilated Conv fimction
Figure 8: REBAR Cross-Attention Architecture
Fig, 8a) contains the full block diagram version of the REBAR cross-attention originally shown in Fig. 3, with of the model details included, such as the RevIN block used for normalization and the aggregation linear layer, used to combine information across heads for final reconstruction. Below we explain each the components and the rationale behind including/excluding them.
Dilated Convolution: We modified version of the BDC transformer proposed in Pulselm- pute (Xu et al., 2022), which is visualized in Fig. 8b). The instance norm (Ulyanov et al., 2016) is used to normalize the inputs, and instance norm is used instead of batch norm so that the model is not dependent on batch size. The Ixl layer acts as bottleneck layer, reducing di- mensionality. This allows for the model to be able efficiently stack convolution layers with exponentially increasing dilation factors, thereby exponentially and efficiently increasing the re- ceptive field. The residual connections are designed to help to facilitate more efficient training. Please see in the Sec. 4.1 in the main text for explanation of the importance of using dilated convolution for fg/k/v rather than vanilla cross-attention's 's linear layer.
Positional Encoding Exclusion: Positional encoding was proposed in the original transformer model (Vaswani et al., 2017) order to address the permutation equivariance problem of self- attention, that the model would have positional awareness of each of the input tokens. How- ever, as noted in Pulselmpute (Xu et al., 2022), the addition of dilated convolution removes the permutation equivariance limitation. We opt not any positional encoding because would like the retrieval function, p(xx (x,). to retrieve regions in the key subsequence based on similarity of the temporal shapes within the key and query in order to assess motif-similarity. rather than considering the specific positions of such structures. Additionally, knowing the ex- act positions is uninformative due to the arbitrary time which sensors begin recording or are segmented. Thus, we seek to instead model relative position through the convolutions that can capture a subsequence's shape.
Partial Convolution: To handle masking, we do drop-in replacement of all convolutions with their partial convolution counterpart (Liu et al., 2018). Partial convolutions ignore the calculation at masked points and reweigh the weights based on the total number of weights that were used in the calculation.
(RevIN) originally proposed to help address temporal distri- bution shifts in forecasting tasks, and uses an ordinary instance norm at inputs and then reverses the normalization at the output (Kim al., 2021). We use RevIN normalize the query and key
RevIN: Reversible instance
14

---

inputs with respect to the query, then is reversed at the end to unnormalize the outputs for recon- struction.
A.1.2 REBAR MASKING PROCEDURE
We must identify effective masking strategy train REBAR(X,, Xx)) learn retrieve and compare class-specific motifs (e.g. heartheat in ECG (Schäfer & Leser, 2022)) and also to apply REBAR (Xanchor Xcand) effectively identify meaningful positive and negative examples from the candidates. We define our binary mask as m and two different types of specific masks, extended and transient, below.
{1 for missing
where m IRT
0 for not-missing
Wextended (i: ri] with ER (1, ,T], else.
Meransient [1, 1. in] = 1 with é1, , ER
0 else.
An extended mask where a contiguous segment of length randomly masked out, and tran- sient mask is where n time stamps are randomly masked out. ER designates random sampling without replacement.
Training REBAR q Xx) before Contrastive Learning: The key idea is that we want learn RE- BAR such that the reconstruction error class- disssdiicrrmiiatiiee, and the model's reconstruction ability direction depends on the retrieval function.
Fig. 9b) shows that training with the transient mask results in dispersed attention distribution, which means that the model is retrieving minor, non-unique motifs. Even if two Xx have two different class labels, we would be able to retrieve this non-unique motif from both X so that the reconstruction performance would not be classddiscrminnatve.
Fig. 9a) shows that the attention weights learned with a extended mask fit exactly onto the region in the key corresponding to the masked region in the query, and thus the model has learned to retrieve specific motifs useful for reconstruction. Because the REBAR model has learned to compare specific motifs between the query and key, when comparing class discrimiriei motifs, the learned reconstruction performance will be dias-discriminative.
REBAR's Attention Weights during Training
Subsequence Reconstruction
Query Subsequence Reconstruction
Legend
Query series Input Reconstraction
Key Subsequence used for Reconstruction
Key Subsequence used for Reconstruction
Query Loc for Attn Weight Viz
Weights
Mrew
a) Trained with an Extended Mask
b) Trained with Transient Mask
Figure 9: a) A extended mask encourages REBAR to learn attention weights that are specific to the corresponding motif in the key. Conversely, with an transient mask. b), the attention is diffused and attends to transient motifs throughout the key. Specific attention weights allow for an effective comparison of dass-discriminntive motifs, so we utilize a extended mask for training.
Applying REBAR(Xanchen Xcand) for Contrastive Learning: During application, the REBAR cross-attention has already been trained to capture potentially class-discriminative motifs, and we use the REBAR reconstruction error to identify pairs that are likely to share a class, so that they can be labeled as positive examples. Then, when these positive pairs are pulled together within the embedding space, they encode class-discriminative information. Therefore, during application, want utilize the trained REBAR cross-attention in way that holistically compares sequence with another, so that we have a better understanding whether they share mutual classes. As noted in Equation we know that pXxx (X,) ) is the only place in which the query, X,, is used for reconstruction. Thus, look at visualizations of p(Xx) (x,) for extended mask sient mask, in Fig. 11, to understand what motifs within the query are compared to the key, during reconstruction.
sub-
tran-
15

---

P x for Exiended Musk on Query
P
P
PO
The motifs in the union of receptive field regions are compared to the blue key sequence (munnnn
P(Xs/Xx, x
for Transient Mask Query
PO
P
PC
Figure 10: Comparison of receptive fields of the query being considered in the query key motif comparison found in p(X,|K,)) We can see that the union of receptive fields of the fy Dilated Convolution covers more area when reconstruction is done over transient mask, rather than extended mask.
At application time, a extended mask could be used, however, in doing so, only the motifs near the contiguously masked out region would be compared the key. This because when reconstructing given masked time-point and another point contiguous to it, the receptive fields to be used to identify motifs be compared to the key are heavily overlapping, see the top of Fig.
A transient mask allows for many different motifs in the query, each of them captured in recep- tive field around the many masked time-points dispersed throughout the signal, to be compared to the key during reconstruction. This allows for a higher coverage of the query in conducting such motif-similarity comparisons with the key. Therefore, during application, when we are testing each candidate as the key for given anchor the query, would be able to identify the candidate that most similar to the anchor as a whole. See the bottom of Fig. 10.
A transient mask preserves more information about the surrounding context of masked out query point compared to an extended mask, and we see that a model trained with a extended mask can still reconstruct signal masked with transient mask, seen in Fig. 11. The vice versa would likely not be true and is reflected in the empirical results shown in Table 3. Even though the masking setting has changed, each of the reconstructions still maintains sparse attention, which implies the retrieval function is still specific and comparing specific, potentially class disrrmmiiiiiii,,,,,, motifs between the query and the key.
REBAR's Attention Weights during Application
MAb Qurrry SuGeeserce Recerdruction
Vod #6c Querr Sabongannc Reconcraction wiglleeleguuunooooo
Mie #11 Query Sabsecannce reccestuction
Lepmal
www. ausecsaeece used Reconitratiin
Quarry Suberqqencce Irput Query Aln Weight
Subresseecc used Reconmnation
sence spe Recesonation
with opoo
Moy Subsequence input Weights
Query Key the same instance
Query and Key from the same class but are different instances
Query and Key from different classes
Figure 11: We use transient mask during application evaluate motif similarity on wider range of motifs present throughout the query, and the sparse attention in c) d) e) show that the model still conducting specific motif comparison. c) and d) utilize keys that differ from the query, but our REBAR cross-attention still retrieves the most relevant information in the key for query reconstruction. The attention weights are conditioned on masked-out downwards dip the query, and each of the weights in c) d) e) are highest for corresponding dip in the key. d)'s key is in the same class as the query, and the corresponding MSE is lower than e)'s.
16

---

Masking Procedure Empirical Validation Experiments: As we in Table 3, Training Extended + Application Transient achieves the best performance in our validation experiment, followed by Training Extended Application Extended, then Training Transient + Application Transient, then Training w/ Transient + Application Extended.
Training Mask Type Evaluation Mask Type
Extended Extended
Extended Intermittient
Intermittient Intermittient
Intermittient Extended
Accuracy
0.3624
0.4580
0.2888
0.2378
Table 3: Comparison of Different Masking Procedures for REBAR. The accuracy reported is from the validation strategy explained in Section 4.3 averaged across all classes.
A.1.3 EMPIASSIZING WITHIN-TIME-SERIN INTERACTIONS vs. BETWEEN-TIME-SERIS INTERACTIONS OUR LOss FUNCTION
The coefficient in our loss function in Equation gives us the flexibility to learn an embedding space that emphasizes how time-series changes over time when decreasing vs. how a time-series differs across patients when increasing a. We hypothesize the best a for each modality.
For ECG, these signals measure the electrical propagation throughout the heart, and each patient's ECG signals is very specific to their own heart and how a specific cardiac condition, such as Atrial Fibrillation, affects it. There is even research showing that ECG signals could be used to potentially identify an individual due to the unique position, shape, size, and structure of the heart (Lee et al., 2018b). Therefore, for ECG, it would make more sense utilize loss function that emphasizes differences among patients with an a = 1.
For HAR, as we see in the time-series visualizations in Appendix A.2, each of the different activ- ities accelerometry signals are quite distinet from each other within a given user's time-series, so learning how time-series changes over time with an a is the most useful for distinguishing classes.
For PPG, this signal modality shares similar properties from both ECG and HAR. Because PPG measures blood volume changes, it models unique properties of how user's heart pumps blood, but it also is important in how models how a specific user responds to stress. Therefore, we use a combination of the two loss functions with an 0.5.
For each of the datasets, we benchmark three different values of a E (0, 0.5, 1} for having only the loss function, both loss functions, only the between-timer time-series loss function, respectively. Please see the results below in Table 4.
ECG
Accuracy AUROC AUPRC
ARI NMI Accuracy AUROC AUPRC ARI
NMI Accuracy AUROC AUPRC ARI
NMI
o=0
0.9535
0.9965
0.9891
0.6258 0.7721 0.4713
0.7215
0,4528
0.0408 0,1 0.8298
0.8946
0,8775 0.1995 0,1470
o=.5
5
0.9549
0,9941
0.9802
0.4787 0.6273
0.4138
0.6977
0.4457 0.1830 0.3422
0.7944
0.9339
0.2263 081000 0.9286
0.9056
0.9898
0,3077 0.4430 0.4138
0.6398
0.3816 0.0586 0.2242 0.8154
0.9146
0,8985 0.2260 0.1796
Table 4: Benchmarking REBAR model with different values of a within our loss function in Eq. 8. a= is only the within-time-series loss function, a=1 is only the berween-time-series loss function, and a=.5 is an equal contribution between the two of them. Bold the variant that achieved the best balance of high linear probe performance and high clusterability, and it is what we reported within our results in the main text, Sec. 5.1.
REBAR achieves strong performance under all loss function variants, but switching between a values results in different trade-offs. For example, in ECG, the model we end up reporting in our results uses an a = 1, and it achieves an accuracy of .8154 and NMI of .179, but if we use a = 0, we can achieve higher accuracy of .8946, but a lower NMI of 0. 1470. Then, in PPG, when then we achieve the highest accuracy of 0.4713, but the NMI is only 0. 0.1163, which is much lower than 0.5. where NMI is 0.3422. In our main results, we utilize the that provides the best balance between high linear probe performance and high clusterability, and these chosen a values align with prior hypotheses.
0.
17

---

Reconstruction Error During REBAR Training
0.04
0.02
Figure 12: As the REBAR Cross-Attention trains with the extended mask, reconstruction error decreases, but reconstruction error is not necessarily an indicator of downstream performance.
A.1.4 REBAR
ABLATION STUDY AND HYPERPARAMETER ANALYSIS
Acc
AUROC
AUPRC
ARI
NMI
Baseline at Epoch 300, Mask= 1S, RF=43
REBAR
0.9535
0.9965
0.9891
0.6258
0.7721
Ablation (Lower better)
w/o DilatedConv
0.8930 (-0.061)
0.9839 (-01130)
0,9302 (-0.059)
0.2845 (-0.341)
0.5094 (-0.263)
w/o RevIN
0.9451 (-0.008)
0.9958 (-0.001)
0.9861 (-0.003)
0,5613 (-0.065)
0.6450 (-0.127)
w/ PosEmbed
0.9479 (-0.006)
0.9957 (-0.001)
0.9863 (-0.003)
0.5550 (-0.071)
0.6522 (-0.120)
Robustness (Equal or Higher is better)
REBAR Epoch 100
0.9592 (+0.006)
0.9970 (+0.000)
0.9900 (+0.001)
0.6169 (-0.009)
0.7589 (-0.013)
REBAR Fpoch 200
0.9592 (+0.006)
0.9967 (+0.000)
0.9896 (+0.001)
0.6830 (+0.057)
0.8425 (+0.070)
REBAR w/ Mask=5
0.9493 (-0.004)
0.9963 (+0.000)
0.9881 (-0.001)
0.6120 (-0.014)
0.7.50
REBAR w/ Mask=25
0.9592 (+0.006)
0.9962 (+0.000)
0.9873 (-0.002)
0.6484 (+0.023)
0.7891 (+0.017)
REBAR w/ RF=31
0.9507 (-0.003)
0.9962 (+0.000)
0.9881 (-0.001)
0.6390 (+0.013)
0.7787 (+0.007)
REBAR w/ RF=55
0.9380 (-0.016)
0.9943 (-0.002)
0.9794 (-0.010)
0.6570 (+0.031)
0.8090 (+0.033)
Failure Cases (Lower is better)
REBAR at Epoch
0.9437 (-0.010)
0.9954 (-0.001)
0.9841 (-0.005)
0.3635 (-0.290)
0.5213 (-0.251)
Table 5: REBAR model Ablation Study and Hyperparameter Analysis on HAR data
Ablation Study of Model Components: Tbl. 5 demonstrates that each of the REBAR cross- attention model's design contributes to its strong contrastive learning performance, especially the dilated convolutions (e.g- .061 drop in accuracy and .263 drop in NMT). This makes sense because the dilated convolutions enable the motif comparison and retrieval. The exclusion of our reversible instance norm leads .008 drop accuracy and a .127 drop in NMI, and the addition of an explicit positional embedding leads to a ,006 drop in accuracy and .120 drop in NMI. As noted in Section 3. we intentionally keep the design of REBAR attention simple emphasize the motif comparison within the coos-attention mechanism.
Effect of Proportion of Masking: The model fairly robust against the size of the extended masks to be used for training the REBAR model. In our ablation study in Tbl. 5, compared to an initial mask size of 15, we found that using a smaller mask size of time-points lowered accuracy by only 0.004 and using larger mask size of 25 increased accuracy by .006.
For each of the datasets, the size of the extended mask is chosen to large enough to mask out potentially class-discriminative motif. This is done so that the model learns how to specifically trieve this ciias-idicrmmiaativee motif from the key and so that during evaluation, discrimiicii motifs between the key and query compared. However, the mask size cannot be too large. be- cause then even if the query and key are of the same class, would be more difficult to precisely match much larger motif with the increased noise.
18

---

Thus, extended mask size be tuned with hyperparameter grid search with expert domain knowledge. In HAR, it is a mask that covers .3 seconds (15 time-pes sime-ooints) because the accelerometry type data typically captures information of movement from short quick jerks. In PPG, it is a mask that covers 4.69 seconds (300 time points), which we do because PPG wave morphology is relatively simple, so we would like capture few quasi-periood In ECG it is mask that covers 1.2 seconds (300 time points), which covers a heartbeat, and ECG wave morphology is generally very descriptive and specific (e.g. QRS complex).
This also helps guide the size of the receptive field that our dilated convolutions in our cross-attention should be. In general. we aim for the size of the receptive field to be three times the size of the extended mask, so that during training, the query dilated convolution would be able to effectively capture the motif surrounding the masked out query region to learn our cross-attention model. Additionally, as we see in the Tbl. 5, our model is robust to the exact receptive field size. Compared to an initial receptive field of 43, a receptive field of 31 decreases accuracy by only .003 while increasing ARI by ,013 and a receptive field of 55 decreases accuracy by only ,016 while increasing ARI by .031.
Effect of Dilated Convolution Receptive Field Size:
Effect of Reconstruction on Downstream Performance: Tbl. 5 shows that REBAR is robust to the hyperparameter of Number of REBAR Cross-Attention Epochs Trained. Fig. 12 shows that reconstruction emror decreases epochs trained increases, reconstruction accuracy is not necessarily an indicator of downstream performance. This is not an issue because the REBAR Cross- Attention's 's goal within contrastive learning framework is not to achieve a good reconstruction. Instead, we would like the REBAR Cross-Altention to learn good retrieval and motif-comparison function between the query and the key that achieves better or worse reconstruction, based on motif similarity. As noted in Section 3.1, we intentionally keep the design of REBAR attention simple to emphasize the motif comparison within the cross-attention mechanism. We note that these results should not imply that the REBAR model should not be trained. Tbl. 5 in the Appendix shows that REBAR is only trained for epoch, then accuracy drops by 0.010 and NMI drops by 0.251.
A.1.5 COMPARING REBAR TO MASKED AUTOENCODER
The Masked Autoencoder (MAE) has been primarily used in NLP (Devlin et al., 2018) and Vision (He et al., 2022), with a few works in time-series (Cheng et al., 2023; Li et al., 2023). These models typically utilize transformers to encode the masked input before reconstructing with a lightweight decoder. While both REBAR and MAES involve masked reconstruction, the fundamental ap- proaches differ substantially. REBAR estimates a pair of subsequences are of the same class via paired reconstruction error and then uses this to contrast subsequences. In comparison, MAEs learn seveen-timee-oine interactions within a subsequence by learning a reconstruction as a composition of the unmasked data and is not designed to explicitly compare subsequences.
A.2 DATASET DETAILS
The time-series are split into a 70/15/15 train/val/test split. We note below that the total time cov- erage of the class-labeled subsequences does not equal the total time coverage of the time-series that they are segmented from, because some of the datasets contain parts that were unlabeled. Ad- ditionally, each of the class-labeled subsequences are non-overlapping.
HAR: Rather than using the extracted time and frequency features, we opt to use the raw accelerom- eter and gyroscopic sensor data (Reyes-Ortiz et al., 2015) to better assess how our methods perform on raw time-series. Our subsequences are 1.56-second-long (128 time points) to match the subse- quence length proposed for classification in the original work (Reyes-Ortiz et al., 2015). There are 6,914 distinct subsequences total for training and testing our models, which are sampled from 59 5-minute-long (15,000) time-series that are sampled at 50 Hz and has 6 channels. There are 4,600 class-labeled subsequences, with the following labels: walking (17.7%), walking upstairs (7.6%), walking downstairs (9.1%), sitting (18.2%), standing (20. 1%), and laying (20.1%).
PPG: From the WESAD dataset (Schmidt et al., 2018), we use 1-minute-long (3840 time points) subsequences to match the subsequence length proposed in the original work (Schmidt et al., 2018). There are 1,305 distinct subsequences total for training and testing our models, which are sampled from 15 87-minute-lome (334,080 time points) time-series, sampled at 64 Hz with a single channel. There are a total of 666 class- subsequences, with the following labels: the labels are baseline
19

---

Doapy YT
Stadd
Lay
Down
Walk Up
Figure 13: Example of one channel of HAR Signal
(42.7%), stress (24.0%), amusement (12.4%), and meditation (20.9%). The data has been denoised following the procedure in Heo et al. (2021).
Baseliae
Stress
Meditation
Aumseneent
Figure 14: Example of PPG Signal
ECG: From the MIT-BIH Atrial Fibrillation dataset (Moody, 1983), we use 10-second-long (2500 time points) subsequences. The original work (Moody, 1983) does not have a proposed subsequence length, use this 10-second-long subsequence length match the prior work for this dataset (Tonckaboni et al., 2021) and the 10-second-long length used in general ECG classification works (Wagner al., 2020). There are 76,590 distinct subsquences total for training and testing mod- els, which are sampled from 23 9.25-hour-loww
(8,325,000 ,,000 time points) time-series sampled at 250 Hz with dual channels. There is a total of 76,567 10-second-long (2500 time points) class-labeled subsequences, with the following labels: atrial fibrillation (41.7%) and normal (58.3%). These sub- sequences originate from 23 25-houur (8.325,000 time points) tmeessrriss sampled at 250 Hz with dual channels.
Albb
Figure 15: Example of one channel of ECG Signal
A.2.1 DISCUSSION ABOUT UCR/UEA TIME-SERIES DATABASE
We note that we opt to not benchmark on the commonly utilized UCR and UEA time-series dataset databases, they do not fit our stated problem area of time-series that series of class-labeled subsequences, 2) would prevent us from benchmarking sampling-basce contrastive learning meth- ods, and 3) many of datasets found within the databases are very small.
Our work is framed for long time-series data that can be described as a series of class-labeled sub- sequences, rather than collection of short our datasets have time-series that are 8,325,000/ / 334.080 15,000 time points long, compared to the median length 621 and 301 for UEA and UCR respectively. Our defined subset of time-series data is the closest primitive to true time-series sensors, in which sensors are designed to collect large volumes of unlabeled data continuously and autonomously. describes the physiological sensor time-series domain, in which the advancement of wearable tech- nologies has enabled large data collection on a user's changing physiological states, but the cost of labeling remains high (i.e. mHealth sensors require constant user annotation and clinical sensors require medical expertise). Hence, we believe that this particular time-series domain has the largest opportunity for impact in the MI. and greater health community.
time-series (c.g. UCR. UEA). For context,
note that this setting particularly
20

---

Using this type of long time-series data in our problem also allows benchmark sampling- based contrastive learning methods, such as TNC (Tonekaboni et al., 2021) and our own REBAR method. These sampling-based methods fully exploit the time-series nature of our task. Because each subsequence is part of a specific user's sensor time-series that is changing over time, we can sample positive and negative examples directly from the larger time-series, rather than attempting to create positive examples from augmentations. Unfortunately, the datasets that support this type of task are limited: TNC only included HAR, ECG, and a small simulated dataset. Our work adds the PPG dataset for stress classification, to introduce new benchmark dataset for this type of time- series contrastive learning task.
Additionally, many of the datasets within UEA and UCR may not suitable for evaluating self- supervised learning. due to their small sizes. Self-supervised learning has risen into prominence due to how it exploits large-scale unlabeled datasets, and such methods have been found to be sen- sitive to dataset size (Goyal et al., 2019). However, 90 out of the 128 datasets in UCR and 123 out of the 183 datasets in UEA have 1,000 fewer sequences to train a model. 8 out of the 128 total datasets in UCR and 14 out of the total 183 total datasets in UEA have 100 or fewer sequences. After segmenting our time-series into the contiguous subsequences that used to train each of our self-supervised learning models, we have a total number of 76,590 / 6,914 1,305 sequences for ECG, HAR, and PPG, respectively.
A.3 BENCHMARK IMPLEMENTATIONS
We Our code has been made publicly available at https://githts http://///ii.tbb...omaaox/////// use the validation set to choose the best model checkpoint to be evaluated on the test set, and tune each model's hyperparameters to achieve its best performance.
For the non-sampling- based methods (i.e. TS2Vec, CPC, SimCLR), we train these methods on sub- sequences. The subsequence sizes used during the contrastive learning stage match the downstream subsequence sizes used for classification: length 128 for HAR, length 3840 for PPG, and length 2500 for ECG. The encoder is kept constant across each of our methods, and we used the encoder from TS2vec (Yue et al., 2022) with an embedding size of 320.
Tully-Superviser uses the same encoder and linear classification head, trained with Mass-balancee weights and cross entropy loss, identical to the other linear probe models, only trained end-to-end
-The PPG and HAR domains were trained with a learning rate of .00001 until convergence, and the ECG domain was trained with a learning rate of ,000001 until convergence.
TS2Vec (Yue et al., 2022) is a current state-of-ther -frt-- time-series contrastive learning method that learns time-stamp level representations with augmentations.
We follow the default implementation used in https: / /ttpt/gtttt/ttt/// com com/yuuzhihaa/////oo ts2vec. In PPG, we have a learning rate of ,0001 and batch size of 16, in ECG, we have a learning rate of 0.00001 and batch size of 64, and in HAR, we have learning rate of 0.00001 and batch size of 64.
TNC (Tonekaboni et al., 2021) samples nearby subsequence as positive and utilizes hyper- parameter to estimate whether a faraway sampled subsequence is a negative.
We follow the default implementation used in https:/ / /github. com/ s sanat onek/ learning and set w to .2. in PPG, we have a learning rate of .0001 and hatch size of 16, in HCG, we have learning rate of .0001 and batch size of 16. and in HAR, we have learning rate of .00001 and batch size of 16. At the end of the encoder, utilize global max pooling layer to pool over time.
TNC ces__rppesentttiinn
Note that our reported TNC results are higher than the original reported results because we use TS2Vec's 's proposed dilated convolution encoder instead of TNC's originally proposed encoder. This was done in order to fairly compare all of the contrastive learning baselines: this encoder backbone constant across all of the baselines. Tec also found that training TNC with the dilated convolution encoder achieves better results (See Appendix C.3 in (Yue et al., 2022)). Additionally, other discrepancies may originate from how TNC evaluated their method by training a classification head with the encoder model end-to-end, rather than freezing the encoding. We opt to freeze the encoder in order to allow for the encoded representation to be directly evaluated via linear classifier (i.e. Acc, AUROC, AUPRC) and clusterability (i.e. Adjusted Rand Index and Normalized Mutual Information).
21

---

CPC (Oord et al., 2018) contrasts based on future timepoint predictions.
We follow the default implementation used in https; / /github...... com/ jefflail0f/ Cont rast ive redi ct istive ve Coding PyTorch. For PPG, we have a learning rate of .001 and batch size of 16, for ECG, we have learning rate of .0001 and batch size of 16, and for HAR, we have a learning rate of .001 and batch size of 64.
SimCLR (Chen et al., 2020) is simple augmentation-basee method we have adapted for time- series. The most common time-series augmentations are used (i.c. shifting, scaling, jittering), allowing to assess how suitable pure augmentation-basee strategy is.
We have 3 augmentations: scaling, shifting, and jittering, with each of the three having a 50% probability of being used. Scaling multiplies the entire time-series with number drawn from (00.5, 1.5). Shifting shifts the time-series by a random number between -subsequence size to subsequence_sizee size. Jittering adds random gaussian noise to the signal, with the Gaussian noise's s standard deviation set to .2 of the standard deviation the values in the entire dataset. For PPG, we have a learning rate of .001, T is I, and batch size of 16, for ECG, we have learning rate of .001, r is.001, and batch size of 16, and for HAR, we have a learning rate of .001, is 1, and batch size of 64. At the end of the encoder, utilize global max pooling layer to pool over time.
Sliding-MSE is simpler version of REBAR, in which a simple method is used to assess the sim- ilarity of a pair of sequences. We slide the candidate subsequence, padded with the true sequence values, along the anchor subsequence and calculate MSE at each iteration. The lowest MSE of the slide is then used as our measure to label the positive and negative examples.
For PPG, we have a learning rate of _001, 20 sampled candidates, and T is 1000, for ECG, have learning rate of .001, 20 sampled candidates, and is .01, and for HAR, have a learning rate .001, 20 sampled candidates, and r is 0.1. We match the a value used for each dataset, according to the best used by REBAR. At the end of the encoder, we utilize global max pooling layer to pool over time.
REBAR is our Betrieval-Basede Reconstruction method for time-series contrastive learning
Our cross- attsenttt model was trained to convergence and the dilated convolution block has an embedding size of 256 channels, initial kemel size of 15 and dilation of 1. The dilation gets doubled for each following layer. The input channel size of the first convolution equal to the number of channels present in the data, and the bottleneck layer has size 32. For the PPG and ECG datasets, we have 6 dilated convolution layers to capture larger receptive field of 883 and for the HAR dataset, we have 2 layers to capture a receptive field of 43. During training, our extended mask sizes for PPG, ECG, and HAR are 300, 300, and 15 respectively. During evaluation, the transient mask masked out 50% of the time points of the signal. During contrastive learning, for PPG data, we have a learning rate of ,0001, 20 sampled candidates, is 10, is 0.5, and batch size of 16, for ECG, we have learning rate of ,001, 20 sampled candidates, 7 is ,01, a is 1, and a batch size of 16, and for HAR, we have learning rate of .001, 20 sampled candidates, is .1, is O, and batch size of 64. At the end of the encoder, we utilize a global max pooling layer to pool over time.
22

---

A.4 EXTRA RESULTS AND DETAILS
A.4.1 REBAR NEAREST NEIGHBOR VALIDATION EXPERIMENT ALGORITHM
Algorithm is the procedure for generating the confusion matrix from the REBAR Nearest Neighbor
Validation Experiment detailed in Sec. 4.3 with Eq. 9. This algorithm uses the notation that has been established throughout the paper, please Sec. 3 for notation details.
Algorithm 1 Confusion Matrix from REBAR Nearest Neighbor Validation Experiment
Require: E R Nxtxx
> N time-series with length U and dimensionality D p total number of classes, C total number of trials, Trials subsequence length, T
Require: C> 2
Require: Trials
Require: 0 < T U
procedure WEARESTNEGHHORRCOCRUUSSOM..L.. C, Trials, 2)
M <- zeros(C. C)
for i range(N) do
for Omne in range(C) do
for in range(TTrils)) do
Randomly segment an anchor subseq with Coue unchor, + RandSegment("" Cre))
X(1)
# Randomly segment candidate subseq from each class x cad,1 + RandSegment(A 11))
X(0
RandSegment(A (A (9) C)
C
Assert: x
anchorr cand, Cmd E RT
x(9
# Nearest neighbor prediction of the anchor's class with distance measured by REBAR measure
Gpeed < argmin d(x(1)
x(O a
anchar,
cef1,.. .C}
# Update confusion matrix
M Cinue, Gprea] + M Caue: Gpeed] +
MCmmee:3 ] < M[ [Cmee :) Trials
return M
A.4.2 FULL T-SNE VISUALIZATIONS
HAR
PPG
ECG
SeCLE
Thingg
RETRAX
SoutLa
RSBAR (oan))
SeacLi
Slicing Mie
RIBAR bam
8 'Vl.
V9ll.1y 6 19l Dove 0.5. Sund km
Bine
0.3 Nermal
Figure 16: Qualitative Clusterability Results with t-SNE visualizations for all three datasets: HAR, PPG, and ECG. n HAR, all methods except for REBAR group walk, walk up, and walk down together. In PPG, all of the methods have poor elustering. but REBAR seems to perform the best, with only the Meditation label being not clearly disjoint. In ECG, REBAR and TS2Vec qualitatively have the best intra-class clustering.
A.4.3 INVESTIGATING FALSE POSITIVES AND FALSE NEGATIVES
False Positives: Unlike the experiments detailed Sec. 4.3, Fig. 5, and Appendix A.4.1, where subsequence is drawn from each class to be compared to the anchor, in this section, we segment 20 subsequences randomly from the time-series, which matches our contrastive learning experiments for REBAR across all of our datasets, giving us the confusion matrices seen below in Fig. 17. As result, these 20 candidates can be any distribution of classes.
candidate set size used in
23

---

HAR
PPG
ECG
TARKING
8:10
CAL
CCE
0..00
06
931
Tha Tale
021
604
0.0
04
EX0
0.30
097
011
641
1.
01
Wakind
Prodeced Land
Prodited Land
Prodeced
Figure 17: Confusion matrix showing predictions of positive instance being the same class as the anchor given candidate set size of 20.
We see that with candidate set size of 20, REBAR achieves a True Positive Rate of predicting the positive instance to be the same class as an anchor of 0.79, 0.68, 0.96 for HAR, PPG, and ECG respectively. The chance of achieving false positives with our candidate size of 20 within our contrastive learning procedure is quite low, and any false positives that do occur are likely to be semantically related to the anchor's true class.
False negatives may occur in both the within-ts negatives and/or the between-ts negatives due to the random sampling of candidates and randomness present in the batch, respec- tively. However, note that in general, false negatives are fundamental issue within contrastive learning more broadly and not specific to our REBAR method. For example, in augmentation-base approaches like SimCLR (Chen et al., 2020), negative instances sampled from the batch, and the batch may contain instances from the same downstream class label as the anchor. However, they able to still achieve strong performance. Similarly, our approach is still able to achieve the best representation learning performance across our baselines even with the potential presence of false negatives. One idea to address this could be thresholding method for our within-time-series nega- tives, but we believe that this method would be ineffective. A threshold should not be designed to be a hyperparameter as would be dependent on how each specific anchor's specific motifs compare to other general instances within the same class. However, it is not obvious how to construct the threshold as function of the anchor because during contrastive learning, we do not have access to the ground truth class labels.
False Negatives:
A.4.4 EXTRA VISUALIZATIONS FOR POSITIVE PAIRS
The visualizations from the main text in Fig. 7 and below were randomly selected and each represent
distinct random sampling of candidate and anchor subsequences from different subject's 's time-
series. The accuracy of our positive candidate identification via REBAR is quite high, and each of the figure captions helps describe and justify any misclassification.
Mua
Walk Anchor
Positive Cand 15
Anchor
Positive Cand 16
(Suas))
(Wil Downd
(Will) Positive Cand 18
(Sit)
Walk Up Ancher
Stand Ancher
Positive Cand
Bledll 10
(Uaidicie)
Mallidell 12
(tanh)
day)
Mmma
(Walk Up)
(aa))
(Uaskieed)
Maikidell
Walk Down Auchor
Desitive - Cand 2
Lay Anchor
Positive - Cand 15
15
17
A
Caldebed
(Walk) Figure 18: 6/6 anchors of each class correctly matched with candidate that shares that class
(Woll Ugo
(Kilk)
(Walk Down)
Candidates
Walk Anchor
Positive Cand 17
Sit.oncbor
Positive Cand 16
mnmm
Saedl
(Rill)
(tani)
Unp
(Unlahlldd)
Walk Up Ansher
Positive = Cand 14
Stand Anchor
Positive = Cand 13
NAAA
Juma Unaazidd
(hill)
maaaa
10
12
13
(Walk Up) Positive = Cand 6
(Slana)
Ualdekal
Layi
6i6
Dewe)
Bay) 19
Walk Down Anchor
Lay Anchor
Positive
Cand 9
18
1h
11
18
M
M
(Las) Figure 19: 5/6 anchors are correctly matched. Although candidate 12 is a "Sit" among the 20 candidates, REBAR matches the "Sit" anchor with "Unlabeled" candidate 16. This seems to be
(Lar)
Duashi
(wal Up9
Drandi
Ahiab:dedi
(Walk Jown)
because, even though Candidate 16 is "Unlabeled" it is qualitatively very similar to the anchor.
24

---

Candidates
Anchor
Posilive 17
Positive 12
m
MalA (Wil))
Stadd)
(Nill)
Oradi
Ahiaiidd
(Walk) Posilive - Cand 11
(Unlabeled) Pasilive - Cand 15
Walk Up Ancher
Sland Ancher
(Wak 12
(lmakazd)
10
11
13
14
(Walk Up)
(Stand
(Wak))
1.90
(1a9)
(Wll tpo
Walk Down Anchor
Positive = Cand 4
Lay Anchor
Pesitive = Cand 9
15
17
18
19
(Uauheed)
(fran))
(Saxa)
(Uniaheled)
(Lay)
Figure 20: 4/6 anchors are correctly matched. There is no "Walk Down" subsequence among the 20
candidates, but REBAR matches the "Walk Down" anchor with an "Unlabeled" candidate, which still is qualitatively similar to the anchor. The "Sit" anchor is matched with a different "Unlabeled" candidate, which not ideal. However, there only "Sit" subsequences among the candidates and matching with an "Unlabeled" class is better than mismatching with specific different class (e.g. matching "Sit" with "Walk").
Candidates
Walk Anclur
Positive Cand 17
Sii Ancher
Positive Cand 10
M
JMMA
m
~~~~~~
Saadl
(Vaakaed)
(Wila)
Slandi
(kill Dosai
(Walk) Positive = Cand 6
Walk Up Ancher
Stand Ancher
Positive
Cand 15
MWW
Walk Dovay
(Unakeldd)
(Wak
(Nill)
Ahiakdali
m
~
(Walk Up)
(Stand)
Walk Down Anchor
Positive Cand
Lay Anchor
Positive Cand
15
16
17
18
19
M
(aa))
(Sead))
(Wal Eower)
(fand)
(aandh
(Walk Desrn)
(Lay)
Figure 21: 6/6 anchors of each class are correctly matched with candidate that shares that class
Candidates
Walk Anclure
Positive
Caud
Sii Anchor
Positive Cand 18
ADDA
m
Stadi
(Kali)
(Wil))
Blasdi
(Wak Up)
(Walk) Positive = Cand 4
Walk Up Ancher
Stand Aneher
Positive
Cand 13
KAALL
(lal Daven)
(hili)
Ahiahdali
10
(Walk Up)
(Stadd)
Wall
Walk Down Anchor
Positive Cand 6
Lay Anchor
Positive
Cand
13
16
17
18
19
(1ar)
(Sand)
/Wak 129
(Saad)
(Walk Deorn)
(Walk)
Figure 22: 5/6 anchors are correctly matched. The "Lay" anchor is incorrectly matched with a
"Walk" candidate. This may be because it is diffieult to do effective motif comparison and retrieval with the flat motifs found in "Lay" query. This is also seen in our Nearest Neighbor experiments shown in Fig. 5, where the flatter signals (i.e. "Sit", "Stand", "Lay") have worse mutual class prediction accuracy compared to the more dynamic signals (i.e. "Walk", "Walk Up", "Walk Down")
Candidates
Walk Anchor
Positive Cand
Anchor
Positive Cand 16
m
mmmmmmm
6ro
(Kill)
(Wik)
(ttnd)
Walk Upi
Walk Up Ancher
Positive = Cand 4
Stand Anchor
Positive = Cand 15
(Unabuid)
Myr
10
12
13
(Walk Positive = Cand II
(Stand) Positive Cand 5
(1ae)
(laakocd)
6i6
Walk Dewn)
Walk Down Anchor
Lay Anchor
15
18
19
MMAA
M
(Las) Figure 23: 6/6 anchors of each class are correctly matched with candidate that shares that class
(Wal Doweo
Deandi
(Walk Down)
25


Reviews:
[{"review_id": "ZoVJoHrfgv", "reviewer": "Reviewer_gCTH", "summary": "This paper introduces a novel approach called Retrieval-Based Reconstruction (REBAR) for self-supervised contrastive learning in time-series data. The REBAR method utilizes retrieval-based reconstruction to identify positive data pairs in time-series, leading to state-of-the-art performance on downstream tasks.", "questions": "1. Ablation study and hyperparameters selection.\n2. Include more visualizations or examples of the positive and negative pairs identified by the REBAR method\n3. \"During evaluation, we use an an intermittent mask\", explain the intuition why different masks are used in the training and evaluation\n4. How does the REBAR method perform on time-series data with different characteristics, such as varying lengths or noise levels?\n5. Provide more detailed explanations of the convolutional cross-attention architecture used in the REBAR method.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Novel approach: The paper introduces a novel approach called Retrieval-Based Reconstruction (REBAR) for self-supervised contrastive learning in time-series data. This approach utilizes retrieval-based reconstruction to identify positive data pairs in time-series, which is a unique and effective way to address the challenges of creating positive pairs via augmentations in time-series data.\n\n2. State-of-the-art performance: The paper demonstrates that the REBAR method achieves state-of-the-art performance on downstream tasks across diverse modalities, including speech, motion, and physiological data.\n\n3. Comprehensive evaluation on two tasks including classification and cluster agreement.", "weaknesses": "1. Lack of ablation studies: The paper does not include ablation studies to analyze the contribution of each component of the REBAR method. This makes it difficult to understand the relative importance of each component and how they interact with each other.\n\n2. Detailed explanation of the results. There is no detailed studies for table 1 and 2, e.g., visualizations of the learned embedding or positive/negative pairs. \n\n3. Limited discussion of hyperparameters: While the paper provides some details about the hyperparameters used in the experiments, it does not provide a comprehensive analysis of the sensitivity of the method to different hyperparameters. \n\n4. Without comparison with baselines, Figure 4 doesn't show any advantages of the proposed model since the diagonal pattern would be obvious for most of the baselines.\n\n5. Section 3.1, notations are used without clear definition", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vpypeO49Lx", "reviewer": "Reviewer_jiyG", "summary": "The paper proposes a novel method for constructing positive pairs for contrastive learning in time-series data. It presents experiments across three datasets to validate the approach.\n\n--- post rebuttal ---\n\nI appreciate the efforts made by the authors in addressing the concerns raised in my initial review. The manuscript has undergone significant changes, resulting in notable improvements in its quality. Considering these enhancements, I have revised my score from 3 to 6.", "questions": "In Table 3, the author evaluates the influence of different mask types on performance. It would be beneficial to clarify why the training stage favors a contiguous mask, while the evaluation stage shows a preference for an intermittent mask.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper is motivated. The proposed method is grounded on a cogent hypothesis *\"if one time-series is useful for reconstructing another, then they likely share reconstruction information and thus should be drawn together within the embedding space as positive examples.\"*. Essentially, the author posits that time-series with similar semantics are capable of aiding in each other's reconstruction.\n\n2. An intriguing observation made in the paper is the difference in sparsity between cross-attention mechanisms when trained with *contiguous masks* versus *intermittent masks.* \n\n3. The author provides a comprehensive comparison, including many relevant baselines.", "weaknesses": "1. The rationale for preferring a contiguous mask over an intermittent mask is presented but could be articulated with greater clarity to enhance its persuasiveness. Additionally, there seems to be some confusion regarding Figure 1. Clarification is needed as to whether the author implies that a) the contiguous mask is utilized during the training of REBAR, and b) the intermittent mask is employed when applying REBAR in contrastive learning. If this is the case, the reasons for using different masks in these contexts should be explicitly stated.\n\n2. The experimental scale appears somewhat limited. The paper does not specify the exact number of samples within the datasets, which seem to be on the smaller side. This limitation is accentuated when compared to previous works, such as TS2VEC [1], which utilized a much larger array of datasets, including 125 from the UCR archive and 29 from the UEA archive.\n\n3. The explanation of results requires expansion. For instance, the acronyms ARI and NMI in Table 2 are not defined within the context of the paper, leaving their significance unclear. Moreover, there is a notable difference in the results reported for the TNC on the HAR dataset between the original TNC paper [2] and this manuscript. In the origianl paper, it was reported AUPRC 0.94, Accuracy 88 while in this manuscript, it is reported AUPRC 0.98 and Accuracy 94. More information about the potential factors leading to these discrepancies would be beneficial for the reader's comprehension.\n\n\n* [1] Yue, Zhihan, et al. \"Ts2vec: Towards universal representation of time series.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 36. No. 8. 2022.\n\n* [2] Tonekaboni, Sana, Danny Eytan, and Anna Goldenberg. \"Unsupervised representation learning for time series with temporal neighborhood coding.\" arXiv preprint arXiv:2106.00750 (2021).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P02y0JRdJG", "reviewer": "Reviewer_mFnV", "summary": "This article proposes a new perspective for determining positive and negative samples in time series contrastive learning. If one subsequence can be successfully reconstructed by retrieving information from another, it should form a positive pair. Based on this, the author trains a cross-attention module to reconstruct the masked query input subsequence from the key input subsequence. The subsequence with the lowest reconstruction error is labelled as positive, and the others are labelled as negative. Experiments show that the REBAR method of this article achieves state-of-the-art results in learning a class-discriminative embedding space.", "questions": "I noticed that when applying the REBAR metric in contrastive learning, an anchor sequence and n candidate sequences are sampled randomly. Only the candidate sequence with the smallest reconstruction loss will be determined as a positive sample of the anchor sequence. Is there a situation where, for example, when sequence A is used as an anchor sequence, the candidate sequence with the smallest reconstruction loss is sequence B, and then A and B are mutually positive samples? However, when B is used as an anchor, the candidate sequence with the smallest reconstruction loss may be another sequence C. Therefore, B and C are positive samples. But if A is also in the candidate sequences, this method will divide A into negative samples of B. This leads to conflicting conclusions when the anchor sequence is different.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The method proposed in this article is intuitive and easy to understand.", "weaknesses": "1. Format issue: All formulas in this article are not numbered. The notations in the first formula on page 5 have no corresponding definition.\n\n2. The experimental volume of this paper is insufficient. As a new perspective in the field of time series contrastive learning, the author should validate the method on a wider dataset to demonstrate its universality for time series.\n\n3. The article lacks discussion and analysis of key parameters. Specifically, when training the cross-attention module, what impact will the length of subsequences and the proportion of random masking have on the reconstruction effect of key subsequences? Is the reconstruction effect of the cross-attention module directly related to downstream task performance? How to set the number of candidate subsequences (proportion of positive and negative samples) when obtaining Pos/Neg labels?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aT6h4JKWIk", "reviewer": "Reviewer_Hx4L", "summary": "This paper proposes a time-series contrastive learning framework that replaces the data augmentation module with a retrieval-based pair construction strategy. The idea sounds interesting and is proved to be effective on three time-series datasets.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work proposes a retrieval-based mask reconstruction strategy to help the model identify similar time series, which I think is a smart design.\n2. The authors show that using contiguous and intermittent masks during the training and evaluation respectively leads to the best performance. Such a result could bring some new insights to the time-series learning community.\n3. By constructing contrastive pairs retrieval, the proposed method does not rely on data augmentations, which could harm the pattern of signals, to perform contrastive learning. Experiments on three datasets demonstrate the effectiveness of the proposed method.", "weaknesses": "1. Figure 4 shows that the diagonal pattern is worse on the PPG and ECG data compared with that on the HAR data. Some explanations need to be provided here to help readers understand the potential limitations of the method.\n2. During the contrastive learning stage, the positive counterpart is selected as the one most similar to the anchor. However, it is possible that there is more than one candidate that shares the same class label with the anchor. Would such false negative pairs influence the performance of contrastive learning? Have you tried other positive selecting strategies such as hard threshold?\n3. Typo: we uniformly at random sample -> we uniformly random sample", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)