Contrast Everything: A Hierarchical Contrastive Framework for Medical Time-Series
Authors: Yihe Wang
Yu Han
Haishuai Wang
Xiang Zhang
Venue: NeurIPS 2023 poster (2023)
Keywords: Deep Learning
Contrastive Learning
Self-supervised Learning
Time Series
Healthcare
TL;DR: A novel self-supervised contrastive representation learning framework that harnesses four levels of data consistency (observation, sample, trial, and patient) to enhance medical time series analysis.

Abstract: Contrastive representation learning is crucial in medical time series analysis as it alleviates dependency on labor-intensive, domain-specific, and scarce expert annotations. However, existing contrastive learning methods primarily focus on one single data level, which fails to fully exploit the intricate nature of medical time series. To address this issue, we present COMET, an innovative hierarchical framework that leverages data consistencies at all inherent levels in medical time series. Our meticulously designed model systematically captures data consistency from four potential levels: observation, sample, trial, and patient levels. By developing contrastive loss at multiple levels, we can learn effective representations that preserve comprehensive data consistency, maximizing information utilization in a self-supervised manner. We conduct experiments in the challenging patient-independent setting. We compare COMET against six baselines using three diverse datasets, which include ECG signals for myocardial infarction and EEG signals for Alzheimer’s and Parkinson’s diseases. The results demonstrate that COMET consistently outperforms all baselines, particularly in setup with 10% and 1% labeled data fractions across all datasets. These results underscore the significant impact of our framework in advancing contrastive representation learning techniques for medical time series. The source code is available at https://github.com/DL4mHealth/COMET.

Contrast Everything: A Hierarchical Contrastive Framework for Medical Time-Series
Yihe Wang
Yu Han'
University of North Carolina Charlotte [rangl45@unce. edu
University of Chinese Academy of Sciences hanyu21 (aauul2mmmlll.mmmm ucas ac.c. on
Haishuai Wang
Xiang Zhang
Zhejiang University
University of North Carolina Charlotte xiang. zhang edu
haishuai. waaishuai.wnggee z ju. edu. cn
Abstract
Contrastive representation learning is crucial in medical time series analysis as it alleviates dependency nabor-intensive, domain-specific, and scarce expert annota- tions. However, existing contrastive learning methods primarily focus on one single data level, which fails fully exploit the intricate nature of medical time series. To address this issue, we present CÓMET, an innovative hierarchical framework that leverages data consistencies at all inherent levels in medical time series. Our meticu- lously designed model systematically captures data consistency from four potential levels: observation, sample, trial, and patient levels. By developing contrastive loss at multiple levels, we can learn effective representations that preserve compre- hensive data consistency, maximizing information utilization in self-supervised manner. We conduct experiments in the challenging matient-independent setting. We compare COMET against six baselines using three diverse datasets, which include ECG signals for myocardial infarction and EEG signals for Alzheimer's 's and Parkinson's diseases. The results demonstrate that COMET consistently out- performs all baselines, particularly in setup with 10% and 1 % labeled data fractions across all datasets. These results underscore the significant impact of our frame- work in advancing contrastive representation learning techniques for medical time series. The source code is available https://gttub..co//D. Aww.//DDDAAAAeee//////MEEE
Introduction
Time series data is crucial in various real-world applications, ranging from finance [1, 2, 3], engineer- ing [4, 5], to healthcare [6. 7]. Unlike domains such as computer vision [8, 9] and natural language processing [10, 11], where human recognizable features exist, time series data often lacks readily discernible patterns, making data labeling challenging. Consequently, the scarcity of labeled data poses significant hurdle in effectively utilizing time series for analysis and classification tasks.
To address the paucity of labeled data in time series analysis, supervupe contrastive learning has emerged promising approach. For example, TimeCL. [12] proposes DTW data augmentation for time series data; TS2vec [13] designs cropping and masking mechanism form positive pairs; ExpCLR [14] introduces a novel loss function to utilize continuous expert features. By leveraging the inherent consistency within unlabeled data, contrastive learning algorithms enable the extraction of effective representations without relying explicit labels. This paradigm shift opens up possibilities for overcoming the data scarcity issue and enhancing the capabilities of time series analysis.
"These authors contributed equally to this work.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Despite recent advancements in contrastive learning methods for time series, existing approaches fail to exploit the full potential of medical time series data, such as electroencephallgramm (EEG) signals. Unlike conventional time series data, medical time series often exhibit more data levels (Figure 1), including patient, trial, sample, and observation levels. Current contrastive learning techniques exclusively employ subsets of these levels (as illustrated in Table 1). Additionally, many of these methods are tailored to specific data types, which restricts their capacity to capture the rich complexity of medical time series. For example, CLOCS [15] presents a contrastive learning method for ECG using sample and patient levels. Mixing-up [16] captures sample-level consistency through a mixing data augmentation scheme. TNC [17] exploits trial-level consistency by contrasting neighbor samples in the same trial as positive pairs. Neither of them leverages all the levels exhibited in the
medical time series.
After reviewing existing contrastive learning methods within the time series domain, we consistently posed a pivotal question to ourselves: Can we design a straightforward yet appliable contrastive learning framework that can be adapted to all forms of medical time series data, akin to the classical model SimCLR [18] in the domain of contrastive learning? Our objective is to craft innovative framework that utilizes all information within medical time series in the context self-supervised contrastive learning. It enables us to harness patient and trial information to learn consistency across instances while leveraging the sample and observation levels' information to facilitate conventional instance discrimination.
In this paper, we propose hierarchical frame- work, COMET, that systematically leverages all four levels of medical time series, namely patient, trial, sample, and observation, to re- duce the reliance on labeled data. By incor- porating self-supervised contrastive learning, our method aims to bridge the gap between the limited availability of labeled data and the need for robust and generalizable models in medical time series analysis. We conduct extensive experiments with six baselines on three diverse datasets in challenging patient- independent setting. COMET outperforms SOTAs by 14% and 13% F1 score with la- bel fractions of 10% and 1 %, respectively, on EEG-based Alzheimer's detection. Further, COMET outperforms SOTAs by 0.17% and 2.66% Fl score with label fractions of 10% and 1 1%, respectively, on detecting Myocardial infarction with ECG. Finally, COMET outperforms SOTAs by 2% and 8% Fl score with label fractions of 10% and 1,, respectively, in the EEG-based diagnosis of Parkinson's 's disease. The results of downstream tasks demonstrate the effectiveness and stability of our method.
Trial
Patient Pt
w
www.mmmm
wwwmmmmmmmm
Sample 2;
www.mmmmmmm
Observation Fi,
Structure of medical time series. Medical time series commonly have four levels (coarse to fine): patient, trial, sample, and observation. An observation is a single value in univariate time series and vector in multivariate time series.
Figure 1:
2 Related Work
Medical time series. Medical time series [19, 20, 21, 22] is distinct type of time series data used for healthcare (e.g., disease diagnosis, monitoring, and rehabilitation). It can be collected in a low-cost, non-invasive manner [23] or a high-cost, invasive manner [24).
Unlike general time series, which typically consist of sample and observation levels, med- ical time series introduces two additional data levels: patient and trial. These extra levels of data information in medical time series enable the development of specialized methods tai- lored to address the unique characteristics and requirements of medical time series analysis. Various types of medical time series, including EEG [28, 29, 30J, ECG [31, 32), EMG [33], and EOG [34], offer valuable insights into spe-
Table 1: Existing methods only utilize partial levels.
Models
Patieut
Trial
Sample
Observation
SimCLR (25)
TF-C [26]
Mixing-up [16]
TNC [17]
TS2vec [13]
TS-TCC [27]
CLOCS [15]
COMET(Ours)
cific medical conditions and play crucial role in advancing healthcare practices.

---

Contrastive learning for time series. Contrastive learning has demonstrated its ability to learn effective representations in various domains, including image processing [35, 18, 36], graph analysis [37, 38, 37]. and time series [39]. The key idea behind contrastive representation learning is to mine data consistency by bringing similar data closer together and pushing dissimilar data further apart.
Many existing works on contrastive learning focus on general time series, while some are designed specifically for medical data [40]. One classic framework, SimCLR, transforms single sample into two augmented views and performs contrastive learning [25]. Other settings, such as using sub-series leverage sample-level consisteney [41]. TF-C [26] contrasts representations learned from the time and frequency domains to exploit sample-level consistency. Mixing-up [16] learn sample-level consistency by utilizing a mixing component as a data augmentation scheme.
and werrappinggsserie
TS-TCC and TS2vec [27, 13] apply data augmentation at the sample-level and perform contrastive learning at the observation-level. TNC [17] learns trial-level consistency by contrasting neighboring and non-neighboring samples. NCL [42] can also be used to learn trial-level consistency if we define samples from a trial as a neighborhood. CLOCS [15] learns patient-level consistency in cardiac signal features by contrasting different leads over time.
Certain prior methods have implicitly utilized hierarchical structure [13, 15]. However, shown in Table 1, none of these methods leverage all the levels present in the medical time series, potentially resulting in the loss of useful information during training. We explicitly present a hierarchical framework in the context of contrastive learning, which can be applied across diverse types of medical time series data. In our paper, we aim to leverage data consistency all levels in medical time series. Our work plays a role in summarizing, inspiring, and guiding future works in self-supervised contrastive learning on medical time series.
3 Preliminaries and Problem Formulation
3.1 Medical Time Series
In this section, we clarify the key conceptions of observation (or measurement), sample (or segment), trial (or recording), and patient (or subject) in the context of medical time series (Figure 1). For better understanding, we illustrate the concepts with an example of Electroencephalograpyy (EEG) signals for Alzheimer's Disease diagnosis (details in Appendix A).
Definition 1: Observation, An observation Ii, E R in medical time series data represents a single data point or a vector captured at a specific timestamp t. Here we use to denote the sample index (see Definition 2) and to denote the timestamp. It may record physiological status, laboratory test results, vital signs, or other measurable health indicators. The observation is a single real value for univariate time series while vector for multivariate time series. The F is the feature dimension if it is multivariate time series.
} is a sequence of consecutive observations, typically measured at regular intervals over a specified period (T' tmmestampss. It can also be called segment or window. Here we use to denote the sample index. In the medical time series, a sample might consist of sequence of heart rate measurements or blood pressure readings.
= 1.... .T. Definition 2: Sample. A sample x, = =
Definition 3: Trial. A trial r; is a collection of consecutive samples. It can also be called a record. Here we use to denote the trial ID. In medical time series, trial is continuous set of observations collected over a not-short period (e.g., 30 minutes). Therefore, a trial is generally too long (e.g., hundreds of thousands of observations) to feed into deep learning models for representation learning directly and is usually split into shorter subsequences (i.e., samples/segments). To represent the aggregate of samples stemming from particular trial r; with trial ID i, we employ the notation R,,
Definition 4: Patient. A patient p; represents a collection of multiple trials stemming from a single patient. It can also be called a subject. Here we use i to denote the patient ID. It is important to note that trials for a given patient may exhibit variations due to differing data collection timeframes. sensor placements, patient conditions, and other contributing factors. As shown in Definition 3, trial is typically divided into many samples for better representation learning. In practical scenarios, a patient, which constitutes a cluster of trials, is also divided into samples that may share identical or distinct trial IDs but maintain the same patient IDD To represent the aggregate of samples stemming from particular patient P, with the corresponding patient ID i, we employ the notation P..
3

---

Medical Time Series Dataset
Patieni-Level Contrastive Block
Encader
Embedding Space
pS
A
A
Lp
n Patient
Patlent
wmw
www
wMw
www
Petient
NEV VMM
mm
A
Pi
Triel-Level Contraetive Block
www Wm
M
www
x²
h
h
www.
w
C
Semple-Level Contrestive Block
by h2
by
R
Pattent-Laevel
-i008 8i
As
wmw
wmw
Tria--Levll
SampleeLLeel
Observalio-Level
Observation-Leval Contrastive Block
Z1. Mww
Biz R. R
Co
wMw
Nagative Pair
Figure 2: Overview of COMET approach. Our COMET model consists of four contrastive blocks, cach illustrating the formulation of positive pairs and negative pairs different data levels. In the observation-loe contrastive, observation i and its augmented view Fi contrastive, sample æ; and its augmented view 2; form a positive pair. Moving to the trial-level contrastive, two samples a and x: from the same trial i. are considered to be positive pair. The patient-level contrastive follows similar pattern, where two samples and x from the same patient p. are regarded as a positive pair. Positive and corresponding negative pairs will be utilized to build contrastive loss in embedding space after being processed by encoder G.
positive pair. Similarly, in the sample-level
In this work, we propose a novel hierarchical contrastive framework to learn representative and generalizable embeddings by comprehensively exploring instance discrimination at observation and sample levels and harnessing consistency across instances at trial and patient levels. Although we elaborate the proposed COMET in the context of medical time series, we note our model can possibly be extended to other time series beyond healthcare as long as extra information is available. For example, climate dataset contains multiple metcorological satellites, each satellite contains multiple measuring units, and each unit contains multiple sensors, and every sensor can measure a specific observation at certain timestamp. The key is to utilize all available information, excluding label data, for contrastive pre-training, such as patient ID. To adapt our approach to other domains, researchers must consider crucial question: Does the dataset have additional information beyond sample labels? If affirmative, can this information be harnessed for contrastive learning? The example of satellite sensor application underscores the potential existence of supplementary information even in non-medical domains.
3.2 Problem Formulation
Problem (Self-Supervised Representation Learning For Medical Time Series). Let unlabeled dataset D consist of set of patients, where each patient P; has multiple trials, each trial r, can be segmented into many samples, and each sample comprises series of observations. We aim to pre-train an encoder G that exploits data consistency all available levels in a self-supervised contrastive manner. For given time series sumple æ; E &RTXF with T timestamps and F feuture dimensions. the encoder learns a sample-level representation h; E TxXK where h IR. is the observation-leve. representation at timestamp with K dimensions.
By exploiting hierarchical consistency at multiple data levels, aim to learn a representation h; that is both representative (yielding good performance in downstream tasks) and generalizable (maintaining stability across different patients). Depending the fine-tuning settings [18], specific fraction of labels corresponding to samples æ; are necessary.
4 Method
In this section, we first present our assumption of data consistency behind designing hierarchical contrastive framework. Then, we describe the architecture of the proposed model COMET (Figure 2).

---

4.1 Hierarchical Data Consistency
Capturing data consistency is crucial in the development of contrastive learning framework [1.3]. Data consistency refers to the shared commonalities preserved within the data, which provide a supervisory signal guide model optimization. Contrastive learning captures data consistency by contrasting positive and negative data pairs, where positive pairs share commonalities and negative pairs do not. We propose consistency across four data levels: observation, sample, trial, and patient, from fine-grained to pparse-grained in the medical time series. Although we present four levels here, our model can easily be adapted to accommodate specific datasets by adding or removing data levels.
Observation-leww data consistency. We assume a slightly augmented observation (e.g., channel masked) will carry similar information the original observation [13]. We use a the anchor observation at timestamp i, and Ty. as the observation at another timestamp t- in the sample Zz- We consider the anchor observation æ;,; and augmented observation T. positive pairs -ei,::...l (with closer embeddings). Conversely, we consider the original observation Ei, and the observations -i,-- and - another timestamp t negative pairs (2:1,, (.,,-), (20i, Ti,,)), with distant embeddings.
Sample-level data consistency. The sample-lveel consistency is based slightly perturbed sample (c.g., temporally masked) should carry similar information as the original sample [18, 16, 26]. We consider the anchor sample æ; and its augmented view T, as positive pair (ar, k3). We regard the anchor sample x; and a different sample 3, and its augmented view 3, as negative pairs: (ar;, 2)) and (at,, m.y)-
assumption that
Trial-level data consistency. We assume that samples sliced from the same trial should carry similar information compared to those obtained from different trials. For simplicity, we use æ to denote the anchor sample and æ= to denote a sample from the same trial ry as the anchor sample, while x to denote a sample from another trial ry. In other words, we have {x, ++} E R2 and x E Rg. We treat sample x and the sample x from the same trial as positive pair (ær, at). ). We regard sample æ and the sample
from different trials as negative pair (x,
Patient-level data consistency. We assume samples originating from the same patient are likely to contain similar information when compared to those from different patients [15]. Here, we use x to denote the anchor sample and x to denote a sample from the same patient ps. while x= from another patient Pj. In other words, there are {I, x+} P. and x E P.j. We have positive pair (æ, x+) including samples from the same patient and negative pair (æ, x that from different patients.
Disease-level data consistency. For completeness, we introduce disease-level data consistency, which suggests that samples associated with the same type of disease should exhibit shared patterns, even when collected from different patients in different ways. However, capturing disease- consistency requires ground truth labels, which are not available in a self-supervised approach. As a result, we do NOT employ disease-level consistency in this paper. Nevertheless, it can be adapted for cupesiupee or supervised contrastive learning and may prove beneficial in learning addaiaee representations for certain diseases across patients and even datasets.
A common principle underlying all definitions is that the X-level data consistency refers to the positive pair belonging to the same X, where X could be observation, sample, trial, patient, or disease. We assume that each patient is associated with only one label, such suffering from specific disease, which implies that all samples from the same patient essentially originate from the same distribution. However, in cases where data from a patient is derived from multiple distributions (c.g., a patient could perform various daily activities; associated with multiple labels), the assumptions of trial-level and patient-level consistency are not satisfied. Therefore, the user can switch on the observation-level and sample-level consistency.
Building upon the concepts of data consistency, introduce four contrastive blocks corresponding to the four data levels. Our model is highly flexible, allowing users to enable or disable any of the blocks based on the requirements of specific task or dataset.
4.2 Observation-Levd Contrastive Block
For given time series sample æs, we apply data augmentation (such as masking) to generate an augmented sample ã; [25, 43]. We input the original sample 2; and its augmented view 2; into
5

---

contrastive encoder G to obtain their respective representations h; G(æ,) and h; = - important note that we apply data augmentation to the samples. which indirectly extends to augmenting the observations, simplifying the encoding process. To capture observation-level consistency, we assume that, after being processed by encoder G, the representation of observation Fi is close to the representation of the augmented observation Zi,-- In contrast, it should be distant from the representations observations _i,t- and T originating from any other timestamp t". Specifically, our positive pair is (æi,e Fi,x) and negative pairs are (2i,,, and (ai,, Li,t Observation-lewe contrastive loss. The observation-level contrastive loss Lo [13] for the input sample z; is defined as:
exp ha,e)
Co Bx,ED BLET
log -
(I)
(e-er(ex(hha. ha,e) + pexe-jexp(hs; hig
where {1.... T is the of all timestamps sample æ, and denotes dot product. [tgtt is an indicator function that equals to 0 when t = t and 1 otherwise.
The
4.3 Sample-Level Contrastive Block
For input time series sample and its augmented view ë, we calculate their representations through h; G (a:) and h, G The augmentation applied here could be the different from the augmentation used in Section 4.2. We assume that after passing through the encoder G, the representation of the sample æ; is close to the representation of its augmented view H,, while far away from the representations of any other samples æ and ēj. In specific, our positive pair is ( H.), and negative pairs are (x,;,,) and (x;,,,) Sample-level contrastive loss. The sample-level contrastive loss Ls [18, 43] for the input sample a; is defined
exp(h: h:)
Cs = E..ED
-log E
(exp(h, h2) [xys)exp(hh; hy))
where [D] represents the total number samples in the dataset D and denotes dot product. The [()]]] is indicator function that equals 0 when i and 1 otherwise.
4.4 Triel-Level Contrastive Block
For an input sample æ E R.,, where R; is a collection of samples segmented from trial rús we feed it into the contrastive encoder G to generate a sample-level representation h = G(x).. ). To seize trial-level data consistency, we assume that the representation of the anchor sample x: E R, is close to the representation of sample x that also come from the trial rú. In contrast, the representation of the anchor sample æ is far away from the representation of sample x that come from a different trial ry, where x E Ry. In other words, we have positive pair (x, x x+) and negative pair (æ, x Teiel-level contrastive loss. The trial-level contrastive loss LR [15, 18] for the input sample æ defined as:
exp(sim(h, G(=+))/T)
I
LR E
EatER.
log Σ/ Ex ER, (exp(sisi h ((x=)))/T))
where J is the total number of trials in the dataset D. The iumuu, v) uTu v/ ||u||] |||||| denotes the cosine similarity, and 7 is temperature parameter to adjust the scale. The G(x and G(æ are learned representations of samples at E R, and E RRj, respectively. To measure the trial-level loss for sample æ, we iterate all the x in R4, and averaging across |Ra positive pairs.
In this block, we do NOT learn a trial-level embedding representing the entire trial. Instead, learn representation for each sample within the trial while considering trial-level data consistency, Similarly, we follow this protocol for the patient-level contrastive block.
4.5 Patient-Level Contrastive Block
For an input sample E Pis where P; denotes all samples from patient Pt, we feed it into the contrastive encoder G' to generate sample-level representation h = G(æ). Similar to the above

---

trial-level contrastive block, we have positive pair (x, x²) and negative pair (z, æ x>),, in which x come from the same patient while æ come from different patient. Patient-level contrastive loss. The patient-level contrastive loss Lp [15] for the input sample æ is defined
exp(sim(h, G(=+))//T)
Lp = EuED
ExteP: EP:
-log ンー= E. EP, (exp(sim(h,
(4)
where M is the total number of patients in the dataset D. In this block, the G(as+) and G(x=)
learned representations of samples x E Pi and x E Pj. respectively.
4.6 Overall Loss Function
The overall loss function L consists of four loss terms' The observation-level loss Lo and sample-level loss Ls encourage the encoder to learn robust representations that are invariant to perturbations. The trial-level loss LR and patient-level loss Lp compel the encoder to learn cross-sample features within a trial or a patient. In summary, the overall loss function of the proposed COMET model is:
L = A1Co + XgCs + X3CR + Sp
(5)
where λ1. A2; λ3, A E [0, 1] are hyper-coefficients that control the relative importance and adjust the scales of each level's loss. Users can simply turn off specific data levels by setting À of those levels to 0. We set À + d2 + Àa + 24 1. We calculate the total loss by taking the expectation of across all samples E D. In practice the contrastive losses are calculated within mini-batch.
5 Experiments
We compare the COMET model with six baselines on three datasets. Following the setup in SimCLR [18],
Training
Testing
we use unlabeled data to pre-train encoder G and evaluate two downstream settings (Appendix E): partial fine-tuning (P-FT: L.e., linear evaluation [18]
Patient- Dependent
and full fine-tuning (F-FT). All datasets are split into training, validation, and test sets in patient- independent setting (Figure 3), which is very chal- lenging due to patient variations (More explanations in Appendix F.1).
P
P2
Patient- Independent
Datasets. (1) AD [44] has 23 patients, 663 trials, and 5967 multivariate EEG samples. There are 4329, 891, and 747 samples in training, validation, and test sets. The sampling rate (frequency) 256Hz. Each sample is a one-second interval with 256 observa- tions. binary label based on whether the patient has Alzheimer's disease is assigned to each sample. (2) PTB [45] has 198 patients, 6237 trials, and 62370 multivariate ECG samples. There are 53950, 3400,
Figure 3: Pedienttdepeeneen/nnddeeedde Set- ting. In the patient- deeeent- setting, samples from the same patient can appear in both the training and test sets. In contrast, in the patient- independent setting, samples from the same patient are exclusively included in the training or test set.
and 5020 samples training, validation, and test sets. The sampling rate (frequency) is 250Hz. Fach sample is heartbeat with 300 observations. A binary label based on whether the patient has Myocardial infarction is assigned to each sample. (3) TDBRAIN [46] has 72 patients, 624 trials, and 1 11856 multivariate EEG samples. There are 8208, 1824, and 1824 samples in training, validation, and test sets. The sampling rate (frequency) is 256Hz. Each sample a one-second interval with 256 observations. A binary label based on whether the patient has Parkinson's 's discase is assigned to each sample. See appendix D for more details about data statistics, train-test split, and data preprocessing
Baselines. We compare with 6 of of-te-of-the-art methods: TS2vec [13], Mixing-up [16], TS-TCC [27]. SimCLR [43], CLOCS [15] and TF-C [26]. Since TF-C is designed for transfer learning, we implement its downstream tasks the same as ours for fair comparison. The evaluation met- rics are accuracy, precision (macro-averagedd) recallmaacr-aaveragdd) Fl score(macro-averagedd
AUROC(macro-averagg)) and AUPRC(macro-avereel
7

---

Table 2: Partial fine-tuning results. A logistic regression(LGG) classifier L is added top of a frozen encoder
on dataset AD. Only fine-tuning the classifier
Models
Accuracy
Precision
Recall
F1 score
AUROC
AUPRC
TS2vec
26.4
67.72.1599
67.40
66.32::5.6
74.12:66.8
72.961721
TF-C
77.03=2200
75.79+507
64.27=5.0
64.85=156
80.71=4.00
79.27+4:15
Mixing-up
46.16=1..3
52.62+490
50.81+1.32
37.37
64.42=6649
62.85+6677
TS-TCC
59.71
61 616662883
60.33+4.26
18.66+8879
67.53++1.04
68. 68.33++9.37
SimCLR
57.16=200
56.67#3391
53.57=221
49.1 44.26
56.67=3.91
52.10:141
CLOCS
66.99-2244
67.17
67.33+22.9
66.91=2.88
73.71=3.62
72.58+3354
COMET (Ours)
76.09=421
77.36
74.68:4462
74.80=488
81.30=4491
80.50
Implementation. We use a ten residual blocks dilated CNN module [47] as backbones for encoders G. To preserve the magnitude feature of time series, which is often an important feature in time series, we utilize timestamp masking as proposed in [13] as an augmentation method. While many existing works apply data augmentation directly on raw time series, we use fully connected projection layer to map the raw input into an embedding with a higher feature dimension. This strategy helps prevent situations where some parts of the raw data are zero, which could result in ineffective augmentation if timestamp masking were directly applied. Additionally, to ensure stability during training, we incorporate hierarchical loss [13], which encapsulates the observation and sample-level losses. To guarantee there are samples with the same trial and patient IDs in a batch for trial and patient levels contrasting, we design two specific shuffle functions. See Appendix C for further details. For datasets whose trial level is absent or limited to one trial per patient, we provide solution in Appendix F.2.
We conduct experiments with five seeds(41-45) based on the same data split to account for variance and report the mean and standard deviation. All experiments expect baseline SimCLR run on NVIDIA RTX 4090. Baseline SimCLR runs on Google Colab NVIDIA A100. See further implementation details in Appendix E.
5.1 Results on Partial Fine-tuning
Setup. Linear evaluation is a widely used P-FT setup to evaluate the quality of representation, where linear classifier is trained on top of a frozen network for sample classification [18, 15, 48, 49]. This evaluation method allows for a quick assessment of representation quality. In linear evaluation, we add logistic regression classifier L on top of the pre-trained encoder G. The training process utilizes 100% labeled training data for sample classification. Notably, the parameters of the encoder G are frozen during training, ensuring that only the classifier L fine-tuned. See further details in E.
Results. We report the experimental results of the P-FT setup on AD in Table 2. On average, our COMET model claims large margin of more than 7% over all baselines on the Fl score AD.
5.2 Results on Full Fine-tuning
Setup. In an F-FT setup, we add a two-layer fully connected network classifier P to the pre- trained encoder G. The training process utilizes 100%, 10%, and 1% of the labeled training data for sample classification, respectively. Unlike the P-FT approach, both the encoder G and classifier P are trainable now, allowing for fine-tuning of the entire network structure. See further details in E.2.
Results. The results for F-FT on the three datasets are presented in Table 3 and Table 4. In general, COMET demonstrates success in 46 out of 54 tests conducted across the three datasets, considering six different evaluation metrics. With 100% labeled data, the Fl score of COMET outperforms the best baseline, Vec, by 2% on the AD dataset and surpasses the best baseline, Mixing-up, by 4% on the TDBrain dataset. Furthermore, it achieves a result comparable to the best baseline, TF-C, on the PTB datasct.
Notably, the Fl score of COMET surpasses the best baseline. TF-C, by 14% and 13% with label fractions of 10% and 1%, respectively, on the AD dataset. Additionally, on the TDBrain dataset, the F1 score of COMET outperforms the best baseline, Mixing-up. by 2% and 8% with label fractions of 10% and 1 1,, respectively. Similarly, the Fl score of COMET outperforms the best baseline, CLOCS, by 0.17% and 2.66% with label fractions of 10% and 1 1,, respectively, on the PTB dataset. It is interesting to observe that the Fl score of COMET with label fractions 10% and 1% outperforms label fraction of 100% on the AD and PTB datasets. This suggests potential overfitting of COMET to the training data. similar phenomenon observed with SimCLR, TF-C, CLOCS, and TS2vec, where higher fraction of labeled data did not necessarily lead to improved performance.
8

---

Table 3: Full fine-tuning results of EEG datasets. two-layer fully connected networks classifier P is trained on top of pre-trained encoder G EEG datasets AD and TDBRAIN, Fine-tuning the whole network includes G and P. We use 100%, 10%, and I % of labeled training data in AD and TDBRAIN.
Datasets Fraction
Models
Accuracy
Precision
Recall
FI score
AUROC
AUPRC
TS2vee
81.26=205
81.21
81,34a204
81.12=206
89.20=177
88.94
TF-C
=8.3
75.87
74.83a8
74.54
79,45am0s
19.33210577
Mixing-up
65.68=789
+421
68,25
63,9 63.982992
84.63=504
83.46:546
TS-TCC
73.55-1000
77.22+6.13
73.83+90s
71.6
86.17=5.1
85.73=5..1
SimCLR
54.77=197
50.15
50.58
43.184427
50.154708
50.42::106
CLOCS
78.37=6m
83.99
76, 76
75.78 78.78
91.17=2.11
90.72:110
COMET (Ours)
84.50=446
88,31
82,95
83.33
94.44=2.37
94,43=2248
TS2vec
73.28
74. 14:14:1:41
73.52
73.00
81.6615.0
TF-C
15.66=11.1
75.48
75.58=11.5
15.38
11.38+14.9
81.56:13.68
Mixing-up
59.38=333
64.85
61,944342
58. 18.17:3341
75.02=6.14
73.44=5.22
AD
10%
TS-TCC
77.83=690
79.73:4749
78.18
76.43
84.1227.33
84.12.7M
SimCLR
56.09=225
53.81
51.73+2.9
44.10:484
53.81
51.08
CLOCS
76.97=3m1
81.70
74.69
74.75=361
86.91
86.70=3:4
COMET (Ours)
91.43=3312
92.52.12.6
90.71
91.14
96.44:224
96.48.182
TS2vec
64.93=381
65.28
65.14
64 54
70.5613.8
68.97
TF-C
75.66=12551
13.26=1391
74.77
74.33
18.96+1617
11.89
Mixing-up
63.67=129
65,02
64.64
63.5 13.55
71.95=339
70.15
1%
TS-TCC
53.04_880
12.39
52,00akm
42.69
18.89
41.41
SimCLR
55,42=241
52.18
51.37
45.02:479
52.18=535
50.87:1445
CLOCS
64,50=415
65.17
63.73
63.0 63.01=511
69. 69.665
68.15
COMET (Ours)
88.22::336
88,55
88.561314
88.14
96.05:1.16
96.12:1131
TS2ver
80.21
81,38
80.21
80.07
89.57.2.31
27.37
TF-C
66.62=175
67.15+1644
66.62
66.35=191
65.43
66. 68.18:490
Mixing-up
81.47=107
82. 12.1221112
81
81.38
90.48:0..9
90.51
100%
TS-TCC
77.42-226
67.68
77.42&2
27.37
87.37
23.22
SimCLR
58.43=177
59.48
58.432177
57.30=207
19.48
55.05
CLOCS
78. 1.13
79.49
78. 11:
77.91
88.49
88.83=1.44
COMET (Ours)
85.47=1116
85.68
85.4
85.45
93.73:144
93.96.:9.9
TS2vec
72.39=1.13
74.49
1..3
7.80==105
80.71=1.00
80.06
TF-C
59.14=304
59.34
59,14 19.14433.44
58. 98:294
59.56:4.0
59.65=2.99
Mixing-up
77.50=20t
80.09
77.50
76.992226
87.2921:1
87. 1:
TDBrain
10%
TS-TCC
71.23-1.57
78.78+0.66
71.23
69.18 69-18:2225
80.5621.9
80.21
SimCLR
59.79=209
60.50
59,79
59.06=282
60.50=190
55.96=141
CLOCS
75.04=205
75.97 6:3.
75.04
24283
84.25
84.37
COMET (Ours)
79.28::444
89.83
79,28
79.19:442
88.39:4.13
38.38:33.6
TS2vec
58. 16=230
58.47
58.164230
57.83
61.26=2.51
60.95::2488
TF-C
53,16=211
53.19:2210
53.16#211
52.99
$2.11
59. 11.8
Mixing-up
63.91=2.59
64.29
63,91
63.70
69.61
68.89
1%
TS-TCC
49.14=138
$1.10#8.54
49.14
40.82:4417
48.05
50.17:3344
SimCLR
59.46=1779
59.77
59,46+1179
59.16=181
59.77
55.69
CLOCS
58.85=aMM
59.03-a..00
58.85
58. 25.38
62.57.20.8
62. 14
COMET (Ours)
72.93-721
74.20
72.93:721
72.57=937
78.72:8.4
77.71:9.00
As a result, COMET demonstrates its superiority and stability across all the datasets. Furthermore, COMET outperforms SOTAs methods significantly with 10% and 1% label fractions, highlighting the effectiveness of our contrastive pre-training approach in reducing the reliance on labeled data.
5.3 Ablation Study, Visualization, Additional Downstream Tasks, and Heavy Duty Baseline
Ablation study. We verify the effectiveness of each contrastive block and other COMET variants. Besides, we study the impact of hyperparameter X. (Appendix G.1)
Visualization. We visualize the embedding space and show our model can learn more distinctive and robust representatiinn (Appendix G.2)
Additional downstream tasks. Apart from the classification tasks in Section 5.1-5.2, we show the proposed COMET outperforms basclines in wide range of downstream tasks, including clustering and anomaly detection. (Appendix G.3)
Heavy duty baseline. We show the superiority of our model does NOT result from the newly added contrastive blocks (i.e., increased model parameters). COMET outperforms the heavy-duty SimCLR and Ve2 with four contrastive blocks and four times pre-training epochs. (Appendix G.4)

---

Table 4: Full fine-tuning results of ECG datasets. two-layer fully connected networks classifier P is trained on top of pre-trained encoder on ECG dataset PTB. Fine-tuning the whole network includes and P. use 100%. 10%, and 1% of labeled training data in PTB.
Datasets Fraction
Models
Accuracy
Precision
Recall
F1 score
AUROC
AUPRC
TS2vec
1..14
87 27.82
16.84
79.6613.6
90.50
90.07
37.50+245
50=500
82.68
83.77
19.59
80.62=15.10
Mixing-up
87.61
89.56=220
79.30=1.67
82.56=2.0
89.29=1.26
88.94=100
100 %
TS-TCC
82.24
28
69.46=585
26.08
87.60=255
86, 86.26=330
SimCLR
84.19=1.32
85, 1
73.89=2.9
76.84=2250
85.85 85.85
70. 20
CLOCS
36.39+2.76
87.06+2281
77.95
80.71 44
90.41=2.00
87. 97.35-+3.0
COMET (Ours)
1.984
87.67=17
81.14=3.68
9..95=1.56
87.47=281
TS2vee
82.491471
80.39 15
1..91
80.18
93.03
92.81.1097
TF-C
85.37
82.8 a2235
79.94.1071
81.09
81.57
84.57
Mixing-up
87.05=141
86,56#324
80.61=2.668
82.62+1.99
89,28=1.43
87.22=276
PTB
10%
TS-TCC
83,38
85. 1::11
72.24
75.27 1:264
86.06
84, 84.34.220
SimCLR
83.84
87. 1.344
45.44
87.19
69 69,99433.4
CLOCS
88.25#248
88, 18.64-2211
81,40=4.64
91.11=240
89, 89.76=3394
COMET (Ours)
88,49
88. 28.9
81,65
84.01 5
94.83=1.0
92.48+2201
TS2vee
81
78. 98
83.83
19.77
90.99
88. 88.69:2144
TF-C
85.82.1234
82, $2.79.. 1:320
81.86.12.6
62.211.28
89.14.2289
89,47
Mixing-up
84.71:4.11
82, k.00
78.17
79.31
89.04
84. 34.67.1111
1%
TS-TCC
18.61
28.27:1129
64.56:22.53
66.2333316
19.97
76.98
SimCLR
84.1911.49
87.15
73.21
13.31
87.15
70.58
CLOCS
88, 88.80
88. 14.37
84.47
85.57
94.73
94,04
COMET (Ours)
90,52:1150
88.58
88.23
88.23:2.30
95.08:1.50
93.78
Conclusion
This paper introduces COMET, a hierarchical contrastive representation learning framework tailored for medical time series. COMET leverages all data levels of medical time series, including patient, trial, sample, and observation levels, to capture the intricate complexities of the data. Through extensive experiments on three diverse datasets, we demonstrate that our method surpasses existing state-of-the-arr methods in medical time series analysis. Our framework also shows its effectiveness in inneeennddddedeeent downstream tasks, highlighting its potential to advance medical time series analysis and improve patient care and diagnosis.
One limitation of our work the presence label conflicts between different levels. patient-level consistency, assume all samples belonging the same patient positive while others negative. In trial-level consistency, we consider samples from the same trial as positive samples and others negative. This means positive pair at the patient level may be considered negative the trial level, as they do not belong to the same trial.
In future research, aim to investigate the efficacy of our method across wider range of datasets, with particular focus on ECG datasets. Additionally, we intend to explore approaches to integrate disease-level consistency within our sulf-supervised contrastive framework.
We discuss the broader impacts with potential negative social impacts in Appendix H.
Acknowledgments and Disclosure of Funding
This work is partially supported by the National Science Foundation under Grant No. 2245894. Any opinions, findings, conclusions or recommendations expressed in this material are those the authors and do not necessarily reflect the views of the funders.
10

---

References
[1] Dawei Cheng, Fangzhou Yang, Sheng Xiang, and Jin Liu. Financial time series forecasting with multi-modality graph neural network. Pattern Recognition, 121:108218, 2022.
[2] Yajiao Tang, Zhenyu Song, Yulin Zhu, Huaiyu Yuan, Maozhang Hou, Junkai Ji, Cheng Tang, and Jianqiang Li. survey on machine learning models for financial time series forecasting. Neurocomputing, 51 512:363-380, 2022.
[3] Xiaoyu He. Suixiang Shi, Xiulin Geng, and Lingyu Xu. Information-aware attention dynamic synergetic network for multivariate time series long-term forecasting, Neurocomputing, 500:143- 154, 2022.
[4] Zhiwen Chen, Jiamin Xu, Tao Peng, and Chunhua Yang. Graph convolutional network-based method for fault diagnosis using hybrid of measurement and prior knowledge. IEEE transactions on cybemetics, 52(9):9157 9169, 2021.
[5] Quan Qian, Yi Qin, Jun Luo, and Dengyu Xiao. Cross-machine transfer fault diagnosis by ensemble weighting subdomain adaptation network. IEEE Transactions on Industrial Electronics, 2023.
[6] Catherine Arsenault, Anna Gage. Min Kyung Kim, Neena Kapoor, Patricia Akweongo, Freddie Amponsah, Amit Aryal, Daisuke Asai, John Koku Awoonor- Awoonor-Williams, Wondimu Ayele, et al. Covid-19 and resilience of healthcare systems in ten countries. Nature Medicine, 1316) 1324, 2022.
[7] Andrea Brizzi, Charles Whittaker, Luciana MS Servo, Iwona Hawryluk, Carlos Prete Jr, William M de Souza, Renato S Aguiar, Leonardo JT Araujo, Leonardo S Bastos, Alexandra Blenkinsop, et al. Spatial and temporal fluctuations in covid-19 fatality rates in brazilian hospitals. Nature medicine, 28(7):1476-144 1485, 2022.
[8] Danyang Tu, Xiongkuo Min, Huiyu Duan, Guodong Guo, Guangtao Zhai, and Wei Shen. End-to- end nauman-gaze-taree detection with transformers. 2022 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2192-2200. IEEE, 2022.
[9] Jaime Spencer, Richard Bowden, and Simon Hadfield. Medusa: Universal feature learning via attentional multitasking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18000-809, 2022.
[10] Haris Bin Zia, Ignacio Castro, Arkaitz Zubiaga, and Gareth Tyson. Improving zero-shot cross- lingual hate speech detection with pseudo-label fine-tuning of transformer language models. In Proceedings of the International AAAI Conference Web and Social Media, volume 16, pages 1435-1439, 2022.
[11] Tuan Dinh, Yuchen Zeng, Ruisu Zhang, Ziqian Lin, Michael Gira, Shashank Rajput, Jy-yong Sohn, Dimitris Papailiopoulos, and Kangwook Lee, Lift: Language-interface fine-tuning for non-language machine learning tasks. Advances in Neural Information Processing Systems, 35:11763-11784 2022.
[12] Xinyu Yang, Zhenguo Zhang, and Rongyi Cui. Timeclr: A self-supervised contrastive learning framework for univariate time series representation. Knowledge-Based Systems, 245: 108606, 2022.
[13] Zhihan Yue, Yujing Wang. Juanyong Duan, Tianmeng Yang. Congrui Huang, Yunhai Tong, and Bixiong Xu. Ts2vec: Towards universal representation of time series. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 1980-8987, 2022.
[14] Manuel T Nonnemmacher, Lukas Oldenburg, Ingo Steinwart, and David Reeb. Utilizing expert features for contrastive learning of time-series representations, In International Conference on Machine Learning, pages 16969-16989, PMLR, 2022.
[15] Dani Kiyasseh, Tingting Zhu, and David A Clifton. Clocs: Contrastive learning of cardiac signals across space, time, and patients. International Conference on Machine Learning, pages 5606-5615. PMLR, 2021.
[16] Kristoffer Wiekstræm, Michael Kampffmeyer, Karl Øyvind Mikalsen, and Robert Jenssen. Mixing up contrastive learning: Self-supervised representation learning for time series. Pattern Recognition Letters, 155:54-61, 2022.
11

---

[17] Sana Tonekaboni, Danny Eytan, and Anna Goldenberg. Unsupervised representaiinnnn learning for time series with temporal neighborhood coding. ICLR, 2021.
[18] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple frame- work for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[19] Minji Lce, Leandro RD Sanz, Alice Barra, Audrey Wolff, Jaakko O Nieminen, Melanie Boly, Mario Rosanova, Silvia Casarotto, Olivier Bodart, Jitka Annen, et al. Quantifying arousal and awareness altered states of consciousness using interpretable deep learning. Nature communications, 13(1):1064, 2022.
[20] Chenxi Sun, Hongyan Li, Moxian Song, and Shenda Hong. A ranking- cross-entropy loss for early classification of time series. IEEE Transactions on Neural Networks and Learning Systems, 2023.
[21] Huizi Cui, Lingge Zhou, Yan Li, and Bingyi Kang. Belief entropy--o-enntrpy and its application in the cardiac interbeat interval time series analysis. Chaos, Solitons & Fractals, 155:1 11736, 2022.
[22] Zongxing Xie, Bing Zhou, Xi Cheng, Elinor Schoenfeld, and Fan Ye. Passive and context-aware in-home vital signs monitoring using co-located uwb-depth sensor fusion. ACM Transactions on Computing for Healthcare, 3-4):1-31, 2022.
[23] Martin Krause, Shannon K McWilliams, Kenneth J Bullard, Lena M Mayes, Leslie C. Jameson, Susan K Mikulich-Giilbertoon, Ana Fernandez-Bustamantte and Karsten Bartels. Neostigmine versus sugammadex for reversal of neuromuscular blockade and effects on re-intubation for respiratory failure or newly initiated non-invasive ventilation-an interrupted time series design. Anesthesia and analgesia, (31 2020.
[24] Mohammad-Pant Hosseini, Tuyen X Tran, Dario Pompili, Kost Elisevich, and Hamid Soltanian- Zadeh. Multimodal data analysis of epileptic eeg and rs-fmri via deep learning and edge computing. Artificial Intelligence in Medicine, 104:101813, 2020.
[25] Johannes Pöppelhaum, Gavneet Singh Chadha, and Andreas Schwung. Contrastive learning based self-supervised time-series analysis. Applied Soft Computing, 10::97 2022.
[26] Xiang Zhang, Ziyuan Zhao, Thcodoros Tsiligkaridis, and Marinka Zitnik. Self-supervised contrastive pre-training for time series via time-firy consistency. In NeurIPS, 2022.
[27] Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee Keong Kwoh, Xiaoli Li, and Cuntai Guan. Time-series representation learning via temporal and contextual contrasting. In [ICAI, pages 2352-2359, 2021.
[28] Siyi Tang, Jared Dunnmon, Khaled Kamal Saab, Xuan Zhang, Qianying Huang, Florian Dubost, Daniel Rubin, and Christopher Lee-Messer. Self-supervised graph neural networks for improved alettrenneephalogaapicc seizure analysis. In International Conference on Learning Representations, 2021.
[29] Chaoqi Yang. M Brandon Westover, and Jimeng Sun. Manydg: Many-domain generalization for healthcare applications. In The Eleventh International Conference Learning Representations, 2023.
[30] Xiaodong Qu, Zepeng Hu, Zhaonan Li, and Timothy Hickey. Ensemble methods and Istm outperformed other eight machine learning classifiers in an eeg-based bei experiment. In International Conference on Learning Representations, 2020.
[31] Uri Shaham, Jonathan Svirsky, Ori Katz, and Ronen Talmon. Discovery of single independent latent variable. Advances in Neural Information Processing Systems. 35:25251-25266 2022.
[32] Xu Chen and Brett Wujek. A unified framework for automatic distributed active learning. IEEE Transactions on Pattern Analysis & Machine Intelligence, 44(1 (111:97744-7786 2022.
[33] Yuanchao Dai, Jing Wu, Yuanzhao Fan, Jin Wang, Jianwei Niu, Fei Gu, and Shigen Shen. Mseva:
A musculoskcletal rehabilitation evaluation system based on emg signals. ACM Transactions on Sensor Networks, 19(1): 2022.
[34] Yingying Jiao, Yini Deng, Yun Luo, and Bao-Liiag Lu. Driver sleepiness detection from eeg and cog signals using gan and lstm networks. Neurocomputing, 408:100-11 2020.
12

---

[35] Jean-Bate Bean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya. Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-212844 2020.
[36] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1tt.::1807.03370 2018.
[37] Chunhui Zhang, Hongfu Liu, Jundong Li, Yanfang Ye, and Chuxu Zhang. Contrastive graph few-shot learning. In NeurlPS 2022 Workshop: New Frontiers in Graph Learning, 2022.
[38] Zekun Tong, Yuxuan Liang. Henghui Ding, Yongxing Dai, Xinke Li, and Changhu Wang. Directed graph contrastive learning. Advances Neural Information Processing Systems,
34:19580-19593, 2021.
[39] Aniruddh Raghu, Payal Chandak, Ridwan Alam, John Guttag, and Collin Stultz. Contrastive pre-training for multimodal medical time series. In NeurIPS 2022 Workshop on Learning from Time Series for Health, 2022.
[40] Ziyu Liu, Azadeh Alavi, Minyi Li, and Xiang Zhang. Self-supervised contrastive learning for medical time series: A systematic review. Sensors. 23(9)2222,, 2023.
[41] Jean- Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. Unsupervised scalable represen- tation learning for multivariate time series. Advances in neural information processing systems,
32, 2019.
[42] Hugo Yèche, Gideon Dresdner, Francesco Locatello, Matthias Hüser, and Gunnar Rätsch. Neigh- borhood contrastive learning applied to online patient monitoring. In International Conference on Machine Learning, pages 11964 1974. PMLR, 2021.
[43] Chi Ian Tang, Ignacio Perez- Pozuelo, Dimitris Spathis, and Cecilia Mascolo. Exploring contrastive learning in human activity recognition for healtheare. Machine Learning for Mobile Health Workshop at NeurlPS, 2020.
[44] J Escudero, Daniel Abásolo, Roberto Hornero, Pedro Espino, and Miguel López. Analysis of electroenncephalograas in alzheimer's disease patients with multiscale entropy. Physiological measurement, 27(11):1091, 2006.
[45] PhysioToolkit PhysioBank. Physionet: components of a new research resource for complex physiologic signals. Circulation, 101(23)ee21-=<220, 2000.
[46] Hanneke van Dijk, Guido van Wingen, Damiaan Denys, Sebastian Olbrich, Rosalinde van Ruth, and Martijn Arns. The two decades brainclinics research archive for insights in neurophysiology (tdbrain) database. Scientific data, 91):3333, 2022.
[47] Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv: 1803.01003.1121, 2018.
[48] Richard Zhang. Phillip Isola, and Alexei A Efros. Colorful image colorization. In European conference on computer vision, pages 649-666. Springer, 2016.
[49] Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by max- imizing mutual information across views. Advances in neural information processing systems, 32, 2019.
[50] Xiang Lan, Dianwen Ng, Shenda Hong, and Mengling Feng. Intra-inter subject self-supervised learning for multivariate cardiac signals. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 4532-4540, 2022.
13

---

Appendix A A Medical Time Series Example
Here we use EEG data of Alzheimer's s as an example. An EEG dataset has many patients with or without Alzheimer's (healthy control). Data collectors take multiple trials on specific patient. These trials could be collected continuously within short time or across long period but follow the same experiment manner. Usually, the timestamps are too long for the deep learning pipeline to learn. For example, a 5 minutes trial with sampling rate 256Hz. has 76800 timestamps. Researchers generally use data preprocessing to split trial into many samples, such as 1-second and 3-second short samples, for further representation learning. Each observation denotes a scalar vector of real value at a specific timestamp. An experiment with a sampling rate 256Hz has 256 observations in one second.
Appendix B Data Augmentation Banks
Binomial masking: Generate mask following a binomial distribution that masks some timestamps of a sample, setting all channels at the masked timestamps to zero.
Channel binomial masking: Generate mask following a binomial distribution that masks some channels of some timestamps sample, setting only subset of channels at the masked timestamps to zero.
Continuous masking: Mask some continuous sequences of timestamps of sample, setting all channels at the masked timestamps to zero
Channel continuous masking: Mask some continuous sequences of timestamps of a sample, setting only a random half of the channels at the masked timestamps to zero.
All true: Do not apply any masking to the sample. Output the raw sample.
Appendix C Shuffle Function Banks
In real-world scenarios, ensuring the presence of samples from the same trial or patient within training batch becomes increasingly low probability as the number of patients grows. This situation can hinder learning meaningful representations at the trial and patient levels. To address this situation, we have designed two distinet shuffle functions that serve to rearrange samples while also upholding the requirement to include samples originating from the same trial and patient. These functions are called the "trial shuffle" and the "batch shuffle".
Trial shuffle: This function shuffles samples originating from the same trial and subsequently shuffles the trial order. Initially, we arrange the samples by sorting them based their trial IDs. Next, samples from the same trial are grouped into sets, and the order of samples within each trial set is shuffled. Finally, we sort the trial sets themselves while preserving the order of samples within each respective trial set.
This function shuffles samples in batch and subsequently shuffles the order of batches. The logic of trial and batch shuffle are similar. Initially, we arrange the samples by sorting them based on their trial IDs. Next, group samples into batch sets, and the order of samples within each batch set is shuffled. Finally, we sort the batch sets themselves while preserving the order of samples within each respective batch set.
Batch shuffle:
Random shuffle: Besides the two specifically designed shuffle functions, this random shuffle function shuffles all the samples in the dataset.
All the shuffle functions mentioned here are designed to shuffle samples within the dataset before training. During training, it is also essential to shuffle samples each epoch to prevent the model from memorizing the dataset and encourage it to learn more useful representations. To address this, we implemented a specially crafted BatchSampler class in PyTorch, following the "batch shuffle" approach. This BatchSampler shuffles the samples locally within each epoch, ensuring that the pre-shuffled sample order is not disrupted significantly. This approach guarantees that each batch contains samples from the same trial. It's worth noting that when a batch consists of samples from the same trial, it must have samples from the same patient.

---

Appendix D Data Preprocessing
D.1 AD Data Preprocessing
The AD dataset [44] comprises EEG recordings from 12 patients with Alzheimer's disease and 11 healthy controls. Each patient has an average of 30.0 + 12.5 trials. Each trial corresponds to a 5-second interval with 1280 timestamps (sampled at 256Hz) and includes 16 channels. Prior to further processing, each trial is scaled using a standard scaler. To facilitate analysis. we segment each trial into nine half-overlapping samples, where each sample has duration of 256 timestamps (cquivalent to 1 second). Additionally, we assign a unique trial ID and patient ID to each sample based its origin in terms of the patient and trial. We split training, validation, and test sets in a inieentindeeeedentt way. We use samples from patient IDs 17 and 18 as the validation set and samples from IDs 19 and 20 as the test set. The rest of the samples are all put into the training set.
D.2 PTB Data Preprocessing
The PTB dataset [45] consists of ECG recordings from 290 patients, with 15 channels sampled at 1000 Hz. There are total of types of heart diseases present in the dataset. For this paper, we focus on binary classification using subset of the dataset that includes 198 patients with major disease labels, specifically Myocardial infarction and healthy controls. To preprocess the ECG signals, they are first normalized using a standard scaler after being resampled to a frequency of 250 Hz. Duc to special peak information in ECG signals, a regular sliding window segmentation approach may result in the loss of crucial information. To address this issue, a different segmentation strategy employed. Instead of sliding windows, the raw trials are segmented into individual heartbeats, with each heartbeat treated as sample. To perform this segmentation, (1) the first step involves determining the maximum duration. The median value of R-Peak intervals across all channels is calculated for each raw trial, and outliers are removed to obtain reasonable maximum interval as the standard heartbeat duration. (2) The next step is to determine the position of the first R-Peak. The median value of the first R-Peak position is calculated and used for all channels. (3) Next, the raw trials are split into individual heartheat segments based on the median value of their respective R-Peak intervals. Each heartbeat is sampled starting from the R-Peak position, with the segments extending to both ends with half the length of the median interval. (4) To ensure the same length of the heartbeat samples, zero padding is applied to align their lengths with the maximum duration. (5) Finally, the samples are merged into trials, where 10 nearby samples are grouped together to form a pseudo-tre trial, similar to the neighborhood idea presented in [17]. We split training, validation, and test sets in a antient-independed the validation set and samples from another 28 patients(7 healthy and 21 positive) as the test set. The rest of the samples are all put into the training set.
way. We use samples from 28 patients(? healthy and 21 positive) as
D.3 TDBrain Data Preprocessing
The TDBrain [46] is large dataset that monitors the brain signals of 12 1274 patients with 33 channels (500 Hz) during EC (Eye closed) and EO (Eye open) tasks. The dataset consists of 60 types of diseases, and it possible for a patient to have multiple diseases simultaneously. This paper focuses on a subset of the dataset, specifically 25 patients with Parkinson's is disease and 25 healthy controls. Only the EC task trials are used for representation learning. To process the raw EC trials, employ a sliding window approach that continuously moves from the middle of the trial to both ends without any overlap. Each raw EC trial is divided into processed pseudo-trials with length of 2560 timestamps (10 seconds) after resampling to 256 Hz. These processed pseudo-trials are then scaled using standard scaler. Furthermore, each pseudo-trial is split into 19 half-overlapping samples, with each sample having length of 256 timestamps (1 second). In addition to the binary label indicating Parkinson's disease or healthy, each sample assigned a patient and trial ID based on the patient and processed trial it originates from. It is important to note that the trial ID refers to the ID of the processed pseudo-trial and not the raw EC trial. We split training, validation, and test way. We use samples from 8 patients(4 healthy and 4 positive) as the validation set and samples from another patients(4 healthy and positive) as the test set. The rest of the samples are all put into the training set.
sets in
2

---

Appendix E COMET and Baseline Implementation Details
We implement the baselines following their corresponding papers, including TS2vec [13], Mixing- up [16], TS-TCC [27], SimCLR [43], CLOCS [15], and TF-C [26]. In our COMET framework and all baselines, we use the last epoch of the contrastive pre-training encoder G for downstream tasks. For the P-FT and F-FT tasks, we save the best model in terms of FI score on the validation set during training and load the saved model to evaluate the performance on the test set.
COMET (our model) We use a two-layer, fully connected network as the projection head to map the input dimension to the output dimension. The input dimension corresponds the feature dimension of the data, while the hidden dimension is set to 128, and the output dimension is set to 64. To augment the data, we apply time series masking on the output dimension using the [all true, all truc, continuous, continuous] (see Appendix B) for the observation, sample, trial, and patient-level contrastive blocks, respectively. The augmented output dimension from the projection head is then passed to the encoder G' for representation learning. For the encoder G, we adopt a dilated CNN module. It consists of 10 hidden blocks, each following the order "GELU- -> DilatedConv -> GELU >> DilatedConv." A residual connection is applied between the beginning and end of each block. The dilation factor of the convolution in the i-th block is set to 2'. Each hidden dimension of the dilated convolution is set to 64, and the kernel size is set to 3. The output dimension of encoder G is fixed at 320. We utilize positive pair selection strategies specific to each contrastive block to build the contrastive loss in the embedding space (after encoder G). A max- max-pooling layer is employed before applying the representation to downstream tasks. During contrastive pre- pre-training, we set the learning rate to 0.0001. The pre- vre-training batch size is 256, and the total number of pre-training epochs is 100. We report the hyperparameters that achieved good results and stability among random seeds. The hyperparameters λ1, '2 λ3, 0 assigned values of (0.25, 0.25, 0.25, 0.25), (0.1, 0.7, 0.1, 0.1), and (0.25, 0.25, 0.25, 0.25) for the AD, PTB, and TDBrain datasets, respectively.
TS2vee [13] introduces contextual consistency using overlapping subseries and hierarchical loss function to capture data consistency at the observation and sample levels. To incorporate their methodology, we utilize the open-source code available at However, their code does not include downstream tasks for F-FT (Full Fine- mun-TTunng). so we implement these tasks using the same setup as COMET. Specifically, we set the number of epochs for contrastive pre-training to 100, the learning rate to 0.0001, and the batch size to 256. To align with model, we adjust the convolution blocks to 10, matching our configuration We adopt the default settings provided by the TS2vec implementation for other settings during pre-training.
TS-TCC [27] leverages temporal and contextual consistency by contrasting strong and weak augmentation. They employ a transformer-based autoregressive as the encoder. They perform a cross-view prediction by using temporal context to predict one view's 's future. Then, they maximize the similarity of contexts generated by the encoder to leverage contextual consistency. We utilize the open-source code available https://github.c https//gituub.com/mmaeelel www.///mmlllddee2225 TCCC.. We set the number of pre-training epochs to 100. We adopt the default settings provided by the TS-TCC implementation for other settings during pre-training.
Mixing-up [16] proposes a mixing-up augmentation by mixing the proportion of two time-series samples. This augmentation involves creating an augmented sample as the convex combination of two randomly selected tmme-eries samples from the dataset. The mixing parameter follows a beta distribution, determining the proportion of the two samples in the augmentation process. We to
utilize the open-source code available at https://github. wwwp:tthu......iiee..e.e wtwp///tttooomiiioooootttiiiiiii implement Mixing-up. Although they only provide downstream tasks for F-FT, we align their method with our setups for all downstream tasks, including P-FT and F-FT. We set the number of pre-training epochs to 100, the learning rate to 0.0001, and the batch size to 256 during pre-training
SiRCLR [18] is the most classic contrastive learning framework first proposed in the CV domain. It applies data augmentation techniques to create augmented views of input samples and constructs a contrastive loss based on these views. While initially designed for images, SimCLR has also been successfully adapted to time series data, as demonstrated in previous work such as [43]. To implement SimCLR, we utilize their open-source code available at https://github.co httggihuub.c////aaaang///// Contrastivel .AR.igHHA...... We set the number of pre-training epochs to 100, the learning rate to 0.0001, and the batch size to 512, aligning with their recommended settings. We use the default values provided in the SimCLR implementation for other configuration parameters during pre- trainin..
3

---

CLOCS [15] employs samples from the same patient as positive pairs to leverage the data invariance ECG recordings. They make use of both temporal and spatial information for contrastive learning. To implement CLOCS, we utilized their open-source code, available at We incorporated their contrastive loss function to im- plement the Contrastive Melti-segment Coding (CMSC) mechanism described in their paper. Addi- tionally, we modified their backbone to use TCN, which shares the same network structure as our COMET, including the configuration parameters. Specifically, we set the number of pre-training epochs to 100, the learning rate to 0.0001, and the batch size to 256.
https://github. https//gituub.com/aaiiiyyy com//anikiyassehhl..cC
TF-C [26] leverages the consistency between time domain and frequency domain. They assume that the time-based and frequency-based representations of the same example exhibit proximity in the fime-frequency space. We utilize their open-source code available at https///gthhub.com com/ Sannsssarvard/I---PPaa pretrainng to implement TF-C. While the original method primarily designed for transfer learning, we extend it to incorporate downstream tasks such P-FT and F-FT in our experiments. We set the number of pre-training epochs to 100, the learning rate to 0.0001, and the batch size to 256. We use the default values provided in the TF-C implementation for other configuration parameters during pre-training.
E.1 Partial Fine-tuning
In the P-FT (Partial (one-Tunngg) setup, we introduce a classifier L on top of the pre-trained encoder G, while keeping the parameters of G fixed. Only the classifier T. is fine-tuned in this setup.
In the COMET, TS2vec, and Mixing-up approaches, we utilize logistic regression from the Sklearn library to implement the classifier L. We use the default settings of Sklearn, except for setting the maximum iteration to 100,000. We employ a one-layer fully connected network as the classifier for the TF-C method. The learning rates are specifically set to 8e-5, 3e-5, and le-4 for the AD, PTB, and TDBrain datasets, respectively. As for TS-TCC and SimCLR, we follow their respective default settings for the partial fine-tuning phase.
E.2 Full Fine-tuning
In the F-FT (Full Fine-Tuning) setup, we introduce classifier P on top of the pre-trained encoder G. where both the parameters of the encoder G and the classifier /' are trainable. In this setup, we fine-tune both the classifier P and the encoder G. We utilize fraction of 100%, 10%, and 1 1% labeled training data for fine-tuning.
In the COMET, TS2vec, and Mixing-up up approaches, we set the finetune learning rate to le-4. We use a batch size of 128 and perform fine-tuning for 50, 100, and 100 epochs for 100%, 10%, and 1 1% label fractions, respectively. The classifier P is implemented hidden dimensions 128. For TF-C, we change the hidden dimension of this two-layer fully connected network to 64. The learning rates are specifically set to 8e-5, 3e-5, and 1e-4 for the AD, PTB, and TDBrain datasets, respectively. Regarding TS-TCC and SimCLR, we follow their respective default settings for the full fine-tuning phase.
two-layer fully connected network with
Appendix F Experimental Setting
F1
Patient-Independdet Experimental Setting
This paper adopts patient-inddpendent setting for the train-validation-ted split. In medical time series classification tasks, two common approaches to splitting the data are patient-independer and patient-dependent settings [15, S1]. Figure 3 illustrates these two settings' differences. In the patient-dependent setting, samples from the same patient can appear in both the training and testing sets, whereas in the patient-independee setting, samples from the same patient are exclusively included either the training or testing set.
Performing antient-indeeendent representation learning poses challenges due to the unique char- acteristics and different data distributions exhibited by each patient [50]. Even if two patients share the same label, the patterns within their data may differ significantly due to individual noise characteristics, potentially overshadowing the common patterns observed across patients. However, in real-world scenarios, it is essential for model to be robust and general to perform patient-independenn

---

representation learning. The goal is to train model on a subset of patients with known labels and utilize it to predict other patients with unknown labels, In contrast. autient-dependen classification is usually impractical since it requires knowledge of the labels for all patients.
F.2 Pseudo-Trin Experimental Setting
For many medical time series datasets, patient information is readily available, but trial information may be absent or limited to a single trial per patient. In such cases, the question arises of effectively employing trial-level contrastive learning. The solution is straightforward. We can generate pseudo- trials rather than merely deactivating the trial-level block by setting À to O.
We can define groups of adjacent samples pscudo-triall and assign them the same trial ID. In our paper, we employed ten neighboring samples as a pseudo-trial for the PTB and TDBrain datasets, and this approach yielded favorable results. This concept is akin the approach taken by TNC [17], where close samples are defined as positive pairs.
Appendix G Ablation Study, Visualization, Additional Downstream Tasks, and Heavy Duty Baseline
G.1 Ablation Study
Ablation study of contrastive blocks We examined the effectiveness of each contrastive block and progressively incorporated each block from the observation-level to all four levels. Table 5 compares the full-level COMET model with its six variants on the AD, PTB, and TDBrain datasets. The variants are as follows: (1) O, S, R. P. which utilize only one level of the contrastive block. We activate that specific level by setting its
and deactivate the other three levels by setting their to O; (2) S+O, which combines the sample and observation-level contrastive blocks. The À values for sample and observation levels are set equally to 0.5: (3) R+S+O, incorporating the trial, sample, and observation-level contrastive blocks. The À values for these three levels are set equally to 0.33; and (4) P+R+S+O, representing our complete COMET method with all four levels of contrastive blocks. The À values for the four levels are set equally to 0.25. The "Fraction" column indicates the fraction of labeled training samples used during fine-tuning.
We observed that variants R and P exhibit strong performance, achieving comparable or even outperforming the full-level COMET model P+R+S+0 on the PTB and TDBrain datasets across different fractions of labeled data (100%, 10%, and 1%). Notably, they also perform well with a 1 % fraction in the AD dataset, although they exhibit significant instability in this case. This finding suggests that adding more contrastive blocks may not necessarily improve final results. Achieving a balance in weights among different levels through is crucial. Nevertheless, training models at individual levels can provide insights into the importance of each level in contrastive representation learning. Furthermore, it's noteworthy that the full-level COMET model P+R+S+0 consistently demonstrates comparable or superior results across all datasets and fractions. Importantly, we did not perform any parameter tuning here, opting to set all the values equally, highlighting the stability of the full-level COMET P+R+S+O compared to other variants.
Ablation study of hy hyperparameter We conducted an analysis to assess the impact of hyper- parameter À on the AD, PTB, and TDBrain datasets in table 6. The values of \1, A2, λa, and Xa from left to right, correspond to the patient, trial, sample, and observation levels, respectively. analysis, all four data levels are incorporated, representing the full-level COMET model P+R+S+O. We applicd a significant weight to one data level by setting its corresponding À to 0.7 while assigning explored scenarios with increased weights
this
lower weights of 0.1 to the other levels. Furthermore, on patient and trial levels or sample and observation levels. The "Fraction" " column indicates the fraction of labeled training samples used during fine-tuning.
We observed that the results exhibited greater stability than the ablation study of contrastive blocks. In the contrastive block ablation study, there were significant discrepancies between different COMET variants at times. For instance, the S and R variants of the TDBrain dataset exhibited substantial differences across various fraction setups. However, in the full-level COMET model, the differences between these inter-running setups were notably reduced, even when a heavy weight was applied to one data level. For instance, the differences between lambda setups (0.1, 0.1, 0.7, 0.1) and (0.1,
5

---

Negative Class
10
Negative Class
Pesiive Class
Positive Cless
10
UNAN Dimersion
US6P Dimersion
Pairwiso
sstances:
10
10
11
UMAP timemsion
UMAP Ulmemioon
(a) COMET(OUrS)
(b) TS2vec
Figure 4: Visualizing the learned representation (a) Visualization for TS2vec. (b) Visualization for COMET(Qurs). The visualized representation is trained in the F-FT setup the AD dataset. Dark blue and light blue denotes the negative class(Health) and positive class (llzeeeee),)) respectively. We calculate the mean Euclidean distance between pairwise samples from two classes for each pair of samples to evaluate the class separability. As the figure shows, our method exhibits superior separation between the two classes. resulting in larger pairwise distances.
0.7, 0.1, 0.1) were not as significant the TDBrain dataset. This observation underscores again the stability and robustness of the full-level COMET model.
G.2 Visualization
To visualize the effectiveness of COMET, we depict the learned representation h, using the F-FT setup on the AD dataset as case study. It is important to note that the learned representation h; consists of 320 dimensions after pooling. To visualize the representations more interpretably, employ UMAP. dimensionality reduction technique with 20 neighbors and minimum distance of 0.2. To establish benchmark for comparison, we utilize TS2vec, which has shown the best performance among all baselines in the F-FT setup on the AD dataset. Additionally, we compute the average pairwise Euclidean distance between the negative (healthy) and positive (Alzheimer) classes, offering quantitative measure of separability between the two classes.
G.3 Performance on Downstream Tasks
Clustering We assess the clustering performance of COMET using the AD dataset case study. Instead of employing a classifier model on top of the encoder, we apply K-means clustering (K=2) to the encoder G. We utilize three widely-used evaluation metrics: Silhouette score, Adjusted Rand Index (ARI), and Normalized Mutual Information (NMI). To establish a benchmark for comparison, consider TS2vec, which has shown the best performance among all baselines in the F-FT setup on the AD dataset. Table illustrates that COMET surpasses TS2vec with an improvement of 0.0586 in Silhouette score, 0.945 in ARI, and 0.881 in NMI.
Anomaly detection We evaluate the anomaly detection performance of COMET using the AD dataset as a case study. While some previous works focus on identifying outlier observations within sample [S2, S3]. we concentrate on sample-level anomaly detection. We construct a very unbalanced AD test set comprising 90% negative (healthy) samples and 10% positive (Alzheimer's samples. The negative samples are considered normal, while the positive samples are treated as outliers. The test set is prepared accordingly, while the remaining aspects of the experiment follow the F-FT setup. Specifically, We utilize the saved trained models from the F-FT setup to evaluate the new test set, and for comparison, we still employ TS2vec as a benchmark. The "Fraction" column indicates the fraction of labeled training samples used during fine-tuning. The experiment result is shown in table The COMET outperforms TS2vee by 5.25%, 15.3% and 11. 11.4% with label fraction 100%, 10% and 1,, respectively.

---

Table 5: Ablation study of contrastive blocks. The ablation study of contrastive blocks is evaluated on the AD, TDBrain, and PTB datasets. We examine the effectiveness of cach contrastive block. Besides, progressively incorporate cach block from the observation-
to all four levels. Here, O, R, and denote the observation, sample, trial, and patient-level contrastive blocks, respectively.
Datasets Fraction
Blocks
Accuracy
Precision
Recall
score
AUROC
AUPRC
O
81.69=1071
87.28+5.1
79.64
18.90
92.54=415
92.36=434
83.53
84,354169
12.81
82,97.3.47
91,0111.95
90.81
R
74.58
83.14=11.60
16.31
74.53+18.0
85.05
84.44
100%
P
12.69
78.67=1181
69.74=1435
67.04
13.30
82.57+14.29
S+O
85,70=
16.14
85.09=2.0
85,35=1196
-
92.09=1.30
R+S+O
85.57:14404
88. 12
84.3 1:4
84.73:24.99
93.28-2.52
92.98
P+R+S+O
84.5014.46
88.31
82.95....9
83.33
94.44 1.37
94,43
O
86.35
87,0949904
85.55 19.9
10.10
92.62-8.99
92.77
77.30
78.45
76.26
76.3611392
85.43_161
84.63
R
77.83117.59
83.45
11941
72.08122.45
13.50
83.27
AD
10%
P
71.41:1531
76.91 1:1.56
18.70
66. 06.54
16.11
76.06
S+O
82,73=2.05
84,33++271
81.51 =2.07
81.98=211
90.02=244
90.02=2240
R+S+O
11.19431.4
91.74+3301
90.70=3344
90,98
95.86-263
95.86=2.67
P+R+S+0
91.43
92.52+236
90.71
91.14
96.44_284
96,48=2281
O
69.56+7.54
71.43++804
68.19 79
67.83+7741
77.32-s.69
77.32=
+9.09
61,08++886
59.83=440
58.1243357
63.99-662
63.33==6.0
R
90.15
93.83:7728
19.03
88.17:1675
96.22
06.02222625
1%
P
85.57
90.36.17.15
83.98115
82.72.180:
94,23 5..82
94, 10..66
S+O
63 13.22
63.5 63.52.1415
63. 63.3014.26
62.72.4144
67. 5.17
67. 67.25 14.99
R+S+O
82.65
83,21
82.97=24.8
82.46:4443
90.07=677
90,24:56.6
P+R+S+0
88. 88.22+3.36
88.55+273
88.56=3.14
88.14+3.37
96.05-1.36
96.12=1.31
85.27
84.21
7..74
79.78
88.13
84.76-:2.14
84.30=2.56
84.00+216
75.68=5.69
77.74
88.66-205
84.80=2.15
R
18.63+144
88.42+145
82,46=2335
84.72+2.12
89.29-3.9
86.21
100%
P
88.85
88.74:220
82.78 6811
84.75
94.32_188
90.61 =2211:2211
S+O
84.33++1.18
75.49:5.69
77.69
90.37=2.99
86.57 =4.32
R+S+O
15.32+1.9
84.82+11.8
77.54
79.64=3994
92.34
1.195
P+R+S+0
86.36+1144
87.18+126
77.87
80.79
93.4
89.09
O
86.75.144
85.42.12.10
80.88 1:.27
88
86.34.10.3
84.7511499
81.21
81.90
88.36
85.20
R
88. 1.77
86.36:1224
84. 84.16:3355
84.80
91. 21.80
90.45 =0.94::11.00
10%
P
90.38
90.33+221
35.69:4336
87.27 +7.27
93.06-3886
91.83=2293
S+O
85 1..74
84. 14.14
80.07 15.18
81.14
90.28 2.775
87.57 1:3.2
R+S+O
85.88
83,6043.11
82.48
90.79-1.79
88.68
P+R+S+0
9..32
89.67+194
85.85
87.35
92.78=1.11
90.46=3309
O
85.73+1.59
13
79.53
81.09
90.15
88.85=2.64
S
1..6
72.36:2241
69.41 :2.5%
70.38
79.09-3.21
75.07 15.0772:188
R
86.28 11.97
84,02.1285
12.06.2208
82.61
90,50_00.9
88.32.1.40
1%
P
91.55
90,28
88.45 ::323
89.25
19.25.111 1311
95.12-1.55
94,80
S+O
8 1..78
77.65
77. 17:22
77.20
86.65_2.55
82, $2.112:1.99
R+S+O
80.71
76.791203
80.51
17.84
1..76_1.38
15.00.2.21
P+R+S+0
86.62..350
82.9914.52
85.36
83.89.1496
93.78_1.82
91.941449
O
89.92.1.11
90.50.2.07
89,92 19.92.1215
19
96.79=1.29
96 1..9
77..6
18.9812251
77.86 77.86.:1
77.6211.01
88.44_2.2
88, 12.96112.14
R
94,44
94,60
94,44
94,44:1100
98.39
98.31:1.4
100%
P
93. 18 3.77
93.27
93.18 :8:70
93.18
97,59-201
97.58 77.58=200
S+O
80.38+4422
81.59+3.70
80.38
80.16
91.35-2224
91.64=205
R+S+0
86.01
86.35
86.01
85.95
94.14-172
94.37=22.x
P+R+S+0
85.47:11.16
85.68
85.47:1111
85,45
93.73=1.02
93.96.1009
O
8-.34++.38
86.03+397
85.34=4.84
85.25
93.18-3.55
93.24=3.51
S
74.02:22.0
74.62:201
74.02
73.8622.17
81.92-1.25
81.76=::41
R
92.96188.12
93,02.1820
92.961822
92,961823
96.14
95,93
TDBrain
10%
P
89,38 11.69
19,58114772
89.38
89,36.11.69
93.23
93.52.11110
74.92+2.57
76.6043307
74.92=2.57
74.54
84.51=3.07
84.40=:3.07
R+S+0
81.90++474
03
81.90=474
81.61+4499
91.21
P+R+S+0
79.28
89.83
79.28-2464
79.19
88.39-413
88.38
O
71.52+7.54
72.174771
71.52
71.31
78.32
77.56=8891
S
58.62+3.84
59.66+335
58.62=3844
57.39+4426
62.85
61.61
R
85.29
85.55+1566
85.29 5..3
85,24
91,19-476
91.11 14774
1%
P
77.23:1342
78. 10:...99
777.23-334
77.04
86. 12-22..88
85.10.4411
61.711229
61,82.1290
61.711297
61.60
66.09
64,94 14.94_1266
R+S+O
72.1516.26
73.39.7.50
72.15
71.91
78.39
76.97
P+R+S+0
72.93+7.11
74.20
72.93-47.1
72.57
78.72-8.42
77.71

---

Table 6: Ablation study of hyperparameter The ablation study hyperparameter evaluated on AD, TDBrain, and PTB datascts. The A1, A2, Xa, da from left to right are for patient, trial, sample, and observation levels, respectively.
Datasets
Fraction
Ag, As, A4
Accuracy
Precision
Recall
F1 score
AUROC
AUPRC
(O.I..1.O.....))))
87.82:7.4
91.08:4411
86.62.s%.
86.77
97.95
97.94
(0..0.1.......II
85.03
89.09
83.32
83.78
95.33
95.30=1.00
(0.1...7.0.I...I)))
82,76
88,00
80.98 1002
80.72=10.30
95.32=2.72
25.26.2771
100%
(0.7.9.1.0.I.0.))
82.22
87.27
80.42.962
80.1 1000000
94.8442264
94,80
(0.2.0.2.033..333)
86.88
89.96-2400
85.46
85.97
94.03-33.56
94.07
(0.3.0.3.0.2.0.2))
86.72:5.09
89.76:3.12
85.4 1::9
85,73
95.14
95.0992204
(0.3..33.0.15.02255
82.84
88.0943.99
80.87
81.04
94.73+a2.98
94. 94.64
(0.1.0.10...0.7))
89.56
91.37
88, 28.55
96.55=4441
96.47
(0.1.0.1.O.....I)))
87.84
88.7245.16
87 a:
87 a2:26666
94.58.5532
94, 385
(0.I.....O.I...I)))
19.89
85.79
17.97=
15.54=19.81
92.89
92.67
AD
(0.7.0.1.0.1.0.1)
18.26
84,9916668
36.38:115 su
19.44
92.74
92.57.77.5
0.2.0.2...3333333)
93,25::171
1.28
92.77 22.00
93
97.75 97.68:4577
0.3.0.3..O2..022))
91.43+33.70
92.71+2.99
90.68
91.09=400
96.81 =:1.2
96.88 1..6
(0.3.0.3,0015.02255)
11.54
92.93
90.77.4425
21.19
97.26
a7.29
(0.1...1.0.....7))
25.88
96.21 92
95.64
95.79
92.72:19.44
13.24
(0.1I..1.0...0..))
85.03-1981
16,16755
85.65
84 14.622210422
94.71
94, 34.73-22.8M
(0.1...7.0.1.0.))
97.19
97.32
97.08
97.15
99.02:1.6
98,98:1.11
(0.7.0.1O...O.I)
97.00
97.10
96.92+1.14
9.1
98.93+1.20
98.89
(0.200.2.00330333)
84.55
84,47
84.75
84,44
89.95=10.8
11.78
(0.3.0...0...0..))
86.96125.9
87.1615..6 66
86.87
86.72.5 19
92.68227.6
92.93
(0.3.033.0.15.02255)
92.40=2.52
92.75
92.16+2244
92.27
97.76=1.11
97.76
(O............77))
85.41
85.54
76.96:44.0
79. 39.60
91.30=22.2
87.11
(0.11..1.0...0.I))
83,94
87,23-1406
76.691561
79.64.559
93.78
90.78:478
(0.1.0.7.0...0..))
1..84
87.67
81.14
83.45
92.95
N7.47
100%
(0.7...1O.....)
87.76=275
14.20
80.65
82.96=5.07
9182+#3.6
88.70=3.08
(0.2.002.03330333)
87.74
87.78
80.79
83, 83.28
92.97=1.09
88.67+177
(0.3.0...0.2.0.22)
87.16=2.02
87.62++1.60
79.47
82.15
94.00
88.83++1.65
(0.3.0.3.0015.02255))
87.56
87.91 10064
80.25
82.92
91. 25.44
88.27
(0.1...1.0.1.0.7))
87.76
87.15
81 15.33.5..0
83.43:5415
92.54
89, 89785
(O.10.11....0..))
88.29
85,80+3.57
86.87
85. +5.38
94.566
93.15
(0.1.0.7.0.1.0..))
88.98
1:600
94.83
12.32
PTB
10%
(.......O...O..)
38.70#3.0
89.47 +1.40
81. +1.85
84.26
93,24
(0..0.2...3333333)
89.25
88.87 22.55
83.98
85 85.72:+3.1
94.26
90.82+3333
(0.30.3.002...22))
89.39
86.8861..
84.43
86.03
23.59
92. 42.1..0
(0.3.033.011..02555)
89.82
19.36:2216
84.71 +
86.34:454
93
90.54
(0.11..1.0.I.0.7)))
89.45
87, 17.4
86.23
86.62a2.7
94.03-22.9
93. 1:000
(0.1...100.....))
12:
81.33
83.06
82.01 2::76
91.12:2214
88, 88.9622292
(0.1.....0...0.)))
90.52
88.58 +1.9
88.23
88.23:2.10
95.08
9..78
1%
(0.7.0.100.1.0.))
10.19
87, 2.22
88.20
87.87 2.11
14.8
93. 23.68
(0.200.2.03...333
85.28 w
81. 11.48:55.0
84. 10.10aa:40 66
82.4 82.47
92.87
90.34
(0..0..3...2.0.2))
86.96=2.56
83.38
85.65
84.32
94.47=11.0
92.31
(0.3,0.3.0115.0.255)
89.43
86.42+196
88.86:4.94
87.37
95.32::1.44
94,29
(0.11..1.0.I.0.77))
2..3
87, 3..
87.20
87.18
94,84
94.96 96.1
(0.1...1.0.....))
88.53
88.6611799
88.53
95.57-22744
(............0..)
87.97
88.33
87.97
87.94
95.58
95.77
100%
(..7...10...0..)
87.25
87.54++274
87.25+2271
87. 87.22
14.553
95.32
(0.2..02..33.3333))
85.69#242
86.00+2.53
85.69+2242
85, 55.66+2241
94.14
14.31
(0.33..3...2.0.2))
85.76
86.24.1375
85.76.33.1
85. 85369
14.34
94, 21.75
(0.3.033.0.15.0.255)
85.5424.11
86.30
85.54:14.11
85.44
94,22::3.6
94,36:2200
(...I..1O.....7))
76.92
+8.35
76.92:4454
76.50
85.85
85, 35,366:4330
(0.1...1.0.7.0.1))
81.91
82.40):5
61.91
81 83
90.38
90.72.15.6
(0.11..7.0.I.0.I))
84.51
84,90:461
84,51
84.45
92.58=3330
92.62.43.09
TDBrain
10%
(0.7..........I))
84.92
85.24
84.92
84.86
92.69=407
92.69 :381
(0.2..2200330333
80.794384
82,001
80.79
89 28.50
89.56
(0.3.0...0...0.2))
78.93
79.78
78.93
78.77
88.64=3352
88.59
(0.3...3.0.15.0.255)
79,96:5..3
81.00+5.52
79.96s3.63
79.76
88.89
88.70
(0.I...1.0.I...7))
68.68+2288
70.46
68.68+2264
68.02
73.79
72.99
(0......0.......)
66.49
68.4647764
66.49
65.03
73.30
72.74
(0.110.7.0.1.0..))
73.081861
14.89
73.08
72.45
78.39
77.79.4418
1
(0.7.9.1.0.1.0.1)
16
15.14
73.1624.72
72.4622006
78.42-448
77.7624.08
0.2.0.2...3..3333
71.55+8.00
72.14:19.44
71.55
71.44
77.14
76.04
00.30....0...0.2)
71.68
70.114807
69.54
76.24=10.55
75.26=1008:
(0.3.0.3.0.15.0.255)
73.59
74.53
73.59
23.33
79.07
77.97:19.2
8

---

Table 7: Performance downstream clustering. The clustering performance is evaluated on the AD dataset.
We compare the baseline TS2vec, which performs best in the F-FT setup.
Method
Silbouette
ARI
NMI
Random Inil.
0.1184.1 0.0082
0.1 111889000064
0.125810.0060
TS2vec
0.0795= 0.0032
0.0013+ 0,0026
0.0018+000016
COMET (Ours)
0.1381 0.0139
L..026
0.8827.10.0144
Table 8: Performance anomaly defection. Sample-level anomaly detection very unbalanced AD set comprising 90% negative (healthy) samples and 10% positive (Alzheimer) samples.
Fraction
Models
Accuracy
Precision
Recall
score
AUROC
AUPRC
TS2vee
82.11
66.05
82.13:321
68.70
91.27:1.47
76.80
100%
COMET (Ours)
83,03=11.65
71.7611760
90.3316.1
73.95
97.99
91.52
TS2vee
66.05
62.48
77.33
62.67
86.7643395
72.21 1:509
10%
COMET (Ours)
88.22=2.8
73.22+3.1
92,49+173
77.97=387
97.91
92,21 #443
TS2vee
67.24=8.05
57.3413112
67.87:645
54.35=6.21
72.75+682
61.322555
1%
COMET (Ours)
74.57
64.72:11.5
84,41
65.74
93.13
79.65
G.4 Heavy Duty Baselines
In COMET, incorporate four contrastive blocks to leverage four levels of data consistency, allowing the data to pass through the model four times within one epoch during contrastive pre- training. To ensure that our superior performance is not due increased data passing, conduct experiments on the AD dataset with two baselines: SimCLR and TS2vec.
SimCLR utilizes only one contrastive block during training. We employ two strategies to match the data passing number with COMET: (1) Run SimCLR with one contrastive block for four times the original number of epochs in pre-training (400 epochs instead of 100). (2) Duplicate the contrastive blocks, resulting in four SimCLR contrastive blocks. The notation 4E signifies running the model for four times the original number of epochs, while 4B indicates the use of four times the number of contrastive blocks compared to the original SimCLR.
TS2vec incorporates two contrastive blocks during training, leading to data passing twice within one epoch. Similarly, we adopt two strategies to align the data passing number with COMET: (1)
Run TS2vec for two times the original number of epochs in pre-training (200 epochs instead of 100).
(2) Duplicate the contrastive blocks, resulting in four TS2vec contrastive blocks. The notation 2E denotes running the model for four times the original number of epochs, while 2B indicates the use of four times the number of contrastive blocks compared the original TS2vec.
The results are presented Table 9. We observe that simply increasing the number of epochs or contrastive blocks does not improve performance but rather leads to decrease in most cases. We speculate that this decrease is caused by overfitting.
Appendix H Broader Impacts
Our approach for sule-supe contrastive learning improves classification performance on target datasets in patient-independent medical diagnosis scenarios. Leveraging different data consistency levels in medical time series is crucial to enable effective and accurate contrastive learning without sufficient labels. Our work will encourage the research community discover universal frameworks for other practical applications based on time series representation learning. We also hope our work can attract more researchers to the more general problem of hierarchical consistency from other related fields.
From the societal perspective, our work and the line of contrastive learning can promote more efficient use of medical time series with the lack of labels. Specifically, our model has the potential to identify patterns and anomalies that may not be immediately apparent to human experts. This could lead to earlier and more accurate diagnoses, improving patient outcomes and reducing healthcare costs. However, practitioners need to be aware of the limitations of the model.

---

Table 9: Heavy Duty Basclines. Run epochs add contrastive blocks SimCLR and TS2vec the AD dataset.
Fraction
Models
Accuracy
Precision
Recall
F1 score
AUROC
AUPRC
4E-SimCLR
26.87=251
4006
53.31
48. 48.2
57.67
51.97
4B-SimCLR
53,92=381
$1.88
51.266:3.05
46.92=507
51.88
1..75:1.69
SimCLR
54.77=199
50.15=7.02
50.58+
43.18
50.15
1.04
100%
2E-TS2vec
76,49=6.10
78.97
77.6945.16
76.21 16.4
88. 28.44
88. 12:12.33
2B-TS2vec
81.61
81,47=1.75
81.53+
81. 146
89.50=1.60
89.22+1.75
TS2vec
81.26=2.08
81.21
81.34:1201
81.12
1.76
88. 1.85
COMET (Ours)
84.50 :.46
88.31
82.95
83.33
94.44 2.33
94.43
4E-SimCLR
77
58.41
53.694225
46.47=577
58.4 =
1..33
4B-SimCLR
$3.57=6.29
53.89
52.97+4.22
50.60=5.46
53. =:500
51.80+2334
SimCLR
56,09=2.25
53.81=5.74
51.73+2.59
44.10 14.1044.84
53.8 $3.81=1574
51.08
10%
2E-TS2vec
66,29=7.86
69.92.s511
67.7916.1
65.36
1.154
77.95
2B-TS2vec
72.61 24446
73.86-436
72.98+3.66
72.30
81.91 =483
81.74-1485
TS2vec
73.28
4..33
73.52
73.00
61.66
81.58
COMET (Ours)
91.43=2312
92.52=2.36
90.71
91.14
96.44=22.4
96.48:1222
4E-SimCLR
58.07
57.72=3.50
54.92
51.93=3.11
57.72=3.50
52.91 =
4B-SimCLR
54.67=5.4
54.86=4944
64.48+44.4
53.68
54.86=4494
$2.67 +:008
SimCLR
55,42=243
52.1815.55
51.371276
45.02:4.79
18.15
50.871145
1%
2E-TS2vec
63.562462
64.97
64.49
63.28
70.26
68. 13.59
2B-TS2vec
64.18-4.5
64.26-480
64.26:14.00
63.93
70.07=5.97
68.62.4625
TS2vec
6..93
65.28
65.1443.59
6..64
70.56
68.97.55.5
COMET (Ours)
88.22:3336
88.55
88.5613314
88.14 34.
96.05
96.12:11.1
All datasets this paper are publicly available and are not associated with privacy or security concerns. Furthermore, have followed guidelines responsible specified by the primary authors of the datasets used in the current work.
10

---

Appendix References
[SI] Cosimo leracitano, Nadia Mammone, Alessia Bramanti, Amir Hussain, and Francesco C Morabito. A convolutional neural network approach for classification of dementia stages based on 2d-spectral representation of eeg recordings. Neurocomputing. 223:96-107, 2019.
[S2] Ling Yang and Shenda Hong. Unsupervised time-series representation learning with iterative bilinear temporal-spectie fusion. In International Conference on Machine Learning, pages 25038-250555 PMLR, 2022.
[S3] Dennis Bäßler, Tobias Kortus, and Gabriele Gühring. Unsupervised anomaly detection in multivariate time series with online evolving spiking neural networks. Machine Learning, 11144:1377-1408, 2022.


Reviews:
[{"review_id": "hGmMFYdSl9", "reviewer": "Reviewer_Trt3", "summary": "This paper proposes a multi-granularity framework leveraging data consistencies at different levels inherent in medical time series data. The model learns with contrastive loss designed at every data granularity, i.e., observation, sample, trial, and patient levels. The method is evaluated with three binary classification downstream tasks, showing significant gain over SOTA timeseries contrastive learning methods.", "questions": "1. The difference between ‘sample’ and ‘trial’ are a bit unclear. It seems that we can also consider a trail as a sample with a longer time span? How to segment trials into distinct samples? ”Each sample is a one-second interval with 256 observations. ” – Any justification for choosing one-second interval here?\n2. I am curious about the implementation details for Observation-level data consistency. How $t^-$ is chosen? Shall it be close to $t$ or distant from $t$? How to guarantee that $x_{i, t^-}$ is more different to $x_{i, t}$ than $\\tilde{x}_{i, t}$, as the time series data often display repetitive periodicity? \n3. When jittering and masking are used for data augmentation, would it broken the continuity of the normal time series data?\n4. The decision of using InfoNCE at observation and sample blocks while using NT-Xent loss at trial and patient level seems to be very arbitrary. Any explanation for it?\n5. SOTA comparison: why NCL and CLOCS are not considered?\n6. Any ambulation study on the hyperparameters $\\lambda_i$ is missing.\n", "limitations": "the limitations are presented in the appendix.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The idea of leveraging data consistencies are multi-granularity is interesting and seemingly effective.\n2. The writing and presentation is easy to follow.\n", "weaknesses": "1. The contrastive learning for each granularity is simple and straight forward, making the novelty of the paper limited.  \n2. Majority of the medical tasks are fine-grained in nature. That is to say, the difference between two samples of difference classes may be very subtle and local. It is unclear how the data augmentation method for each granularity is designed to ensure not disturbing the sample label. \n3. The evaluation is a bit unpersuasive. The downstream tasks are all simple binary classification problems (e.g. for dementia detection, Myocardial infarction, or Parkinson’s disease).\n4. For medical diagnosis tasks, accuracy and F1 shall not be used as the main metric as the data distribution is usually drastically imbalance. I would suggest either focus on AUROC or AUPRC. In this case, COMET shows inferior performance on PTB than TS2vec. Any explanation for this results?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are happy you feel our paper is easy to follow, interesting, and effective. We briefly respond to your concerns point-by-point. If you do not feel we have sufficiently justified a higher score, please let us know where we can further improve our work. Thank you again! \n***\n**Q1** The contrastive learning for each granularity is simple and straightforward, limiting the novelty. \n\n**A1** We agree that contrastive learning of each granularity is straightforward and not overcomplicated. However, we sincerely believe that proposing a simple solution to address a challenging problem should exhibit the strengths of our work. In this paper, we posited the granularity conceptions in medical time series and designed a simple-yet-effective framework that uses all possible granularities in a self-supervised manner. Our COMET covers and is compatible with existing SOTAs on different granularities, which will play a significant role in summarizing, inspiring, and guiding future works in contrastive learning of medical time series.\n\n***\n**Q2** The difference between two samples of different classes may be subtle and local. How does the data augmentation method for each granularity not disturb the sample label?\n\n**A2** Our answer contains two parts.\n\n1) We did not apply data augmentation to patient and trial levels. Instead, we employed patient and trial IDs to establish positive and negative pairs for learning consistency across instances (Lines 167-176).\n\n2) We used timestamp masking [6] for data augmentation at sample and observation levels. To prevent useless masking on zero-value regions, we applied masking to the projected embedding rather than the raw data (Lines 278-283). This method also mitigates the risk of masking affecting sample labels in medical tasks, as the embeddings of two samples in different classes could exhibit substantial differences after projection, even if their raw data appears similar.\n\n***\n**Q3** The evaluation is a bit unpersuasive. The downstream tasks are all simple binary classification problems.\n\n**A3** Our answer contains three parts.\n\n1) In the initial submission, we evaluated COMET on diverse downstream tasks besides binary classification, including clustering and anomaly detection (Appendix F.3). \n\n2) We mainly presented the results of binary problems because the majority of real-world applications of medical time series for disease diagnosis are binary classification tasks [7][8].\n\n3) We newly added a large-scale dataset PTB-XL (17596 patients, 191400 samples, 5 classes) to prove our model works well on multi-classification problems. See R-table 3. Our method performs better in 11 of 12 tests than the other three SOTAs. Particularly with label fraction 1%, we outperform SOTAs by about 5% F1 score and 3% AUROC, demonstrating the effectiveness of contrastive pre-train of COMET in reducing the dependence of labeled data.\n***\n**Q4** 1) Suggest focusing on AUROC or AUPRC on imbalanced datasets. 2) COMET shows inferior performance on PTB than TS2vec.\n\n**A4** \n1) We agree that AUROC and AUPRC are more appropriate for medical tasks. We presented six metrics (including AUROC and AUPRC) to provide comprehensive results to readers so they can analyze the results with their preferences. We will emphasize more about AUROC in the camera-ready version. \n\n2) We did a careful ablation study on each block as you suggested and found increasing the lambda weight on the observation block benefits the performance (See R-table 2). We think the periodicity of ECG signals causes the reason. After adding more weight to the observation block, we showed better results on AUROC(93.67±2.34) compared with TS2Vec.\n***\n**Q5** 1)The difference between ‘sample’ and ‘trial’ is unclear. Is the trial a sample with a longer time span? 2) How to segment trials into distinct samples? 3) Any justification for choosing a 1-second interval?\n\n**A5** Our answer contains three parts.\n\n1) A trial can indeed be considered a sufficiently long sample. In the original submission, we have provided a detailed example in Appendix A to elucidate the distinctions between trial and sample. A trial is usually a much longer sample. It can span hours or even days, containing hundreds of thousands of timestamps, which is impractical for models to train on entire trials effectively. Consequently, researchers commonly partition them into shorter samples for training purposes [1][2][3].\n\n2) Signal segmentation has been a standard operation in signal processing for decades. Our approach to segmenting each dataset is detailed in Appendix C. \n\n3) The window size can be regarded as a hyper-parameter. Empirically, this selection ranges from 1 to 10 seconds, contingent on factors such as sampling frequency, data types, etc.[2][3]. We opt for 1 second simply due to its superior performance. \n\n\n***\n**Q6** Implementation details for Observation-level data consistency. How $t^-$ is chosen? How to guarantee that $x_{i,t^-}$ is more different to $x_{i,t}$ than $\\widetilde{x}_{i,t}$, as time series data often display repetitive periodicity?\n\n**A6** Our answer contains three parts.\n\n1) Only observations augmented from the same raw observation are treated as positive pairs, while all observations with distinct timestamps within a sample are designated as $t^-$ (Fig. 2; Sec. 4.2). The principle to select negative observations ($x_{i,t^-}$) aligns with the majority of contrastive learning studies for negative samples [6]. \n\n2) As we employed a projection layer to map raw data into embeddings before applying data augmentation, such mapping disrupts the periodicity of the raw data.\n\n3) Similar to sample-level contrasting, there is no assurance that all negative samples are exclusively hard-negatives (true negatives). It is quite common for certain negative samples to share the same label as the anchor sample [9]. In other words, there is no certainty, nor necessary, that all negative observations are dissimilar to the anchor observation. \n"}, {"review_id": "3Osx9kVCoI", "reviewer": "Reviewer_RVrY", "summary": "Medical time-series data, unlike domains such as CV and NLP, lack data labeling but contained more layers of information corresponding to observation, trial, and individual physiologies. Unlike previous methods that overlooked the multi-granularity, the authors proposed COMET to trained the combined contrastive losses from all 4 granularities and presented COMET’s elevated performance on multiple datasets, outperforming many up-to-date contrastive baselines. With ablation studies and disclosed code repo, the authors evaluated the effectiveness of proposed COMET and concluded it as a pioneering framework tailored for medical time series. ", "questions": "•\tCan authors address the limitations? is this work specific to cyclical time-series data?\n•\tHow do the results compare to other ECG contrastive pretraining methods?\n•\tReading the “comet.py” and “models/encoder.py’s TSEncoder”, it seemed like the same inputs “x” were used to calculate 4 levels of contrastive losses. Then what exactly is “x” (the time series input of the COMET), is it observation-level? Sample-level? Can authors better describe the actual inputs to the system?\n\n", "limitations": "•\tOnly addresses one modality of medical timeseries data\n•\tDoes not compare to other ECG contrastive pretraining methods\n\nThese have not been adequately addressed.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "•\tFigures are well done and make it much easier to understand what the authors are talking about when they mention the different levels of granularity for medical time series data\n•\tThe paper is very well written and easy to understand. The method seems simple but this is likely due to the authors’ explanation of information granularity. The paper follows a clear logical flow.\n•\tCapturing multi-granularity information with contrastive learning is a very good idea backed by performance improvements. \n•\tThey backed their proposed COMET with comprehensive experiments on multiple datasets with many up-to-date baselines\n•\tThe performance of their method is very high in comparison the methods they consider to be related work.\n", "weaknesses": "•\tWhile the generalization is likely to follow from this work - this method is somewhat limited to ECG data. ECG data has many structural patterns in medical time series data that come from the cyclical nature of a cardiac cycle that other medical time series might not. As a result, the methods might actually be limited to pulsatile-type medical time series, which would still be a contribution.\nFor example On page 2, line 71, the statement that medical time series is \"low cost and non-invasive.\" would not necessarily be correct for data collected from invasive A-line, for example. However, I mostly agree with the paragraph on EEG, ECG, etc., and how they benefit healthcare practices. I would suggest authors better approach this limitation which does not take away from the contribution of the work overall.\n•\tWhile many works are compared, other ECG-related methods, one of which they mention in the paper, CLOCs have not been compared. How does this method compare to ECG specific methods?\n•\tIt was unclear how the time series features were fed into the COMET and how they were broken into different granularity and evaluated by different contrastive losses (see first bulletins of questions). \n•\tIt would be better to include ablation study on the process of discovering the optimal set of hyper-coefficients for each dataset. \n•\tOn Page 6 line 215, D is defined as dataset, but it should be better defined before referenced, namely in Section 4.2 or earlier?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your very thoughtful feedback. We highly appreciate you carefully reading our figure, method, appendix, and code! We are happy you feel our paper follows a clear logic and is easy to follow!! Again, thank you for your comments to help us improve our paper. Here are the response to your questions.\n***\n\n**Q1**: Can authors address the limitations? is this work specific to cyclical time-series data? • How do the results compare to other ECG contrastive pretraining methods?\n\n**A1**: Thank you for raising these questions. We add CLOCS and NCL as new baselines for comparison and add one more large-scale ECG dataset PTB-XL (17596 patients, 191400 samples, 5 classes)for the experiment. See R-table 3 for experiment results. Our method performs better in 11 of 12 tests than the other three SOTAs. Particularly with label fraction 1%, we outperform SOTAs by about 5% F1 score and 3% AUROC, demonstrating the effectiveness of contrastive pre-train of COMET in reducing the dependence of labeled data.  \n***\n\n**Q2**: Reading the “comet.py” and “models/encoder.py’s TSEncoder”, it seemed like the same inputs “x” were used to calculate 4 levels of contrastive losses. Then what exactly is “x” (the time series input of the COMET), is it observation-level? Sample-level? Can authors better describe the actual inputs to the system?\n\n**A2**: Thank you again for carefully reading our code! The input \"x\" utilized for each contrastive block is all sample-level data, as our primary objective entails acquiring sample-level representations.  A batch of sample-level data \"x\" is passed to the encoder to process and apply data augmentation to generate embeddings for each block. Distinct levels exhibit varying data augmentation approaches or no augmentation. Subsequently, the embeddings from each level, coupled with their respective patient/trial IDs, are fed into the system to compute the contrastive loss. Each level is strategically designed to capitalize on consistency at differing granularities, engendering distinct contrastive loss formulations. For instance, positive and negative pairs are generated through data augmentation at the observation and sample levels, whereas at the patient/trial level, they are formed based on the presence of the same patient/trial ID. "}, {"review_id": "ABYkEiBrAz", "reviewer": "Reviewer_jbn1", "summary": "This paper proposes a multi-granularity contrastive learning method (named COMET) on medical time-series data. The methods build postive and negative pairs from patient-level, trail-level, sample-level, and observation-level. The proposed method is evaluated on two EEG, and one ECG dataset.", "questions": "see above", "limitations": "NA", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well organized and clearly written. Figure 2 is very informative. Related works are also comprehensive.\n2. The performance of the proposed method is decent based on the experimental tables.", "weaknesses": "1. The proposed method is not new and many multi-granular contrastive learning models have been proposed in the past few years. Based on Figure 1, CLOCS, TS2Vec, TS-TCC are also multi-granular contrastive learning methods specifically in medical timeseries domain. The contribution and extension (from existing two granular methods to the proposed four granular method) of this paper seems trivial given these exisiting methods.\n2. Overall, the proposed COMET method can be viewed as \"a more finer-granular way of doing data augmentation\". In that sense, the data augmentation methods for all models (including baselines) seem different, then does the experimental design fair?\n3. The considered two EEG and one ECG datasets all seems small in medical timeseries domain. What is the number of parameters in the model? Usually, the training data size is expected to be 10 times larger than the number of parameters. Contrastive learning methods naturally need a lot more unlabeled data to train. SHHS is a large EEG dataset, and the ECG 2020 challenge datasets are also large https://physionet.org/content/challenge-2020/1.0.2/.\n4. Although the authors have gave another example on satellite sensor applications, the generalizability of the proposed model is still questionable. Given that many multi-granular contrastive learning models have been developed in other domains as well, could the authors distinguish this paper and highlights the key features of COMET if generalizable to other domains?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your thoughtful feedback and happy that you feel our paper is well-organized and the figure is informative. The following response to the questions about our method's weaknesses.\n***\n**Q1**: The proposed method is not new since many multi-granular contrastive learning models exist.\n\n**A1**: We agree that others have implicitly utilized multi-level information, albeit within highly specific models tailored to different data types. In contrast, we introduce a general framework that applies to all types of medical time series data. We explicitly present the concept of multi-granularity in the context of contrastive learning.\n\nAfter reviewing numerous existing contrastive learning methods within the time series domain, we consistently posed a pivotal question to ourselves: Can we design a straightforward yet appliable contrastive learning framework that can be adapted to all forms of medical time series data, akin to the classical model SimCLR in the domain of contrastive learning? In contrast to general time series data, medical time series data typically exhibit two additional granularities: patient and trial. Our objective is to craft an innovative framework that utilizes all information within medical time series in a self-supervised manner. This approach enables us to harness the information of patient and trial level to learn consistency across instances. Simultaneously, we leverage the sample and observation levels to facilitate conventional instance discrimination. In COMET, users can easily toggle between different granularities or adjust their weights by setting the hyper-parameter λ, depending on the distinctive features of the medical time series.\n\nSome existing methods like TS2Vec, Clocs, TS-TCC are also multi-granularity. They can be viewed as a particular case of our model. In other words, we are not an extension of their model, but rather, in the domain of medical time series, our approach includes and is compatible with their model. Our work will play a role in summarizing, inspiring, and guiding future works in contrastive learning on medical time series.\n\n***\n**Q2**: If the data augmentation methods for all models (including baselines) seem different, then does the experimental design fair?\n\n**A2**: As mentioned in A1, our model is not an extension but rather includes and is compatible with their models within the domain of medical time series. Additionally, existing methods like TFC, TS2Vec, and TimeCLR [10] have their specific data augmentation techniques integrated as part of their methods. Therefore, their original data augmentation methods should remain unchanged when using them as baselines for comparison.\n\nIn our original submission, considering the extra contrastive blocks in our method, we did the heavy-duty case study by running extra epochs during contrastive pre-training or implementing extra contrastive blocks for baselines. (Appendix F.4)\n\n\n***\n\n**Q3**: The two EEG and one ECG datasets seem small in the medical time series domain. The training data size is expected to be 10 times larger than the number of parameters. Use SHHS or the ECG 2020 challenge datasets.\n\n**A3**: Our responses to this question contain three parts.\n\n1) Regarding the PTB dataset, our model comprises 687,552 parameters. While increasing the size of the training data to exceed the number of parameters is indeed an effective approach to counter overfitting, modern techniques such as L2 regularization, dropout, and batch normalization have proven useful in mitigating overfitting risks, enabling models to yield favorable performance even with limited data [11][12].\n\n2) Our datasets are generally not small within the scope of medical time series. The medical time series domain faces challenges in labeling due to factors like the complexity of data collection and associated expenses, including issues such as ethnic considerations [1][2]. Moreover, datasets centered around specific ailments, such as Alzheimer's disease, are rarely accessible and often not publicly available [7].\n\n3) In the original submission, we utilized the PTB dataset, part of the ECG 2020 challenge available at Physionet 2020. Additionally, we newly add a considerably larger dataset  PTB-XL (17596 patients, 191400 samples, 5 classes) within the ECG 2020 challenge. See R-table 3. Our method performs better in 11 of 12 tests than the other three SOTAs. Particularly with label fraction 1%, we outperform SOTAs by about 5% F1 score and 3% AUROC, demonstrating the effectiveness of contrastive pre-train of COMET in reducing the dependence of labeled data.\n\n***\n**Q4**: The generalizability of the proposed model?\n\n**A4**: Our response has three parts.\n\n1) As indicated in the title and abstract, COMET is uniquely tailored for medical time series, capitalizing on their distinctive characteristics. It is not a universally applicable model for all types of time series, as general time series typically only have sample and observation granularities.\n\n2) We offer an illustrative example involving satellite sensor applications, intending to inspire researchers in diverse domains. The key is to utilize all available information, excluding label data, for contrastive pre-training, such as patient ID. To adapt our approach to other domains, researchers must consider a crucial question: Does the dataset has additional information beyond sample labels? If affirmative, can this information be harnessed for contrastive learning? The example of satellite sensor application underscores the potential existence of supplementary information even in non-medical domains.\n\n3) There are also other multi-granularity papers in different domains. We review and compare them with our methods in Appendix E.2. Some papers focus on learning representations of various granularities, which differs from ours. In our case, the distinct granularities are employed for constructing positive and negative pairs during contrastive learning.\n\n\n\n\n\n"}, {"review_id": "JJcIgGuSfS", "reviewer": "Reviewer_p5Bk", "summary": "This paper investigates the contrastive learning for medical time series and develops multiple contrastive objectives at different levels (patient, trial, observation, sample). The downstream applications is the classification on EEG, ECG, EMG, EOG signals. The proposed method has obtained the best or competitive performance on two EEG datasets and one ECG dataset.", "questions": "Check the weakness part.", "limitations": "Lack of sufficient verifications on large-scale and more diversified datatsets.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- It is interesting to see a hierarchical view for a specific type of medical time series, such as ECG and EEG. This hierarchical multi-level view captures the unique characteristics of this type of time series.\n- Given this novel view, the proposed contrastive learning at all levels is straightforward and reasonable.\n- The paper is well written and easy to understand.\n", "weaknesses": "- My first concern is the discrepancy between the title and the actual methodology. \"multi-level contrastive learning\" sounds a better choice than \"multi-granularity representation learning\" because event at the patient or trial level, the representation learning still happens at the sample-level input, and you do not squeeze or aggregate representations at higher levels.Thus, I feel there is no multiple granularity. Please correct me if I misunderstand your method.\n- The second concern is that the proposed method is only evaluated on limited datasets. It seems that the method only works on EEG-related datasets (AD and TDBrain), beating other baselines significantly, but is modest on PTB, an ECG dataset. I would like to see broader empirical demonstrations and analyses.\n- Moreover, the ablation tests are only conducted on the AD dataset, on which the proposed method has achieved the most performance improvements. I would like to see more detailed ablation tests and analyses on other datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are happy that you are interested in our paper and feel our paper is well-written and easy to understand. Thank you very much for your three valuable concerns! Here we respond in detail to each of your concerns. If you do not feel we have sufficiently justified a higher score, please let us know where we can improve our work and your further concern. Thank you again!\n***\n**Q1**: My first concern is the discrepancy between the title and the actual methodology. \"multi-level contrastive learning\" sounds like a better choice than \"multi-granularity representation learning\" because even at the patient or trial level, the representation learning still happens at the sample-level input, and you do not squeeze or aggregate representations at higher levels. Thus, I feel there is no multiple granularity. Please correct me if I misunderstand your method.\n\n**A1**: Thanks for bringing up this concern. We acknowledge that we do not compress or aggregate representations at higher levels; the representation learning consistently occurs at the sample-level input.\n\nHowever, our objective is not to acquire patient or trial level representations. Instead, the various granularities serve to construct positive and negative pairs for contrastive learning. By harnessing patient and trial level information, we extend our representation learning beyond conventional instance discrimination methods. This approach facilitates capturing consistencies across samples (instances) through a self-supervised framework. In the original submission, we have taken cognizance of your concern and examined the difference between our method and multi-granularity approaches in other domains. Please refer to Appendix E.2 for a detailed analysis.\n\nMoreover, we appreciate your attention to the paper's title. We will thoroughly consider whether to modify it in the camera-ready version. Indeed, we deliberated on the paper's title extensively. Alternately, we contemplated a title such as \"Hierarchical Contrastive Learning.\" What are your thoughts on this title?\n***\n**Q2**: The second concern is that the proposed method is only evaluated on limited datasets. It seems that the method only works on EEG-related datasets (AD and TDBrain), beating other baselines significantly, but is modest on PTB, an ECG dataset. I would like to see broader empirical demonstrations and analyses.\n\n**A2**: Thank you for raising this concern. We add one more large-scale ECG dataset (PTB-XL; 17596 patients, 191400 samples, 5 classes). See R-table 3 for experiment results. Our method performs better in 11 of 12 tests than the other three SOTAs. Particularly with label fraction 1%, we outperform SOTAs by about 5% F1 score and 3% AUROC, demonstrating the effectiveness of contrastive pre-train of COMET in reducing the dependence of labeled data. \n\n***\n**Q3**: Moreover, the ablation tests are only conducted on the AD dataset, on which the proposed method has achieved the most performance improvements. I would like to see more detailed ablation tests and analyses on other datasets.\n\n**A3**: Thank you for raising this concern. We add the ablation study on the TDBrain and PTB. See R-table 1 and 2 for experiment results."}]

Author rebuttal:
We appreciate the valuable feedback provided by all the reviewers. We highly appreciate the reviewers who believe our work is solid, effective, easy to follow, and well-written, with a logical presentation of the methodological contribution of the proposed method. In response to the reviewers' insightful comments, we have conducted additional experiments, including two new ablation studies, incorporating two new baselines, and utilizing a new dataset. We addressed your concerns and suggestions in the individual responses. We will incorporate the newly added content into the final version of the paper for camera-ready submission.

Thank you again for your thoughtful comments. We work hard to refine our paper, and we sincerely hope our responses are informative and helpful. If you feel we have not sufficiently addressed your concerns to motivate increasing your score, we would love to hear from you further on what points of concern remain and how we to improve our work. Thank you again!

Due to space limitations, we have included **some responses, all the tables, and all the references** within this global rebuttal. The experiment results of the ablation study in R-tables 1 and 2 can be found in the attached PDF file. Here is R-table 3. 
***
**R-table 3: Comparison of the new large-scale dataset PTB-XL with new baseline CLOCS, NCL, and best old baseline TS2Vec.**
|   Label Fraction    |   Method    | Accuracy |  F1 |  AUROC |  AUPRC  |
| ----------- | ----------- | ----------- | ----------- | ----------- | ----------- |
| 100%  | CLOCS | 72.88±0.35 |  59.97±0.66|  88.30±0.17| 62.79±0.20|
| 100%  | NCL | 72.63±0.09 | 61.16±0.39 | 89.28±0.00 | 65.17±0.45 | 
| 100%  | TS2Vec | 72.48±0.39 | 60.50±0.20 | 89.18±0.17 | 64.84±0.58 | 
| 100%  | COMET(Ours) | **74.18±0.02** | **61.63±0.06** | **89.29±0.11** | **66.12±0.52** | 
| 10%  | CLOCS | 67.00±0.82 |  54.81±0.36 |  83.66±0.45| 55.97±0.63|
| 10%  | NCL | 68.12±0.04 | 56.62±0.39 | 86.02±0.12 | 60.17±0.47 | 
| 10%  | TS2Vec | 69.21±0.48 | 57.29±1.23 | **86.55±0.09** | 61.12±0.41 | 
| 10%  | COMET(Ours) | **70.69±0.19** | **57.80±0.23** | 86.37±0.25 | **61.78±0.10** | 
| 1%  | CLOCS | 52.16±2.35 |  36.39±2.43 |  69.68±1.85 | 35.48±2.13 |
| 1%  | NCL | 49.92±1.13 | 31.96±0.64 | 68.99±0.36 | 33.75±0.73 | 
| 1%  | TS2Vec | 54.76±0.69 | 38.93±1.66 | 74.47±0.35 | 39.60±0.09 | 
| 1%  | COMET(Ours) | **60.38±0.21** | **43.67±0.65** | **77.59±0.75** | **45.75±0.93** | 

***
## Here are some continued responses to reviewer **Trt3**'s questions.
***
**Q7** Would jittering and masking break the continuity of the normal time series data? 

**A7** We respectively clarify that we did not use jittering ( Line 196 just serves as an example). Our experiments solely used masking(Sec. 5 implementations; Appendix B data augmentation bank). As previously mentioned in A2, we carefully designed the strength of data augmentation (10% masking ratio) to maximize the preservation of features within the time series, which will not distort the data continuity. 
***

**Q8** Explain the usage of InfoNCE at observation and sample blocks while using NT-Xent loss at trial and patient levels.

**A8** We observed that TS2Vec performs effectively for observation and sample levels. Concurrently, CLOCS introduced contrastive learning on ECG data, utilizing patient-level information, which bears similarity to our implementation at the trial level. Thus, we adopted their loss functions as a foundation for designing our own loss functions across distinct granularities. Broadly speaking, these loss functions are all variants of the InfoNCE loss function.
***
**Q9** SOTA comparison: why NCL and CLOCS are not considered?

**A9** We did test running for CLOCS before submission and found it extremely time-consuming (More than 10 hrs for the PTB dataset in a single run). As a result, we chose some other SOTAs as baselines. Regarding NCL, their definition of neighborhood contrasting inspired us to design trial level contrasting. We compare with CLOCS and NCL on the newly added large-scale dataset PTB-XL. See R-table 3.
***
**Q10** Any ablation study on the hyperparameters $\lambda_i$ is missing.

**A10** We add ablation study on $\lambda_i$. See R-table 1 and 2.
***
# Reference

[1] Kiyasseh, Dani., et al. "Clocs Contrastive learning of cardiac signals across space, time, and patients." ICML, 2021.

[2] Lan, Xiang., et al. "Intra-inter subject self-supervised learning for multivariate cardiac signals."  AAAI, 2022.

[3] Banville, Hubert., et al. "Uncovering the structure of clinical EEG signals with self-supervised learning." Journal of Neural Engineering, 2021

[4] Tonekaboni., et al. "Unsupervised representation learning for time series with temporal neighborhood coding." arXiv 2021

[5] Ieracitano., et al. "A convolutional neural network approach for classification of dementia stages based on 2D-spectral representation of EEG recordings." Neurocomputing, 2019

[6] Yue, Zhihan., et al. "Ts2vec Towards universal representation of time series." AAAI, 2022.

[7] Tzimourta, Katerina D., et al. "Machine learning algorithms and statistical approaches for Alzheimer’s disease analysis based on resting-state EEG recordings A systematic review." Inter. J. of Neural Syst, 2021

[8] Zhang, Xiang, and Lina Yao. Deep Learning for EEG-Based Brain–Computer Interfaces Representations, Algorithms and Applications. 2021.

[9] Kalantidis, Yannis., et al. "Hard negative mixing for contrastive learning." Neurips, 2020

[10] Xinyu Yang., et al. “A self-supervised contrastive learning framework for univariate time series representation.” Knowledge-Based Systems, 2022. 

[11] Machine Learning Mastery. "Introduction to Regularization to Reduce Overfitting and Improve Generalization Error."  https://machinelearningmastery.com/introduction-to-regularization-to-reduce-overfitting-and-improve-generalization-error/ 

[12] TensorFlow. "Overfit and Underfit." https://www.tensorflow.org/tutorials/keras/overfit_and_underfit 

Meta-review: Overall, the authors provided very detailed responses to address reviewer concerns in a satisfactory manner, which also involved revising the paper to significantly improve it during the author-reviewer discussion period (with extensive clarifications and also more experimental results including more baselines). Three out of the four reviewers at this point lean on acceptance of the paper including one favoring strong acceptance. The only dissenting reviewer voted for a borderline reject but it seems like the authors have sufficiently addressed this reviewer's concerns.
Decision: Accept (poster)