SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yihe Wang *  
University of North Carolina - Charlotte  
ywang145@uncc.edu
Yu Han*  
University of Chinese Academy of Sciences  
hanyu21@mails.ucas.ac.cn
Haishuai Wang  
Zhejiang University  
haishuai.wang@zju.edu.cn
Xiang Zhang  
University of North Carolina - Charlotte  
xiang.zhang@uncc.edu
Contrastive representation learning is crucial in medical time series analysis as it alleviates dependency on labor-intensive, domain-specific, and scarce expert annotations. However, existing contrastive learning methods primarily focus on one single data level, which fails to fully exploit the intricate nature of medical time series. To address this issue, we present COMET, an innovative hierarchical framework that leverages data consistencies at all inherent levels in medical time series. Our meticulously designed model systematically captures data consistency from four potential levels: observation, sample, trial, and patient levels. By developing contrastive loss at multiple levels, we can learn effective representations that preserve comprehensive data consistency, maximizing information utilization in a self-supervised manner. We conduct experiments in the challenging patient-independent setting. We compare COMET against six baselines using three diverse datasets, which include ECG signals for myocardial infarction and EEG signals for Alzheimer's and Parkinson's diseases. The results demonstrate that COMET consistently outperforms all baselines, particularly in setup with  $10\%$  and  $1\%$  labeled data fractions across all datasets. These results underscore the significant impact of our framework in advancing contrastive representation learning techniques for medical time series. The source code is available at https://github.com/DL4mHealth/COMET.
Time series data is crucial in various real-world applications, ranging from finance [1, 2, 3], engineering [4, 5], to healthcare [6, 7]. Unlike domains such as computer vision [8, 9] and natural language processing [10, 11], where human recognizable features exist, time series data often lacks readily discernible patterns, making data labeling challenging. Consequently, the scarcity of labeled data poses a significant hurdle in effectively utilizing time series for analysis and classification tasks.
To address the paucity of labeled data in time series analysis, self-supervised contrastive learning has emerged as a promising approach. For example, TimeCLR [12] proposes a DTW data augmentation for time series data; TS2vec [13] designs a cropping and masking mechanism to form positive pairs; ExpCLR [14] introduces a novel loss function to utilize continuous expert features. By leveraging the inherent consistency within unlabeled data, contrastive learning algorithms enable the extraction of effective representations without relying on explicit labels. This paradigm shift opens up possibilities for overcoming the data scarcity issue and enhancing the capabilities of time series analysis.
Despite recent advancements in contrastive learning methods for time series, existing approaches fail to exploit the full potential of medical time series data, such as electroencephalogram (EEG) signals. Unlike conventional time series data, medical time series often exhibit more data levels (Figure 1), including patient, trial, sample, and observation levels. Current contrastive learning techniques exclusively employ subsets of these levels (as illustrated in Table 1). Additionally, many of these methods are tailored to specific data types, which restricts their capacity to capture the rich complexity of medical time series. For example, CLOCS [15] presents a contrastive learning method for ECG using sample and patient levels. Mixing-up [16] captures sample-level consistency through a mixing data augmentation scheme. TNC [17] exploits trial-level consistency by contrasting neighbor samples in the same trial as positive pairs. Neither of them leverages all the levels exhibited in the medical time series.
After reviewing existing contrastive learning methods within the time series domain, we consistently posed a pivotal question to ourselves: Can we design a straightforward yet applicable contrastive learning framework that can be adapted to all forms of medical time series data, akin to the classical model SimCLR [18] in the domain of contrastive learning? Our objective is to craft an innovative framework that utilizes all information within medical time series in the context of self-supervised contrastive learning. It enables us to harness patient and trial information to learn consistency across instances while leveraging the sample and observation levels' information to facilitate conventional instance discrimination.
In this paper, we propose a hierarchical framework, COMET, that systematically leverages all four levels of medical time series, namely patient, trial, sample, and observation, to reduce the reliance on labeled data. By incorporating self-supervised contrastive learning, our method aims to bridge the gap between the limited availability of labeled data and the need for robust and generalizable models in medical time series analysis. We conduct extensive experiments with six baselines on three diverse datasets in a challenging patient-independent setting. COMET outperforms SOTAs by  $14\%$  and  $13\%$  F1 score with la
Figure 1: Structure of medical time series. Medical time series commonly have four levels (coarse to fine): patient, trial, sample, and observation. An observation is a single value in univariate time series and a vector in multivariate time series.
label fractions of  $10\%$  and  $1\%$ , respectively, on EEG-based Alzheimer's detection. Further, COMET outperforms SOTAs by  $0.17\%$  and  $2.66\%$  F1 score with label fractions of  $10\%$  and  $1\%$ , respectively, on detecting Myocardial infarction with ECG. Finally, COMET outperforms SOTAs by  $2\%$  and  $8\%$  F1 score with label fractions of  $10\%$  and  $1\%$ , respectively, in the EEG-based diagnosis of Parkinson's disease. The results of downstream tasks demonstrate the effectiveness and stability of our method.
Medical time series. Medical time series [19, 20, 21, 22] is a distinct type of time series data used for healthcare (e.g., disease diagnosis, monitoring, and rehabilitation). It can be collected in a low-cost, non-invasive manner [23] or a high-cost, invasive manner [24].
Unlike general time series, which typically consist of sample and observation levels, medical time series introduces two additional data levels: patient and trial. These extra levels of data information in medical time series enable the development of specialized methods tailored to address the unique characteristics and requirements of medical time series analysis. Various types of medical time series, including EEG [28, 29, 30], ECG [31, 32], EMG [33], and EOG [34], offer valuable insights into spe
Table 1: Existing methods only utilize partial levels.
Models	Patient	Trial	Sample	Observation
SimCLR [25]	✓
TF-C [26]	✓
Mixing-up [16]	✓
TNC [17]	✓
TS2vec [13]	✓	✓
TS-TCC [27]	✓	✓
CLOCS [15]	✓	✓
COMET(Ours)	✓	✓	✓	✓
cific medical conditions and play a crucial role in advancing healthcare practices.
Contrastive learning for time series. Contrastive learning has demonstrated its ability to learn effective representations in various domains, including image processing [35, 18, 36], graph analysis [37, 38, 37], and time series [39]. The key idea behind contrastive representation learning is to mine data consistency by bringing similar data closer together and pushing dissimilar data further apart.
Many existing works on contrastive learning focus on general time series, while some are designed specifically for medical data [40]. One classic framework, SimCLR, transforms a single sample into two augmented views and performs contrastive learning [25]. Other settings, such as using sub-series and overlapping-series, leverage sample-level consistency [41]. TF-C [26] contrasts representations learned from the time and frequency domains to exploit sample-level consistency. Mixing-up [16] learn sample-level consistency by utilizing a mixing component as a data augmentation scheme. TS-TCC and TS2vec [27, 13] apply data augmentation at the sample-level and perform contrastive learning at the observation-level. TNC [17] learns trial-level consistency by contrasting neighboring and non-neighboring samples. NCL [42] can also be used to learn trial-level consistency if we define samples from a trial as a neighborhood. CLOCS [15] learns patient-level consistency in cardiac signal features by contrasting different leads over time.
Certain prior methods have implicitly utilized hierarchical structure [13, 15]. However, as shown in Table 1, none of these methods leverage all the levels present in the medical time series, potentially resulting in the loss of useful information during training. We explicitly present a hierarchical framework in the context of contrastive learning, which can be applied across diverse types of medical time series data. In our paper, we aim to leverage data consistency at all levels in medical time series. Our work plays a role in summarizing, inspiring, and guiding future works in self-supervised contrastive learning on medical time series.
3 Preliminaries and Problem Formulation
3.1 Medical Time Series
In this section, we clarify the key conceptions of observation (or measurement), sample (or segment), trial (or recording), and patient (or subject) in the context of medical time series (Figure 1). For better understanding, we illustrate the concepts with an example of Electroencephalography (EEG) signals for Alzheimer's Disease diagnosis (details in Appendix A).
Definition 1: Observation. An observation  $\boldsymbol{x}_{i,t} \in \mathbb{R}^{F}$  in medical time series data represents a single data point or a vector captured at a specific timestamp  $t$ . Here we use  $i$  to denote the sample index (see Definition 2) and  $t$  to denote the timestamp. It may record physiological status, laboratory test results, vital signs, or other measurable health indicators. The observation is a single real value for univariate time series while vector for multivariate time series. The  $F$  is the feature dimension if it is a multivariate time series.
Definition 2: Sample. A sample  $\boldsymbol{x}_i = \{\boldsymbol{x}_{i,t}|t = 1,\dots ,T\}$  is a sequence of consecutive observations, typically measured at regular intervals over a specified period ( $T$  timestamps). It can also be called a segment or window. Here we use  $i$  to denote the sample index. In the medical time series, a sample might consist of a sequence of heart rate measurements or blood pressure readings.
Definition 3: Trial. A trial  $r_i$  is a collection of consecutive samples. It can also be called a record. Here we use  $i$  to denote the trial ID. In medical time series, a trial is a continuous set of observations collected over a not-short period (e.g., 30 minutes). Therefore, a trial is generally too long (e.g., hundreds of thousands of observations) to feed into deep learning models for representation learning directly and is usually split into shorter subsequences (i.e., samples/segments). To represent the aggregate of samples stemming from a particular trial  $r_i$  with trial ID  $i$ , we employ the notation  $\mathcal{R}_i$ .
Definition 4: Patient. A patient  $\pmb{p_i}$  represents a collection of multiple trials stemming from a single patient. It can also be called a subject. Here we use  $i$  to denote the patient ID. It is important to note that trials for a given patient may exhibit variations due to differing data collection timeframes, sensor placements, patient conditions, and other contributing factors. As shown in Definition 3, a trial is typically divided into many samples for better representation learning. In practical scenarios, a patient, which constitutes a cluster of trials, is also divided into samples that may share identical or distinct trial IDs but maintain the same patient ID. To represent the aggregate of samples stemming from a particular patient  $\pmb{p_i}$  with the corresponding patient ID  $i$ , we employ the notation  $\mathcal{P}_i$ .
Figure 2: Overview of COMET approach. Our COMET model consists of four contrastive blocks, each illustrating the formulation of positive pairs and negative pairs at different data levels. In the observation-level contrastive, an observation  $\boldsymbol{x}_{i,t}$  and its augmented view  $\widetilde{\boldsymbol{x}}_{i,t}$  serve as a positive pair. Similarly, in the sample-level contrastive, a sample  $\boldsymbol{x}_i$  and its augmented view  $\widetilde{\boldsymbol{x}}_i$  form a positive pair. Moving to the trial-level contrastive, two samples  $\boldsymbol{x}$  and  $\boldsymbol{x}^{+}$  from the same trial  $\boldsymbol{r}_i$  are considered to be a positive pair. The patient-level contrastive follows a similar pattern, where two samples  $\boldsymbol{x}$  and  $\boldsymbol{x}^{+}$  from the same patient  $\boldsymbol{p}_i$  are regarded as a positive pair. Positive and corresponding negative pairs will be utilized to build contrastive loss in embedding space after being processed by encoder  $G$ .
In this work, we propose a novel hierarchical contrastive framework to learn representative and generalizable embeddings by comprehensively exploring instance discrimination at observation and sample levels and harnessing consistency across instances at trial and patient levels. Although we elaborate the proposed COMET in the context of medical time series, we note our model can possibly be extended to other time series beyond healthcare as long as extra information is available. For example, a climate dataset contains multiple meteorological satellites, each satellite contains multiple measuring units, and each unit contains multiple sensors, and every sensor can measure a specific observation at a certain timestamp. The key is to utilize all available information, excluding label data, for contrastive pre-training, such as patient ID. To adapt our approach to other domains, researchers must consider a crucial question: Does the dataset have additional information beyond sample labels? If affirmative, can this information be harnessed for contrastive learning? The example of satellite sensor application underscores the potential existence of supplementary information even in non-medical domains.
3.2 Problem Formulation
Problem (Self-Supervised Representation Learning For Medical Time Series). Let an unlabeled dataset  $\mathcal{D}$  consist of a set of patients, where each patient  $p_i$  has multiple trials, each trial  $\boldsymbol{r}_i$  can be segmented into many samples, and each sample  $\boldsymbol{x}_i$  comprises a series of observations. We aim to pre-train an encoder  $G$  that exploits data consistency at all available levels in a self-supervised contrastive manner. For a given time series sample  $\boldsymbol{x}_i \in \mathbb{R}^{T \times F}$  with  $T$  timestamps and  $F$  feature dimensions, the encoder  $G$  learns a sample-level representation  $\boldsymbol{h}_i \in \mathbb{R}^{T \times K}$ , where  $\boldsymbol{h}_{i,t} \in \mathbb{R}^K$  is the observation-level representation at timestamp  $t$  with  $K$  dimensions.
By exploiting hierarchical consistency at multiple data levels, we aim to learn a representation  $\pmb{h}_i$  that is both representative (yielding good performance in downstream tasks) and generalizable (maintaining stability across different patients). Depending on the fine-tuning settings [18], a specific fraction of labels  $y_i$  corresponding to samples  $\pmb{x}_i$  are necessary.
In this section, we first present our assumption of data consistency behind designing a hierarchical contrastive framework. Then, we describe the architecture of the proposed model COMET (Figure 2).
4.1 Hierarchical Data Consistency
Capturing data consistency is crucial in the development of a contrastive learning framework [13]. Data consistency refers to the shared commonalities preserved within the data, which provide a supervisory signal to guide model optimization. Contrastive learning captures data consistency by contrasting positive and negative data pairs, where positive pairs share commonalities and negative pairs do not. We propose consistency across four data levels: observation, sample, trial, and patient, from fine-grained to coarse-grained in the medical time series. Although we present four levels here, our model can easily be adapted to accommodate specific datasets by adding or removing data levels.
Observation-level data consistency. We assume a slightly augmented observation (e.g., channel masked) will carry similar information as the original observation [13]. We use  $\boldsymbol{x}_{i,t}$  as the anchor observation at timestamp  $t$ , and  $\boldsymbol{x}_{i,t^{-}}$  as the observation at another timestamp  $t^{-}$  in the sample  $\boldsymbol{x}_i$ . We consider the anchor observation  $\boldsymbol{x}_{i,t}$  and an augmented observation  $\widetilde{\boldsymbol{x}}_{i,t}$  as positive pairs  $(\boldsymbol{x}_{i,t},\widetilde{\boldsymbol{x}}_{i,t})$  (with closer embeddings). Conversely, we consider the original observation  $\boldsymbol{x}_{i,t}$  and the observations  $\widetilde{\boldsymbol{x}}_{i,t^{-}}$  and  $\boldsymbol{x}_{i,t^{-}}$  at another timestamp  $t^{-}$  as negative pairs  $(\boldsymbol{x}_{i,t},\widetilde{\boldsymbol{x}}_{i,t^{-}}),(\boldsymbol{x}_{i,t},\boldsymbol{x}_{i,t^{-}})$ , with distant embeddings.
Sample-level data consistency. The sample-level consistency is based on our assumption that a slightly perturbed sample (e.g., temporally masked) should carry similar information as the original sample [18, 16, 26]. We consider the anchor sample  $\pmb{x}_i$  and its augmented view  $\widetilde{\pmb{x}}_i$  as positive pair  $(\pmb{x}_i, \widetilde{\pmb{x}}_i)$ . We regard the anchor sample  $\pmb{x}_i$  and a different sample  $\pmb{x}_j$  and its augmented view  $\widetilde{\pmb{x}}_j$  as negative pairs:  $(\pmb{x}_i, \widetilde{\pmb{x}}_j)$  and  $(\pmb{x}_i, \pmb{x}_j)$ .
Trial-level data consistency. We assume that samples sliced from the same trial should carry similar information compared to those obtained from different trials. For simplicity, we use  $\pmb{x}$  to denote the anchor sample and  $\pmb{x}^{+}$  to denote a sample from the same trial  $\pmb{r}_i$  as the anchor sample, while  $\pmb{x}^{-}$  to denote a sample from another trial  $\pmb{r}_j$ . In other words, we have  $\{\pmb{x},\pmb{x}^{+}\} \in \mathcal{R}_{i}$  and  $\pmb{x}^{-} \in \mathcal{R}_{j}$ . We treat sample  $\pmb{x}$  and the sample  $\pmb{x}^{+}$  from the same trial as positive pair  $(\pmb{x},\pmb{x}^{+})$ . We regard sample  $\pmb{x}$  and the sample  $\pmb{x}^{-}$  from different trials as negative pair  $(\pmb{x},\pmb{x}^{-})$ .
Patient-level data consistency. We assume samples originating from the same patient are likely to contain similar information when compared to those from different patients [15]. Here, we use  $\pmb{x}$  to denote the anchor sample and  $\pmb{x}^{+}$  to denote a sample from the same patient  $\pmb{p}_i$ , while  $\pmb{x}^{-}$  from another patient  $\pmb{p}_j$ . In other words, there are  $\{\pmb{x},\pmb{x}^{+}\} \in \mathcal{P}_{i}$  and  $\pmb{x}^{-} \in \mathcal{P}_{j}$ . We have positive pair  $(\pmb{x},\pmb{x}^{+})$  including samples from the same patient and negative pair  $(\pmb{x},\pmb{x}^{-})$  that from different patients.
Disease-level data consistency. For completeness, we introduce disease-level data consistency, which suggests that samples associated with the same type of disease should exhibit shared patterns, even when collected from different patients in different ways. However, capturing disease-level consistency requires ground truth labels, which are not available in a self-supervised approach. As a result, we do NOT employ disease-level consistency in this paper. Nevertheless, it can be adapted for semi-supervised or supervised contrastive learning and may prove beneficial in learning domain-adaptable representations for certain diseases across patients and even datasets.
A common principle underlying all definitions is that the  $X$ -level data consistency refers to the positive pair belonging to the same  $X$ , where  $X$  could be observation, sample, trial, patient, or disease. We assume that each patient is associated with only one label, such as suffering from a specific disease, which implies that all samples from the same patient essentially originate from the same distribution. However, in cases where data from a patient is derived from multiple distributions (e.g., a patient could perform various daily activities; associated with multiple labels), the assumptions of trial-level and patient-level consistency are not satisfied. Therefore, the user can switch on the observation-level and sample-level consistency.
Building upon the concepts of data consistency, we introduce four contrastive blocks corresponding to the four data levels. Our model is highly flexible, allowing users to enable or disable any of the blocks based on the requirements of a specific task or dataset.
4.2 Observation-Level Contrastive Block
For a given time series sample  $\pmb{x}_i$ , we apply data augmentation (such as masking) to generate an augmented sample  $\widetilde{\pmb{x}}_i$  [25, 43]. We input the original sample  $\pmb{x}_i$  and its augmented view  $\widetilde{\pmb{x}}_i$  into
contrastive encoder  $G$  to obtain their respective representations  $\pmb{h}_i = G(\pmb{x}_i)$  and  $\widetilde{\pmb{h}}_i = G(\widetilde{\pmb{x}}_i)$ . It is important to note that we apply data augmentation to the samples, which indirectly extends to augmenting the observations, simplifying the encoding process. To capture observation-level consistency, we assume that, after being processed by encoder  $G$ , the representation of observation  $\pmb{x}_{i,t}$  is close to the representation of the augmented observation  $\widetilde{\pmb{x}}_{i,t}$ . In contrast, it should be distant from the representations of observations  $\pmb{x}_{i,t-}$  and  $\widetilde{\pmb{x}}_{i,t-}$  originating from any other timestamp  $t^-$ . Specifically, our positive pair is  $(\pmb{x}_{i,t},\widetilde{\pmb{x}}_{i,t})$  and negative pairs are  $(\pmb{x}_{i,t},\pmb{x}_{i,t-})$  and  $(\pmb{x}_{i,t},\widetilde{\pmb{x}}_{i,t-})$ .
Observation-level contrastive loss. The observation-level contrastive loss  $\mathcal{L}_{\mathrm{O}}$  [13] for the input sample  $x_{i}$  is defined as:
$$
\mathcal {L} _ {\mathrm {O}} = \mathbb {E} _ {\boldsymbol {x} _ {i} \in \mathcal {D}} \left[ \mathbb {E} _ {t \in \mathcal {T}} \left[ - \log \frac {\exp \left(\boldsymbol {h} _ {i , t} \cdot \widetilde {\boldsymbol {h}} _ {i , t}\right)}{\sum_ {t ^ {-} \in \mathcal {T}} \left(\exp \left(\boldsymbol {h} _ {i , t} \cdot \widetilde {\boldsymbol {h}} _ {i , t ^ {-}}\right) + \mathbb {1} _ {[ t \neq t ^ {-} ]} \exp \left(\boldsymbol {h} _ {i , t} \cdot \boldsymbol {h} _ {i , t ^ {-}}\right)\right)} \right] \right] \tag {1}
$$
where  $\mathcal{T} = \{1,\dots ,T\}$  is the set of all timestamps in sample  $\pmb{x}_i$  and  $\cdot$  denotes dot product. The  $\mathbb{1}_{[t\neq t^{-}]}$  is an indicator function that equals to 0 when  $t = t^{-}$  and 1 otherwise.
4.3 Sample-Level Contrastive Block
For an input time series sample  $\boldsymbol{x}_i$  and its augmented view  $\widetilde{\boldsymbol{x}}_i$ , we calculate their representations through  $\boldsymbol{h}_i = G(\boldsymbol{x}_i)$  and  $\widetilde{\boldsymbol{h}}_i = G(\widetilde{\boldsymbol{x}}_i)$ . The augmentation applied here could be the same as or different from the augmentation used in Section 4.2. We assume that after passing through the encoder  $G$ , the representation of the sample  $\boldsymbol{x}_i$  is close to the representation of its augmented view  $\widetilde{\boldsymbol{x}}_i$ , while far away from the representations of any other samples  $\boldsymbol{x}_j$  and  $\widetilde{\boldsymbol{x}}_j$ . In specific, our positive pair is  $(\boldsymbol{x}_i, \widetilde{\boldsymbol{x}}_i)$ , and negative pairs are  $(\boldsymbol{x}_i, \widetilde{\boldsymbol{x}}_j)$  and  $(\boldsymbol{x}_i, \boldsymbol{x}_j)$ .