CLOCS: Contrastive Learning of Cardiac Signals Across Space, Time, and Patients
Authors: Dani Kiyasseh
Tingting Zhu
David A. Clifton
Venue:  (2021)
Keywords: Contrastive learning
physiological signals
healthcare

Abstract: The healthcare industry generates troves of unlabelled physiological data. This data can be exploited via contrastive learning, a self-supervised pre-training method that encourages representations of instances to be similar to one another. We propose a family of contrastive learning methods, CLOCS, that encourages representations across space, time, \textit{and} patients to be similar to one another. We show that CLOCS consistently outperforms the state-of-the-art methods, BYOL and SimCLR, when performing a linear evaluation of, and fine-tuning on, downstream tasks. We also show that CLOCS achieves strong generalization performance with only 25\% of labelled training data. Furthermore, our training procedure naturally generates patient-specific representations that can be used to quantify patient-similarity. 

Under review as a conference paper at ICLR 2021
CLOCS: CONTRASTIVE LEARNING OF CARDIAC SIG- NALS ACROSS SPACE, TIME, AND PATIENTS
Anonymous authors Paper under double-blind review
ABSTRACT
The healthcare industry generates troves of unlabelled physiological data. This data can be exploited via contrastive learning, a self-supervised pre-training method that encourages representations of instances to be similar to one another. We propose a family of contrastive learning methods, CLOCC, that encourages representations across space, time, and patients to be similar to one another. We show that CLOCS consistently outperforms the state-of-the- methods, BYOL and SimCLR, when performing linear evaluation of, and fine-tuning on, downstream tasks. We also show that CLOCS achieves strong generalization performance with only 25% labelled training data. Furthermore, our training procedure naturally generates patient-speciff representations that can be used to quantify patient-similarity.
INTRODUCTION
At present, the healthcare system unable to sufficiently leverage the large, unlabelled datasets that it generates on a daily basis. This is partially due to the dependence of deep learning algorithms on high quality labels for good generalization performance. However, arriving at such high quality labels in clinical setting where physicians are squeezed for time and attention is increasingly difficult. To overcome such an obstacle, supervuuer techniques have emerged as promising methods. These methods exploit the unlabelled dataset to formulate pretext tasks such predicting the rotation of images (Gidaris et al. 2018), their corresponding colourmap (Larsson et al., 2017). and the arrow of time (Wei al. 2018). More recently, contrastive learning was introduced as way learn representations of instances that share some context. By capturing this high-level shared context (e.g., medical diagnosis), representations become invariant the differences (e.g., input modalities) between the instances.
Contrastive learning can be characterized by three main components: 1) positive and negative of examples, 2) set of transformation operators, and 3) a variant of the noise contrastive estimation loss. Most research this domain has focused on curating a positive set of examples by exploiting data temporality (Oord et al. 2018). data augmentations (Chen et al. (2000), and multiple views of the same data instance (Tian et al. 2019). These methods are predominantly catered the image-domain and central to their implementation is the notion that shared context arises from the same instance. We believe this precludes their applicability to the medical domain where physiological time-series are plentiful. Moreover, their interpretation of shared context is limited to data from a common source where that source the individual data instance. In medicine, however, shared context can occur at higher level, the patient level. This idea is central to contributions and will encourage the development of representations that are patient-specific. Such representations have the potential to he used in tasks that exploit patient similarity such as disease subgroup clustering and discovery. As result of the process, medical practitioners may receive more interpretable outputs from networks.
In this work, we leverage electrocardiogram (ECG) signals to learn patient-specific representations in self-supervised manner via contrastive learning. To do so, we exploit the fact that ECG signals summarize both temporal and spatial information. The latter can be understood in terms projections of the same electrical signal onto multiple axes, also known as leads.
Contributions. Our contributions are the following:
1. We propose a family of patient-speciff contrastive learning methods, entitled CLOCS, that exploit both temporal and spatial information present within ECG signals.

---

Under review as a conference paper at ICLR 2021
2.
We show that CLOCS outperforms -ft----ffhhhaaa-- methods, BYOL and SimCLR, when perform- ing a linear evaluation of, and fine-tuning on, downstream tasks involving cardiac arrhythmia classification.
RELATED WORK
Contrastive Learning. In contrastive predictive coding, Oord et al. (2018) use representations of current segments to predict those of future segments. More recently, Tian et al. (2019) propose contrastive multi-view coding where multiple views of the same image are treated as 'shared context'. He et al. (2019); Chen al. (2020); Grill et al. (2020) exploit the idea of instance discrimination (Wu et al. [2188) and interpret multiple views as stochastically augmented forms of the same instance. They explore the benefit of sequential data augmentations and show that cropping and colour distortions the most important. These augmentations, however, do not trivially extend to the time-series domain. Shen et al. (2020) propose to create mixtures of images to smoothen the output distribution and thus prevent the model from being overly confident. Time Contrastive Learning (Hyvarinen & Moriokal 2016) performs contrastive learning over temporal segments in signal and illustrate the relationship between their approach and ICA. In contrast to our work, they formulate their task as prediction of the segment index within signal and perform limited experiments that do not exploit the noise contrastive estimation (NCE) loss. [Bachman et al. (2019) Time Contrastive Networks (Sermanet) jet al. 2017) attempt to learn commonalities across views and differences across time. contrast, our work focuses identifying commonalities across both spatial and temporal components of data.
Self-Supervision for Medical Time-Series. Miotto et al. (2016) propose DeepPatient, a 3-layer stacked denoising autoencoder that attempts to learn patient representation using electronic health record (EHR) data. Although performed on a large proprietary dataset, their approach focused on EHRs and does not explore contrastive learning for physiological signals. [Sarkar & Etemad (2020) apply existing self-supervised methods on ECG recordings in the context of affective computing. The methods implemented include defining pretext classification tasks such temporal inversion, negation, time-warping, etc. Their work is limited to affective computing, does not explore contrastive learning, and does not exploit multi-lead data as we do. Lyu al. (2018) explore a sequence to sequence model to learn representations from EHR data in the eICU dataset. In the process, they minimize the reconstruction error of the input time-series. Li 2020) leverage the aforementioned unsupervised learning technique on large clinical dataset, CPRD, to obtain uncertainty estimates for predictions.
3 BACKGROUND
3.1 CONTRASTIVE LEARNING
Assume the presence of a learner fo E ERD + h RE parameterized by 0, which maps D D-dimensional input, x, to an E-dimensional representation, h. Further assume the presence of an unlabelled dataset, X E IR NxD where N is the total number of instances.
Each unlabelled instance, x X, is exposed to a set of transformations, TA and TB, such that z = (x*)) and z' = (x*x).. Such transformations can consist of two different data augmentation procedures such as random cropping and flipping. These transformed instances now belong to an augmented dataset, X' E IR NxDxV, where V is equal to the number of applied transformations. In contrastive learning, representations, h² = fo(x',) and hi = So (=()), are said to share context. As result of this shared context, these representations constitute positive pair because (a) they are derived from the same original instance, x², and (b) the transformations applied to the original instance were class- paeseerresrvvng.. Representations within positive pair encouraged to be similar to another and dissimilar to representations of all other instances, h hig Vj + i. The similarity of these representations, s(h',, hig), ) is quantified via metric. s, such as cosine similarity. By encouraging high similarity between representations in the positive pair, the goal is to learn representations that invariant different transformations of the same instance.
2

---

Under review as a conference paper at ICLR 2021
4 METHODS
4.1 POSITIVE AND NEGATIVE PAIRS OF REPRESENTATIONS
Representations that derived from the same instance are typically assumed to share context. This approach, however, fails to capture commonalities present across instances. In the medical domain, for example, multiple physiological recordings from the same patient may share context. is important to note that if the multitude of physiological recordings associated with a patient were collected over large time-scales (e.g., on the order of years) and in drastically different scenarios (e.g., at rest vs. during a stress test), then the shared context across these recordings is likely to diminish. This could be due to changing patient demographies and disease profiles. With the previous caveat in mind, we propose to leverage commonalities present in multiple physiological recordings by redefining positive pair to refer to representations of transformed instances that belong to the same patient. We outline how to arrive at these transformed instances next.
4.2 TRANSFORMATION OPERATORS
When choosing the transformation operators, T. that are applied to each instance, the principal desideratum is that they capture invariances in the ECG recording. Motivated by the observation that ECG recordings reflect both temporal and spatial information, we propose to exploit both temporal and spatial invariance. We provide an intuition for such invariances in Fig. 1
Temporal Invariance
Spatial Invariance
Hexiaxial Diagram
S saconds
S seconds
Class 1
Class
Lead
+50
aVR
S/2 spconds
$/2 seconds
Class
Class 1
Class
150
Lead
Figure 1: ECG recordings reflect both temporal and spatial information. This is because they measure the electrical activity of the heart using different leads (views) over time. Temporal Invariance. Abrupt changes to the ECG recording are unlikely to occur on the order of seconds, and therefore adjacent segments of shorter duration will continue to share context. Spatial Invariance. Recordings from different leads (at the same time) will reflect the same cardiac function, and thus share context.
As is pertains to temporal invariance (Fig. D leftt, we assume that upon splitting an ECG recording, associated with Class 1, into several segments, each of them remain associated with Class 1. We justify this assumption based on human physiology where abrupt changes in cardiac function (on the order of seconds) are unlikely to occur. If these segments were collected years apart, for example, our assumption may longer hold. As for spatial invariance (Fig. [ right), we leverage the hexiaxial diagram which illustrates the location of the leads relative to the heart. We assume that temporally- aligned ECG recordings from different leads (views) are associated with the same class. This based on the idea that multiple leads (collected at the same time) will reflect the same underlying cardiac function. Occasionally, this assumption may not hold, for example, a cardiac condition afflicts a specific part of the heart, making it detectable by only few leads. We now describe how to exploit these invariances for contrastive learning.
Contrastive Multi-segment Coding (CMSC). Given an ECG recording, æ², with duration S seconds, we can extract V non-overlapping temporal segments, each with duration S/V seconds. If V 2, for example, _ T: (x') and x1₂ T=² where t indicates the timestamp of the temporal segment (see Fig. [eft). We exploit temporal invariances in the ECG by defining representations of these adjacent and non-overlapping temporal segments as positive pairs.
Contrastive Multi-lead Coding (CMLC). Different projections of the same electrical signal eman- ating from the heart characterized by different leads, L. For example, with two leads, L1 and L2, then æ'1 = Tra (x*) and x'y2 (x*) (see Fig. [ right). We exploit spatial invariances in the ECG by defining semporally-aligned representations of these different projections as positive pairs.
3

---

Under review as a conference paper at ICLR 2021
Contrastive Multi-segment Multi-lead Coding (CMSMLC), We simultaneously exploit both tem- poral and spatial invariances in the ECG by defining representations of non-overlapping temporal segments and different projections positive pairs. For example. in the presence of two temporal segments with timestamps, 11 and f2, that belong to two leads, L1 and 7.2, then o #1,L1 = Ta,, (x*) and 1,2 (2)".
4.3
PATIENT- NOISE CONTRASTIVE ESTIMATION LOss
Given our patient-centrie definition of positive pairs, we propose to optimize patient-specific noise contrastive estimation loss. More formally, Given a bini-batch of K instances, we apply a pair of transformation operators and generate 2 transformed instances (a subset of which is shown in Fig. 2 We encourage a pair of representations, h1 and 3. i, P, from the same patient, P, to be similar to another and dissimilar to representations from other patients. We quantify this similarity using the cosine similarity, s. with a temperature scaling parameter, т. (see Eq. 4 as is performed in (Tian et al. 2019; Chen et al. 2020).. We extend this to all representations in the mini-batch to form a similarity matrix of dimension
x K. In this matrix, we identify positive pairs by associating each instance with its patient ID. By design, this includes the diagonal elements and results in the loss shown in Eq. 2 If the same patient reappears within the mini-batch, then we also consider off-diagonal elements, resulting in the loss shown in Eq. 3 The frequency of these off-diagonals is inconsistent due to the random shuffling of data. We optimize the objective function in Eq. for all
pairwise combinations of transformation operators, TA and TB, where we include Eq. and Eq. twice to consider negative pairs in both views.
L = ETATT Cha.k
+
Cha.ha ding
+ - Loss-
(1)
diog
= -REP log
E, (+ Mipe
(2)
fo(m')) fo(x') 1
hh Lotf Ei,keP log E, cn(hitntit))
(3) Ahh
Pairwise Similarity Matrices
CMSC
CMLC
CMSMLC
Segment 2- Lead
Segment Lead aVR
Segment Lead avR
Segment Lead II
50
K
1
2
50
K
50
K
2
50
50
50
K
K
Figure 2: Similarity matrix for mini-batch of K instances in (Left) Contrastive Multi-segment Coding. (Centre) Contrastive Multi-lead Coding, and (Right) Contrastive Multi-segment Multi- lead Coding. Additional matrices would be generated based on all pairs of applied transformation operators, TA and Tn- Exemplar transformed ECG instances are illustrated along the edges. To identify positive pairs, we associate each instance with its patient ID. By design, diagonal elements (green) correspond to the same patient, contributing to Eq. 2 Similarly, instances and 50 (yellow) belong to the same patient, contributing to Eq. 3 The blue area corresponds to negative examples as they pertain to instances from different patients.
4

---

Under review as a conference paper at ICLR 2021
5 EXPERIMENTAL DESIGN
5.1 DATASETS
We conduct our experiments using PyTorch (Paszke al. 2019) four ECG datasets that include
cardiac arrhythmia labels. PhysioNet 2020 (Perez Alday et al. 2020) consists of 12-lead ECG recordings from 6,877 patients alongside different classes of cardiac arrhythmia. Each recording
can be associated with multiple labels. Chapman (Zheng et al., 2020) consists of 12-lead ECG recordings from 10,646 patients alongside 11 different classes of cardiac arrhythmia. As is suggested by /Zheng et al. (020),, we group these labels into 4 major classes. PhysioNet 2017 (Clifford al. 2001)) consists of 8,528 single-lead ECG recordings alongside 4 different classes. Cardiology (Hannun et al. 2019) consists of single-lead ECG recordings from 328 patients alongside 12 different classes of cardiac arrhythmia. An in-depth description of these datasets can be found Appendix A
All datasets were split into training, validation, and test sets according to patient ID using 60, 20, 20 configuration. In other words, patients appeared in only one of the sets. The exact number of instances used during self-supervised pre-training and supervised training can be found in Appendix A.2
5.2
IMPLEMENTATION
We conduct our pre- pre-training experiments on the training set of two of the four datasets: PhysioNet 2020 and Chapman. We chose these datasets as they contain multi-lead data. In CMSC, we extract pair of non-overlapping temporal segments of S = 2500 samples. This is equivalent to either 10 or seconds worth of ECG data from the Chapman and PhysioNet 2020 datasets, respectively. Therefore, our model is presented with a mini-batch of dimension K x S x 2 where K is the batchsize, and scenarios with a different number of leads
S is the number of samples. In CMLC, we explore corresponding to the same instance. Our mini- batch dimension is K x S x L, where L is the number of leads. Lastly, in CMSMLC, we incorporate an additional temporal segment in each mini-batch. Therefore, our mini-batch dimension is K x 2S x L. To ensure a fair comparison between all methods, we expose them to an equal number of patients and instances during training. In CMLC or CMSMLC, we either pre-train using 4 leads (II, V2, aVL, aVR) or all 12 leads. We chose these 4 leads as they cover a large range of axes.
5.3 EVALUATION ON DOWNSTREAM TASK
We evaluate pre-trained methods in two scenarios. In Linear Evaluation of Representations, we are interested in evaluating the utility of the fixed feature extractor in learning representations. Therefore, the pre-trained parameters frozen and multinomial logistic regression is performed on the downstream supervised task. In Transfer Capabilities of Representations, we are interested in evaluating the inductive bias introduced by pre-training. Therefore, the pre-trained parameters used as an initialization for training on the downstream supervised task.
5.4 BASELINES
We compare our pre-training methods to networks that are initialized randomly (Random Init.), via supervised pre-training (Supervised), or via a multi-task pre-training mechanism introduced specifically for ECG signals (MT-SSL) (Sarkar & Etemad 2020). We also compare BYOL (Grill et al., 2020) and CLR (Chen et al., 2020). which encourage representations of instances and their perturbed counterparts to be similar to one another, with the aim of learning transformation-inve -nvvariant representations that transfer well. As SimCI.R has been shown to be highly dependent on the choice of
perturbations, we explore the following time-series perturbations (see Appendix B for visualizations):
we add e (0, a) to the time-series signal where we chose a based on the amplitude of the signal. This was motivated by the work of et al. (2020) who recently showed the effect of additive noise on ECG signals.
Gaussian
Flip- we flip the time-series signal temporally (Flipy.,, reversing the arrow of time, or we invert the time-series signal along the x-axis (Flipx)).
SpecAugment (Park et al. 2019 we take the short-time Fourier transform of the time-series
signal, generating spectrogram. We then mask either temporal (SA,) spectral (SA,) bins
5

---

Under review as a conference paper at ICLR 2021
of varying widths before converting the spectrogram the time domain. We also explore the application sequential perturbations to the time-scries signal.
5.3 HYPERPARAMETERS
During self self-supervised pre-training, we chose the temperature parameter, 7 0.1, as per Chen et al. (2020). For BYOL, we chose the decay rate, Tu 0.90, after experimenting with various alternatives (see Appendix [F). We use the same network architecture for all experiments. Further implementation details can be found in Appendix
6
EXPERIMENTAL RESULTS
6.1 LINEAR EVALUATION OF REPRESENTATIONS
In this section, we evaluate the utility the supe-sup representations learned using four Ieads on downstream linear classification task. In Table show the test AUC Chapman and PhysioNet 2020 using 50% the labelled data (F 0.5) after having learned representations, with dimension E 128, using the same two datasets.
We show that CMSC outperforms BYOL and SimCLR on both datasets. On the Chapman dataset, CMSC and SimCLR achieve an AUC = 0.896 and 0.738, respectively, illustrating 15.8% improvement. Such a finding implies that the representations learned by CMSC are richer and thus allow for improved generaliza- tion. We hypothesize that this is due to the setup of CMSC whereby the shared context is across segments (temporally) and patients. Moreover, show that CLOCS (all 3 proposed methods) outperforms SimCLR in 100% of all conducted experiments, even when pre-training and evalu- ating with all 12 leads (see Appendix
Table I: Test AUC of the linear evaluation of the representations F 0.5, after having pre-trained on Chapman or PhysioNet 2020 with E 128. Pre-trate Pre-training and evaluating multi-lead datasets* using 4 leads (II, V2, aVL, aVR). Mean and standard deviation are shown across seeds.
Dataset
Chapman"
PhysioNer 2020*
MT-SSL
0.677
0.024
0.665 : 0.015
BYOL
0.643
0.043
0.595 1: 0.018
SimCLR
0.738 0.034
06615 0.014
CMSC
0.896 + 0.005
0.715 = 0.033
CMSMLC
0.847 = 0.024
0.680 0.008
6.2 EFFECT OF PERTURBATIONS ON PERFORMANCE
So far, have presented CLOCS without having incorporated any perturbations during pre-training. However, contrastive learning methods, and in particular SimCLR, are notorious for their over- dependence on the choice of perturbations. To explore this dependence, we apply diverse set of stochastic perturbations, P. (sce Appendix B) during pre-training and observe its effect on generalization performance. We follow the setup introduced by Chen et al. (2020) and apply either single perturbation to each instance, x², whereby a Pi''),, or sequential perturbations whereby 2 = (x))).
We upply such perturbations while pre-training with SimCLR CMSC on PhysioNet 2020 using 4 leads and, in Fig. 3 illustrate the test AUC in the linear evaluation scenario. We show that, regardless of the type and number of perturbations, CMSC continues to outperform SimCLR. For example, the woost-performing CMSC implementation (Flipy) ) results in an AUC = 0.661 which is still greater than the best-performing SimCLR implementation (Gaussian - SAa) with an AUC 0.636. In fact, we find that pre-training with CMSC without applying any perturbations (see Table T still outperforms the best performing SimCLR implementation. Such finding suggests that CMSC's already strong performance is more likely to stem from its redefinition of the "shared context' to include both time and patients than from the choice of perturbations.
6.3 TRANSFER CAPABILITIES OF REPRESENTATIONS
In this section, evaluate the utility of initializing network for downstream task with parameters learned via self-supervision using four leads. In Table 2, we show the test AUC on downstream datasets F 0.5 for the various self-supervised methods with F = 128.

---

Under review as a conference paper at ICLR 2021
SimCLR
Single Perturbation
Sequential Perturbations
0.70
0.615
0.598
0.603
0.592
0.630
Gaussian
0.615
0.609
0.612
0.615
0.636
0.65
Gaussian
Flipy
Flipx
SAr
SAc
Gaussian
Flipy
Flipx
SA₁
SAt
CMSC
0.70
0.724
0.661
0.706
0.729
0.697
Gaussian
0.724
0.695
0.701
0.691
0.699
0.65
Gaussian
Flipy
Flipx
SAr
SA
Gaussian
Flipy
Flipx
SA
0.60
Figure Effect of single (blue) and sequential (green) perturbations applied the (top) SimCLR and (hottom) CMSC implementations on linear evaluation. Sequential perturbations involve a Gaussian perturbation followed by one the remaining four types. Pre-training and evaluation was performed on PhysioNet 2020 using 4 leads. Evaluation was performed at / = 0.5 and results are averaged across 5 seeds. We show that CMSC outperforms SimCLR regardless of the applied perturbation.
We show that, with few exceptions, self-supervision is advantageous relative to Random Initial- ization. This can be seen by the higher AUC achieved by the former relative to the latter. We also show that, depending on the downstream dataset, either CMSC or CMSMLC outperform BYOL and SimCLR. For example, when pre-training on Chapman and fine-tuning on Cardiology, CMSMLC achieves AUC 0.717, 4.1% improvement compared SimCLR. This implies that by en- couraging representations across space, time, and patients to be similar to one another, networks are nudged into favourable parameter space. In Appendix E.T] we extend these findings and illustrate that CLOCS outperforms SimCLR in at least 75% of all experiments conducted, on average. When pre- pre-training, fine-tuning, and evaluating using all 12 leads, we show that CMSC outperforms all other methods in at least 90% of all experiments conducted (see Appendix E.2
Table 2: Test AUC in the fine-tuning scenario at F = 0.5, after having pre-trained on Chapman or PhysioNet 2020 with 128. Pre-training, finettuing, and evaluating multi- datasets using leads. Mean and standard deviation are shown across 5 seeds.
Pretraining Dataset
Chapman
PhysioNet 2020*
Downstream Dataset
Cardiology
PhysioNet 2017
PhysioNet 2020*
Cardiology
PhysioNet 2017
Chapman*
Random Init. Supervised
0.678 + 0.011 0.684 0.015
0.763 = 0.005
0.803 + 0.008 0.827 0.001
0.678 0.011
0.763 = 0.005
0.907 + 0.006
0.799 0.008
0.730 0.002
0.009
0.954 0.003
Self-supervised Priiiinggg
MT-SSL
0.650 0.009
0,741 = 0.012
0.774 + 0.010
0.661 0.01
0.746 0.016
0.923 + 0.007
BYOL
0.678 = 0.021
0.748 0.014
0,802 0.013
0.674 0.022
0.757 0.010
0.916 0.009
SimCLR
0.676 0.011
0.772 0.010
0.823 0.011
0.658 0.022
0.762 0.009
0.923 0,010
CMSC
0.695 : 0.024
0.773 = 0.013
0,830 1 0.002
0.714:0..00
0.760 0.013
0.932 : 0.008
CMLC
0.665 0.016
0.767 0.013
0810 0.011
0.675 0.003
0.762 0.007
0.910 = 0.012
CMSMLC
0.717 0.006
0.774 0.004
0.814 : 0.009
0.698 : 0.011
0.774 0.012
0.930 =: 0.012
6.4 DOING MORE WITH LESS LABELLED DATA
Having established that supervisiin can nudge networks favourable parameter space, we set out to investigate whether such a space can lead to strong generalization with less labelled data in the downstream task. In Fig. 4 we illustrate the validation AUC of networks initialized randomly or via CMSC and fine-tuned on two different datasets.
We find that fine-tuning a network based on a CMSC initialization drastically improves data-efficiency, In Fig. 4a we show that network initialized with CMSC and exposed to only 25% of the labelled data outperforms one that is initialized randomly and exposed to 100% of the labelled data. This can be seen by the consistently higher AUC during, and at the end of, training, A similar outcome can be seen in Fig. (46] This suggests that self-supervised pre- pre-traine training exploits data efficiently such that can do more with less on downstream classification tasks.

---

Under review as a conference paper at ICLR 2021
0.030
0865
0.575
0.550
Sanden,
Sanden,
0.500
5
10
15
20
30
050
0
10
20
40
50
60
Epochs
Epocts
(a) PhysioNet 2020 - Cardiology
(b) Chapman - PhysioNet 2017
Figure 4: Validation AUC of network initialized randomly via CMSC and which is exposed to different amounts of labelled training data, F. Results are averaged across 5 seeds. Shaded area represents standard deviation.
6.5 EFFECT OF EMBEDDING DIMENSION, E, AND AVAILABILITY OF LABELLED DATA, F
The dimension of the representation learned during self- self-superviiion and the availability of labelled training data can both have an effect on model performance. In this section, we investigate these claims. In Figs. 5a and 5b illustrate the test AUC for all pre-training methods function of E = (32, 64, 128, 256) and = (0.25, 0.50, 0.75, 1).
SECCUR
CHIM
OMSC
CMUC
0.66
0.65
AUC
0.56
0.5
0.50
30
64
121l
256
0.50
025
115
0.75
1.12
Eimzdkking Eimemsion
(a) Chapman > Cardiology, 0.25
(b) Chapman
Cardiology, E
Figure 5: Effect of (a) embedding dimension, E. and (b) labelled fraction, F, the test AUC when pre-training on Chapman and fine-tuning on Cardiology. Results are averaged across 5 seeds. Error bars represent standard deviation.
In Fig. 5al we show that networks initialized randomly via SimCLR are not significantly affected by the embedding dimension. This can be seen by the AUC x 0.63 and ss 0.65, for these two methods across all values of E. In contrast, the embedding dimension has greater effect on CMSC where AUC = 0.66 + 0.69 as E = 32 + 128. This implies that CMSC is still capable of achieving strong generalization performance despite the presence of few labelled data (F 0.25). We hypothesize that the strong performance of CMSC, particularly F = 128, is driven by its learning of patient-specific representations (see Appendix GD that cluster tightly around one another, a positive characteristic especially when such representations map the same downstream class.
In Fig. 5b we show that increasing the amount of labelled training data benefits the generalization performance of all methods. This can be seen by the increasing AUC values as F 0.25 - 1. We also show that at all fraction values, CMSMLC outperforms its counterparts. For example, at F 1, CMSMLC achieves AUC 0.732 whereas SimCLR achieves an AUC 0.718. Such superiority still holds at F 0.25 where the two methods achieve an AUC = 0.675 and 0.652, respectively. This outcome emphasizes the robustness of CMSMLC to scarce labelled training data.
8

---

Under review as a conference paper at ICLR 2021
6.6 CLOCS LEARNS PATIENT-SPECIFG REPRESENTATIONS
We redefined "shared context' refer to representations from the same patient, which in turn should produce patient-specifie representations. To validate this hypothesis, we calculate the pairwise Euclidean distance between representations of the patient (Intra-Patient) and those of different patients Liner-PPttient, On average, the former should be smaller than the latter. In Fig. 6 we illustrate the two distributions associated with the intra and inter-patient distances F 128. We also find that increasing the embedding dimension shifts these distributions to higher values (see 9
We show that these two distributions have large mean values and overlap significantly when imple- menting SimCLR, as seen in Fig. 6a This expected as SimCLR is blind to the notion of a patient. In contrast, when implementing CMSC, the intra-patient distances are lower than those found in SimCLR, as seen in Fig. 6b Moreover, the intra and inter- inter-patient distributions are more separable. This implies that pre- pre-training with CMSC leads to patient-specific representations. We note that this phenomenon takes place while concomitantly learning better representations, as observed previous sections.
B03 HAINT
Bak Fuleet
0.5
0.4
0.15
0.3
0.10
0.2
0.06
0.00
32 Parvitoe Luckidenn Distaence
14
15
18
0.0
2
6
8
10
12
14
16
10
Painiise Eudideee Ussaance
(a) SimCLR
(b) CMSC
Figure 6: Distribution of pairwise Euclidean distance between representations (E = 128) belonging to the same patient
and those belonging to different patients Liner-Patteett) Self- supervision performed on PhysioNet 2020. Notice the lower average intra- aatte-patient distance and improved separability between the two distributions with CMSC than with SimCIR.
7 DISCUSSION AND FUTURE WORK
In this paper, we proposed a family of self-supervised pre-training mechanisms, entitled CLOCS, based on contrastive learning for physiological signals. In the process, encourage representations across segments (temporally) and leads (spatially) that correspond to instances from the same patient be similar to one another. We show that our methods outperform the ofate-of-the-art methods, BYOL and SimCLR, when performing a linear evaluation of, and fine-tuning on, downstream tasks. This conclusion also holds when applying a range of perturbations and when pre-training and evaluating with different number of leads. We now elucidate several avenues worth exploring.
Quantifying patient similarity. We have managed to learn patient-specific representations. These representations can be used to quantify patient-similarity in order to assist with diagnosis gain better understanding a diseased condition. Validation of these representations can be performed by comparing known similar patients.
Multi-modal transfer. We transferred parameters from one task to another that shared the same input modality, the ECG. Such data may not always be available for self self-supervision. An interesting path would be to explore whether contrastive self-supervision on one modality can transfer well to another modality.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Philip Bachman, Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In Advances in Neural Information Processing Systems, pp. 15509-15519, 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709 2020.
Gari D Clifford, Chengyu Liu, Benjamin Moody, H Lehman Li-wei, Ikaro Silva, Qiao Li, AE John- son, and Roger G Mark. Af classification from short single lead ECG recording: the physionet/computing in cardiology challenge 2017. 2017 Computing in Cardiology, pp. 1-4, 2017.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv: 2018.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.0773: 2020.
Xintian Han, Yuxuan Hu, Luca Foschini, Larry Chinitz, Lior Jankelson, and Rajesh Ranganath. Deep learning models for electrocardiograms are susceptible to adversarial attack. Nature Medicine, pp. 1-4, 2020.
Awni Y Hannun, Pranav Rajpurkar, Masoumeh Haghpanahi, Geoffrey H Tison, Codie Bourn, Mintu P Turakhia, and Andrew Y Ng. Cardiologist-leve arrhythmia detection and classification in ambulatory electrocardiograms using a deep neural network. Nature Medicine, 25(1):65, 2019.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. arXiv preprint 19Xiv:191..0577 2019.
Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. In Advances in Neural Information Processing Systems, pp. 3765-3773, 2016.
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Colorization as proxy task for visual understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6874-6882 2017.
Yikuan Li, Shishir Rao, Abdelaali Hassaine, Rema Ramakrishnan, Yajie Zhu, Dexter Canoy, Gholam- reza Khorsiid, Thomas Lukasiewicz, and Kazem Rahimi. Deep bayesian gaussian pro- cesses for uncertainty estimation in electronic health records. arXiv preprint arXiv:2003. 101700 2020.
Xinrui Lyu, Matthias Hueser, Stephanie L. Hyland, George Zerveas, and Gunnar Rätsch. Improving clinical predictions through unsupervised time series representation learning. arXiv preprint 11Xiv:1812.004490 2018
Riccardo Miotto, Li Li, Brian A Kidd, and Joel Dudley. Deep patient: an unsupervised representa- tion to predict the future of patients from the electronic health records. Scientific Reports, 6(1): 1-10, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: /807.0.0 0374083748 2018.
Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le. Specaugment: A simple data augmentation method for automatic speech recognition. arXiv preprint 1XX4:19940007799 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-perid perfforrromm deep learning library. In Advances Neural Information Processing Systems, pp. 8024-8035, 2019.
10

---

Under review as a conference paper at ICLR 2021
E. A. Perez Alday, A. Gu, A. Shah, C. Liu, A. Sharma, S. Seyedi, A. Bahrami Rad, M. Reyna, and G. Clifford. Classification of 12-lead ECGs: the PhysioNet computing in cardiology challenge 2020 (version 1.0.1). PhysioNet, 2020.
Pritam Sarkar and Ali Etemad. Self-supervised ecg representation learning for emotion recognition. arXiv preprint 0rX:822002.03888, 2020.
Pierre Sermanet, Corey Lynch, Jasmine Hsu, and Sergey Levine. Time-contrastiive networks: Self- supervised learning from multi-view observation. In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 486-487. IEEE, 2017.
Zhiqiang Shen, Zechun Liu, Zhuang Liu, Marios Savvides, and Trevor Darrell. Rethinking image mixture for unsupervised visual representation learning. arXiv preprint arXiv:2003.05433 2020.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint arXiv: 05849, 2019.
Donglai Wei, Joseph J Lim, Andrew Zisserman, and William Freeman. Learning and using the arrow of time. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, pp. 8052-8060, 2018
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 3733-3742, 2018.
Jianwei Zheng, Jianming Zhang, Sidy Danioko, Hai Yao, Hangyuan Guo, and Cyril Rakovski. A 12-lead electrocardiogram database for arrhythmia research covering more than 10,000 patients. Scientific Data, 7(1):1-8, 2020.


Reviews:
[{"review_id": "nnVXxVNw08Y", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper describes a method for unsupervised learning of patient representations from ECG data.\nUsing contrastive learning, ECG recordings from different time periods and different leads are optimized to be similar for the same patient and different for all the other patients.\nThe method is evaluated on several datasets, showing improvement over random initialization and an alternative method using data permutations.\n\nThe method is simple and a relatively minor extension of contrastive learning to time series. \nBut it is elegant, shows good results over alternative approaches and can be a useful solution for time series.\n\nThe paper doesn't currently  contain a description of the actual input and the network architecture. There are 29 pages of appendices and these have some additional details, but the main paper should contain some of this important information as well.\n\nEquations 1-4 need to come with some explanation. At the moment, several variables there are left undefined and the overall intuition behind structuring the loss equations should be explained.\n\nCould these additional objectives or data augmentations be applied during the fine-tuning phase as well? If so, would that reduce the benefit of unsupervised pretraining?", "extra_scores": {}, "extra_text": {}}, {"review_id": "km-jwTBw8WO", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to use contrastive learning to learn representations from cardiac signals (ECGs). The model incorporates ECG domain knowledge, patient-specific, and relationships between multiple leads (channels), in the learning process. The targeted task is very important, enormous ECGs are collected and stored, but seldom people are mining them as they are unlabelled. This paper might reinvigorate them. \n\nMajor: \n\nHowever, some of the domain knowledge-based intuitions are unclear (might even be wrong). For example, for Contrastive Multi-lead Coding, we know that ECGs are 2D projections from the heart's 3D electricity activities. The similarity between different leads is related to the projection surface. What if two leads are orthogonally projected? They should be unrelated at all. Thus would even bring the noise in the model. Maybe this is the reason why CMSC in Table 1 is the best, but adding Multi-lead Coding (CMSMLC) drops the performance. \n\nIn fact, unsupervised learning usually needs (much) larger datasets. But all four experimental datasets are actually well annotated (relatively small) ECG databases. It seems more reasonable to experiment on a larger unlabelled dataset, such as the MIMIC-III waveform database (https://physionet.org/content/mimic3wdb/1.0/). And it might give more convincing and desired results. \n\nOthers: \n\n(1) This paper mainly focuses on design positive and negative pairs from the perspective of ECG domain knowledge - between leads, within subject. The technical contribution looks limited. \n\n(2) Some related papers about contrastive learning on physiological signals. \n\nPublished (you might reconsider the last sentence in Related Work.):\n\n[1] Banville, Hubert, et al. \"Self-supervised representation learning from electroencephalography signals.\" 2019 IEEE 29th International Workshop on Machine Learning for Signal Processing (MLSP). IEEE, 2019.\n\nPreprint:\n\n[2] Cheng, Joseph Y., et al. \"Subject-aware contrastive learning for biosignals.\" arXiv preprint arXiv:2007.04871 (2020).\n\n[3] Banville, Hubert, et al. \"Uncovering the structure of clinical EEG signals with self-supervised learning.\" arXiv preprint arXiv:2007.16104 (2020).\n\n(3) Multi-lead ECG is more common in clinical usage. Can this method be extended to learn representations from multi-lead ECG?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Lp3p_qUXl6Y", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a contrastive learning method for cardiac signals.\n\nThis strong points of the paper are the following:\n1. It presents its problem clearly. The problem it targets is an impactful problem in the medical domain.\n2. The method it proposes is straight forward and easily understandable.\n3. It demonstrates the advantage of the proposed method on a real dataset by comparing against the SOTA CL methods like BYOL and SimCLR.\n\nThe paper also has weaknesses.\n1. unsupported claims In the abstract.\n it is claimed that \"our training procedure naturally generates patient-specific representations that can be used to quantify patient-similarity\". Then the next time where the \"patient-similarity\" appears is in the discussion. It says \"We have managed to learn patient-specific representations....\". However, in the main paper, I can not find any evidence showing the representation it learns is capturing the patient-similarity. It is unacceptable to claim something by just claiming it.\n2.unclear contributions.\n In the introduction, the author is listing their methods as contributions. Usually, the contribution should be something achieved by you while not previously achieved. It could the new best performance, a new solution to a problem, a new angle of viewing the problem. The method itself alone hardly can be the contributions. I can propose any algorithms. They can not be contributions if they are not useful. So I suggest the author think about elaborating on what are your contributions.\n3.not sufficient description of the task.\n In the whole paper, the only thing I get about the task is that the author focuses on is a classification task related to cardiac arrhythmia. This is far from sufficient. The author should provide the backgrounds of the task. What are the extra 11 or 4 classes of cardiac arrhythmia? Is the task label also time series? or it is a global label that does change across time? For most of the readers from the ML community, they do not have a background in cardiac signals. To let the audience understand the tasks, I suggest the author to including visualization about the class label and the cardiac signals. Currently, I have no idea about what the task exactly is.\n4.lack of insight.\n Currently, the paper is mainly written in a way like I did A,B,C,D; they are better than previous method E,F; see my numbers. Here are my questions. Why doing A,B,C,D? For example, why Constrasitve Multi-segments Coding make sense? This question is also related to my previous point. What is your class label? If it is also a time series, considering the following case. At time step t1, the patient is normal, while at time step t2, the patient is in some state of cardiac arrhythmia. Why it makes sense to encourage the representations at t1 and t2 to be similar?\nActually, the experimental result is interesting. From Table 2, I notice that the best performance is achieved by either CMSC or CMSMLC. It means that contrastive multi-segments across time is useful. It contradicts my intuition in the last paragraph. Can the author give any explanation?\nThere are more questions about the results the author showed. For example, in Figure 4(a), CMSC has an abnormally high AUC when embedding dimension = 128. Why does it happen?\n\nI hope the author can include more insight or deep analysis in the paper to help the reader learn more from this paper.\nThe paper has a good topic, good methods, and good experiments. However, it still needs some effort to be a good paper. I hope my comments can help the author improve the paper and make it good work. My rating is not final. I will raise the score if my main concerns are addressed.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "8kd928a8UUN", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis work presents a new self-supervised training framework for multi-channel ECG signals. The authors use contrastive learning by exploiting the fact that a single patient can generate multiple ECG signals, and there are multiple views (i.e. leads) for the same ECG signals. Compared to popular self-supervised training methods BYOL and SimCLR, the proposed method shows superior performance on the arrhythmia classification task for 4 different datasets in various scenarios.\n\nPros:\n- The proposed approach is easy to follow, makes much sense, and is well-motivated based on the domain-specific knowledge of ECG signals.\n- The proposed approach outperforms popular baselines (although borrowed from the vision community) for the arrhythmia classification task on various combinations of experiment setups.\n\nCons:\n- The most critical shortcoming of this paper is that the baselines, although widely known, are borrowed from the computer vision community only, which is the reason for rating 4. There are already several papers proposing self-supervised learning methods for time-series data such as audio signals, ECGs and EEGs [1, 2, 3, 4, 5]. The authors should compose a stronger baseline set to demonstrate the value of this work. (Audio signals are usually single-channel unlike ECG signals, but CMSC alone already demonstrates good performance, hence methods such as wave2vec could be a strong baseline)\n- The third paragraph of section 6.5, where the authors claim the superiority of CMSC across various embedding dimensions seems like a stretch. The AUC difference of SimCLR (which is 0.02) and that of CMSC (0.03) is not that large, and CMSC even underperforms when the embedding dimension is 256. The proposed method has already shown good performance in various cases, so there is probably no need to make a stretched claim.\n\nReference\n1. Schneider, S., Baevski, A., Collobert, R. and Auli, M., 2019. wav2vec: Unsupervised pre-training for speech recognition. arXiv preprint arXiv:1904.05862.\n2. Baevski, A., Zhou, H., Mohamed, A. and Auli, M., 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. arXiv preprint arXiv:2006.11477.\n3. Banville, H., Moffat, G., Albuquerque, I., Engemann, D.A., Hyvärinen, A. and Gramfort, A., 2019, October. Self-supervised representation learning from electroencephalography signals. In 2019 IEEE 29th International Workshop on Machine Learning for Signal Processing (MLSP) (pp. 1-6). IEEE.\n4. Sarkar, P. and Etemad, A., 2020. Self-supervised ecg representation learning for emotion recognition. arXiv preprint arXiv:2002.03898.\n5. Cheng, J.Y., Goh, H., Dogrusoz, K., Tuzel, O. and Azemi, E., 2020. Subject-aware contrastive learning for biosignals. arXiv preprint arXiv:2007.04871.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a representation learning approach from cardiac signals, which adopts contrastive learning to incorporates knowledge on patient-specificity. This problem is highly motivating because of potential application to medicine and healthcare and large amounts of accumulating unlabeled physiological data. The presentation needs to be substantially improved – e.g., lack of clear description of the contribution, the details such as input data and network architecture, a clear description of the downstream tasks, missing explanations of equations, etc – some of which were addressed in the revision. Major concerns include lack of comparison with relevant prior methods developed for cardic signals, and the need for further refinement of domain knowledge-based intuitions. 
Decision: Reject