Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a contrastive learning method for cardiac signals.

This strong points of the paper are the following:
1. It presents its problem clearly. The problem it targets is an impactful problem in the medical domain.
2. The method it proposes is straight forward and easily understandable.
3. It demonstrates the advantage of the proposed method on a real dataset by comparing against the SOTA CL methods like BYOL and SimCLR.

The paper also has weaknesses.
1. unsupported claims In the abstract.
 it is claimed that "our training procedure naturally generates patient-specific representations that can be used to quantify patient-similarity". Then the next time where the "patient-similarity" appears is in the discussion. It says "We have managed to learn patient-specific representations....". However, in the main paper, I can not find any evidence showing the representation it learns is capturing the patient-similarity. It is unacceptable to claim something by just claiming it.
2.unclear contributions.
 In the introduction, the author is listing their methods as contributions. Usually, the contribution should be something achieved by you while not previously achieved. It could the new best performance, a new solution to a problem, a new angle of viewing the problem. The method itself alone hardly can be the contributions. I can propose any algorithms. They can not be contributions if they are not useful. So I suggest the author think about elaborating on what are your contributions.
3.not sufficient description of the task.
 In the whole paper, the only thing I get about the task is that the author focuses on is a classification task related to cardiac arrhythmia. This is far from sufficient. The author should provide the backgrounds of the task. What are the extra 11 or 4 classes of cardiac arrhythmia? Is the task label also time series? or it is a global label that does change across time? For most of the readers from the ML community, they do not have a background in cardiac signals. To let the audience understand the tasks, I suggest the author to including visualization about the class label and the cardiac signals. Currently, I have no idea about what the task exactly is.
4.lack of insight.
 Currently, the paper is mainly written in a way like I did A,B,C,D; they are better than previous method E,F; see my numbers. Here are my questions. Why doing A,B,C,D? For example, why Constrasitve Multi-segments Coding make sense? This question is also related to my previous point. What is your class label? If it is also a time series, considering the following case. At time step t1, the patient is normal, while at time step t2, the patient is in some state of cardiac arrhythmia. Why it makes sense to encourage the representations at t1 and t2 to be similar?
Actually, the experimental result is interesting. From Table 2, I notice that the best performance is achieved by either CMSC or CMSMLC. It means that contrastive multi-segments across time is useful. It contradicts my intuition in the last paragraph. Can the author give any explanation?
There are more questions about the results the author showed. For example, in Figure 4(a), CMSC has an abnormally high AUC when embedding dimension = 128. Why does it happen?

I hope the author can include more insight or deep analysis in the paper to help the reader learn more from this paper.
The paper has a good topic, good methods, and good experiments. However, it still needs some effort to be a good paper. I hope my comments can help the author improve the paper and make it good work. My rating is not final. I will raise the score if my main concerns are addressed.

review_title: interesting topic; lack of insight; unsupported claims 

final_score: rating: 7: Good paper, accept
review_content: The paper describes a method for unsupervised learning of patient representations from ECG data.
Using contrastive learning, ECG recordings from different time periods and different leads are optimized to be similar for the same patient and different for all the other patients.
The method is evaluated on several datasets, showing improvement over random initialization and an alternative method using data permutations.

The method is simple and a relatively minor extension of contrastive learning to time series. 
But it is elegant, shows good results over alternative approaches and can be a useful solution for time series.

The paper doesn't currently  contain a description of the actual input and the network architecture. There are 29 pages of appendices and these have some additional details, but the main paper should contain some of this important information as well.

Equations 1-4 need to come with some explanation. At the moment, several variables there are left undefined and the overall intuition behind structuring the loss equations should be explained.

Could these additional objectives or data augmentations be applied during the fine-tuning phase as well? If so, would that reduce the benefit of unsupervised pretraining?
review_title: Simple but elegant

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary:
This work presents a new self-supervised training framework for multi-channel ECG signals. The authors use contrastive learning by exploiting the fact that a single patient can generate multiple ECG signals, and there are multiple views (i.e. leads) for the same ECG signals. Compared to popular self-supervised training methods BYOL and SimCLR, the proposed method shows superior performance on the arrhythmia classification task for 4 different datasets in various scenarios.

Pros:
- The proposed approach is easy to follow, makes much sense, and is well-motivated based on the domain-specific knowledge of ECG signals.
- The proposed approach outperforms popular baselines (although borrowed from the vision community) for the arrhythmia classification task on various combinations of experiment setups.

Cons:
- The most critical shortcoming of this paper is that the baselines, although widely known, are borrowed from the computer vision community only, which is the reason for rating 4. There are already several papers proposing self-supervised learning methods for time-series data such as audio signals, ECGs and EEGs [1, 2, 3, 4, 5]. The authors should compose a stronger baseline set to demonstrate the value of this work. (Audio signals are usually single-channel unlike ECG signals, but CMSC alone already demonstrates good performance, hence methods such as wave2vec could be a strong baseline)
- The third paragraph of section 6.5, where the authors claim the superiority of CMSC across various embedding dimensions seems like a stretch. The AUC difference of SimCLR (which is 0.02) and that of CMSC (0.03) is not that large, and CMSC even underperforms when the embedding dimension is 256. The proposed method has already shown good performance in various cases, so there is probably no need to make a stretched claim.

Reference
1. Schneider, S., Baevski, A., Collobert, R. and Auli, M., 2019. wav2vec: Unsupervised pre-training for speech recognition. arXiv preprint arXiv:1904.05862.
2. Baevski, A., Zhou, H., Mohamed, A. and Auli, M., 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. arXiv preprint arXiv:2006.11477.
3. Banville, H., Moffat, G., Albuquerque, I., Engemann, D.A., Hyvärinen, A. and Gramfort, A., 2019, October. Self-supervised representation learning from electroencephalography signals. In 2019 IEEE 29th International Workshop on Machine Learning for Signal Processing (MLSP) (pp. 1-6). IEEE.
4. Sarkar, P. and Etemad, A., 2020. Self-supervised ecg representation learning for emotion recognition. arXiv preprint arXiv:2002.03898.
5. Cheng, J.Y., Goh, H., Dogrusoz, K., Tuzel, O. and Azemi, E., 2020. Subject-aware contrastive learning for biosignals. arXiv preprint arXiv:2007.04871.

review_title: Review

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes to use contrastive learning to learn representations from cardiac signals (ECGs). The model incorporates ECG domain knowledge, patient-specific, and relationships between multiple leads (channels), in the learning process. The targeted task is very important, enormous ECGs are collected and stored, but seldom people are mining them as they are unlabelled. This paper might reinvigorate them. 

Major: 

However, some of the domain knowledge-based intuitions are unclear (might even be wrong). For example, for Contrastive Multi-lead Coding, we know that ECGs are 2D projections from the heart's 3D electricity activities. The similarity between different leads is related to the projection surface. What if two leads are orthogonally projected? They should be unrelated at all. Thus would even bring the noise in the model. Maybe this is the reason why CMSC in Table 1 is the best, but adding Multi-lead Coding (CMSMLC) drops the performance. 

In fact, unsupervised learning usually needs (much) larger datasets. But all four experimental datasets are actually well annotated (relatively small) ECG databases. It seems more reasonable to experiment on a larger unlabelled dataset, such as the MIMIC-III waveform database (https://physionet.org/content/mimic3wdb/1.0/). And it might give more convincing and desired results. 

Others: 

(1) This paper mainly focuses on design positive and negative pairs from the perspective of ECG domain knowledge - between leads, within subject. The technical contribution looks limited. 

(2) Some related papers about contrastive learning on physiological signals. 

Published (you might reconsider the last sentence in Related Work.):

[1] Banville, Hubert, et al. "Self-supervised representation learning from electroencephalography signals." 2019 IEEE 29th International Workshop on Machine Learning for Signal Processing (MLSP). IEEE, 2019.

Preprint:

[2] Cheng, Joseph Y., et al. "Subject-aware contrastive learning for biosignals." arXiv preprint arXiv:2007.04871 (2020).

[3] Banville, Hubert, et al. "Uncovering the structure of clinical EEG signals with self-supervised learning." arXiv preprint arXiv:2007.16104 (2020).

(3) Multi-lead ECG is more common in clinical usage. Can this method be extended to learn representations from multi-lead ECG?

review_title: Questions about intuitions of Contrastive Multi-lead Coding

Meta-review: This paper proposes a representation learning approach from cardiac signals, which adopts contrastive learning to incorporates knowledge on patient-specificity. This problem is highly motivating because of potential application to medicine and healthcare and large amounts of accumulating unlabeled physiological data. The presentation needs to be substantially improved – e.g., lack of clear description of the contribution, the details such as input data and network architecture, a clear description of the downstream tasks, missing explanations of equations, etc – some of which were addressed in the revision. Major concerns include lack of comparison with relevant prior methods developed for cardic signals, and the need for further refinement of domain knowledge-based intuitions. 
Decision: Reject