Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: ICLR PAPER 

PCPs: Patient Cardiac Prototypes


##########################################################################

Summary:

 
The paper proposes unsupervised neural network models to learn patient-specific representations for ECG applications. The architecture implements ideas from prototype networks and contrastive learning in order to discover similar patients across datasets and compress datasets based on the representations only. 

##########################################################################

Reasons for score: 

 
Overall, I vote for marginal reject. The paper is very well written and I like the idea of leveraging the representations of ECG data but my major concerns are the difference in terms of novelty when compared to other concurrent papers submitted to this venue, namely DROPS, CLOPS (arxiv, Kiyasseh et al.), and CLOCS (arxiv, Kiyasseh et al.). They all seem to use the same datasets and building upon the contrastive learning paradigm. The tasks might be slightly different however all share significant common ground.

 
##########################################################################Pros: 

 
1. The paper leverages one of the most important and overlooked data modalities, that of ECG. The community should invest more in models tailored to this kind of sensor data.

2. This paper provides comprehensive experiments, including both qualitative analysis and quantitative results, to show the effectiveness of the proposed framework. I especially enjoyed the euclidean distance plots.

 
##########################################################################

Cons: 

 
1. My major concern, as described above is the novelty compared to other methods proposed recently, possibly by the same team. A comprehensive comparison/ablation across all these methods is needed in order to evaluate what achieves better results in which task.
 
2. Also, I struggle to understand if the resulting PCPs are fewer data points than the original training set. As far I understand, a patient generates a very long time series of ECG which has to be segmented to chunks or windows. Now, PCPs compress all these windows to a single latent vector (?). In this regard, why do we need the fancy hypernetwork? Could we just average all latent representations of the same patient after training is done? It seems that the paper is using a "mean" baseline but this is still part of the PCPs, not all training samples. 
 
3. In terms of tasks, the paper presents the patient similarity and the dataset distillation as the most important applications of this method. However there are no comparisons with any other methods, for example, established methods from distillation (teacher-student, core-sets). 

 
##########################################################################

Questions during the rebuttal period: 

 
Please acknowledge all concurrent papers using similar datasets/methods/tasks.

The hypernetwork module is not motivated sufficiently.

Are there other tasks that this method could be useful to?


review_title: Important application domain but unclear contribution

final_score: rating: 7: Good paper, accept
review_content: Intuitive concept and interesting study.

Some general comments. 

It would be helpful to know why authors used cosine similarity score as opposed to other metric. Just because of its simplicity? explain.

What is the difference between this work and a clustering approach, this seems to be somehow a clustering solution they are providing. I suggest having a comparison with existing phenotypic (clustering) that have been proposed and show how this new model is better (if) and if such comparison is unreasonable, then I except to have a clear justification on why and how in theory this is different and perhaps better. 

The motivation can be further emphasized. I understand the rational given my experience working in healthcare setting; however, other readers may not see the true value of this project as presented. 

The English is at time very assertive and strong use of statement, where instead the patten of distributions are not confirming but rather corroborating since the experiments are not systemic but rather authors show few distributions that seem to demonstrate what they want to highlight. I do not say, the authors are cherry picking but the way it is presented can be misleading.   

In terms of manuscript, the abstract is too short and should have the typical sections for the reader to get a sense before jumping to the full manuscript. Furthermore, as abstracts are usually used in automated searches and meta-analysis, is highly important to have a clear and comprehensive abstract (background, method, results, discussion and conclusion). 

Code should be made available for this submission. 
review_title: Intuitive, however, clinical utility maybe limited as presented

final_score: rating: 2: Strong rejection
review_content: This paper proposes to learn patient-specific representation using patient physiological signals.  The authors design a PCP representation for each patient, which is learned to agree with signals from the same patients and disagrees with the remaining patients. In the supervised part, the classifier is generated from patient-specific parameters by meta-learning. The model was evaluated on three large ECG datasets: PhysioNet 2020 ECG, Chapman ECG, PTB-XL ECG.

Strength: 
This is an important problem
The paper is easy to follow, and the experiment settings are well elaborated 

___________________________________________________________________________
Weakness:

The authors are ignorant to several lines of existing literature in deep learning for healthcare, particularly deep phenotyping and deep patient subtyping. They also seem not aware of the existing works on explaining Black-box models.   For deep phenotyping algorithms, the patient representations generated from existing deep phenotyping algorithms are ALL patient specific.  To list a few, the authors may take a look at [1-4]. For more related works on deep phenotyping, you may refer to [5]. For more related works on deep learning for ECG data, you may refer to [6]. To learn disease subtypes, patient prototypes, there are also a series of works. Below are several the authors could refer to [7-10]. In addition, there are several model agnostic algorithms designed for explaining deep learning models that could be used here to add more explanability. For example [11-13]. In addition, there are many existing works around contrastive learning on ECG signals, see [14-16]. The authors are either not aware of these lines of works or have wrong understanding on them, which cause several major issues as listed below.

(a) Lack of Novelty. Given the existing works listed below. The novelty of the paper is not enough for the conference. The methodology of the paper consists of a standard contrastive learning method and a standard meta-learning setting. All have been done in one or more existing works.

(b) Lack of Baselines. The paper does not include any baseline. It is unclear how it compares with existing works.  

(c) The experiment results are mostly unconvincing. For example,
- In Experiment 5.1, the paper states that “PCPs exhibit tighter clusters than those found with training representations” in Figure 2. However, it is hard to tell which result is more separable from my point of view. A possible suggestion is that the authors could use quantitative clustering metrics to demonstrate, such as Adjusted Rand index. Also, the resolution of figure 2 could be improved.
- In Experiment 5.2,  the detail of meta-learning network, hypernetwork, is not mentioned in the paper. Is it just a matrix transform? Or a more complicated neural network? The detailed implementation of hypernetwork could be discussed.
- In Experiment 5.3, The result is not surprising, since the model have used contrastive loss to maximize similarity of “PCP to Same Training Patient”. Therefore, it should have smaller distances, naturally. Also, we encourage the paper to discuss why using the Euclidean distance not the cosine similarity as mentioned in Sec. 3.1.
- In Experiment 5.4, the paper shows two examples of similar patients, where the performance is hard to evaluate. The reviewer will suggest two ways: (i) using demographics, physiology, or treatment features to match two similar patients, and then quantitatively compare with the matching results given by the paper; (ii) setting distance threshold or cosine similarity threshold to decide whether two patients are similar, and then justify the threshold.
- In Experiment 5.5. This experiment is very problematic. First, the paper claims and show that the PCP representation could provide the same classification performance as training on the whole training set. It is also not surprising, because when obtaining the PCP representations, the model already uses all the training data, then of course PCP would give the same performance; Second, we encourage the paper to compare with using different proportions of the training set. Often the time, the whole training set could be redundant in ECG setting.
- The authors are encouraged to analyze the computational complexity of the proposed method.
- In appendix, figure 9 (bottom) has exactly the same channel signals, which is a significant mistake. Figure 10 has a wrong title description.
- For dataset PhysioNet 2020 ECG, the paper states that “Each recording can be associated with multiple labels”. It is unclear how to use the dataset for evaluation.
- The motivation of the paper could be improved. Also, some related works on contrastive learning are missing, for example [17-18].

References

[1] RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanism. NeurIPS 2016

[2] MiME: Multilevel Medical Embedding of Electronic Health Records for Predictive Healthcare. NeurIPS 2017

[3] MINA: Multilevel Knowledge-Guided Attention for Modeling Electrocardiography Signals. IJCAI 2019

[4] RAIM: Recurrent Attentive and Intensive Model of Multimodal Patient Monitoring Data. KDD 2018

[5] Opportunities and challenges in developing deep learning models using electronic health records data: a systematic review, Journal of the American Medical Informatics Association 2019

[6] Opportunities and Challenges in Deep Learning Methods on Electrocardiogram Data: A Systematic Review, Computers in Biology and Medicine 2020

[7] Patient subtyping via time-aware LSTM networks, KDD 2017

[8] DDL: Deep Dictionary Learning for Predictive Phenotyping, IJCAI 2019

[9] PEARL: Prototype Learning via Rule Learning, ACM BCB 2019

[10] Identifying Sepsis Subphenotypes via Time-Aware Multi-Modal Auto-Encoder, KDD 2020

[11] Why should I trust you?: Explaining the predictions of any classifier. KDD 2016

[12] A Unified Approach to Interpreting Model Prediction. NeurIPS 2017

[13] Anchors: High-Precision ModelAgnostic Explanation AAAI 2018

[14] CLOCS: Contrastive learning of cardiac signals. arXiv preprint arXiv:2005.13249, 2020. 

[15] A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709, 2020. 

[16] Subject-aware contrastive learning for biosignals. arXiv preprint arXiv:2007.04871, 2020.

[17] Momentum contrast for unsupervised visual representation learning. CVPR 2020

[18] Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748.

review_title: Several major issues

Meta-review: The paper proposes the use of contrastive learning to learn patient specific representations from medical data. The authors show how their method can be used to find similar patients within and across datasets. 

The paper has some issues, as indicated by the reviewers:
- similarity to past work; in the response to R1, the authors specify differences to related papers; however, experimental comparisons should still be performed against CLOCS and DROPS
- the evaluation is not fully convincing (the follow up comments of Reviewer 3), including the retrieval of similar patients
Decision: Reject