Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: Paper presents for retrieving cardiac signal data corresponding to a patient described  by a set (3) of attributes (Sex, age, disease type). This is built on a model constructed by clustering such cardiac signals representations and learning their corresponding patient attributes.
	Although well written, the paper suffers, in our opinion, from targeting a very simplified problem (number of attributes, attribute sampling) that lends it to building a bespoke solution that overfits the used data and thus overperforms other methods.  The novelty was not clearly apparent to us from other clustering/retrieval methods.
main_review: - "by the rapid growth of large-scale clinical databases and the increased prevalence of unlabeled instances; those for which patient attribute information is unavailable."
	- This covers a multitude of possible cases (some diagnostics data missing, variability in acquired diagnostics data from different origins (equipment, institutions, countries etc.) from different patients, missing patient data). A few examples of the ones you specifically target would be welcome at this stage.
- "Such methods, however, are exclusively unsupervised; they do not exploit patient attribute information."
	- Clustering approaches are unsupervised by design and meant to be. What do they cluster if they do not exploit patient attribute information? This part needs clarification. At this stage should define what data you are using and how you are segmenting it (patient attributes (disease, age, sex etc., other data (diagnostics?)).
- Figure 2: 
	- Need to explain what the attributes are, what sinus rhythm/Atrial fib./ Bradycardia are (disease, disease class etc. for non domain expert. (later one learns of 5/6 disease classes used). 
	- Is that a "dual" repulsion line towards the triangles of atrial fib.? 
	- For the Similarity probability mass, can you provide the domain knowledge supporting it? Why/Is gender/sex that more indicative of cardiac signal discrepancies? How do the triangle/+ symbols used relate to the age, disease class attributes?
 
Notes:
- Please introduce your acronyms at first use to avoid confusions. 
    - Example: Electronic Health Record (EHR) data. 
- "... a centroid groups together instances that share some similarities.": 
    - Isn't the centroid rather a single point used to represent a group of instances that share some similarities?

After rebuttal:
-----------------
Thanks to the authors for their responses.
After reading the rebuttals, many unclear points were explained. 
I am still not convinced by the adequacy of the chosen experiment attributes and the generalization power of the approach. 
Therefore, I am raising my evaluation to "Marginally below the acceptance threshold". 
Regards.
limitations_and_societal_impact: No specific ones come to mind.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work presents a framework for information retrieval and clustering of patients based on their ECG’s. To do this, the authors framed the problem as a supervised contrastive learning problem – this is a very hot topic in machine learning for healthcare and I believe the framework and experimental design they propose is technically appropriate. 
main_review: Overall, my opinion was that the work was technically sound, and my primary concerns are about novelty and the chosen benchmarks for comparison. If these two points can be clarified I believe this can be a suitable paper for this venue.

I had trouble differentiating the authors technical contribution and from prior work. I believe the authors have substantially built upon past work and have novel contributions but they should make it more clear what their key technical contributions are. I believe this is likely the key major point to evaluate this work for a venue like NeurIPS.

In line with this comment – the authors may want to break out a section from the clinical representation learning to include more of the recent contrastive learning papers. 

Figure 3 is confusing - what data were used? is this a result or part of the methods? (simulated?). The legend also isn't explained (e.g. sttc). 

I believe there are better metrics to show in table 1 (e.g. traditional classification metrics + confidence intervals). 

The performance of examples such as SeLA, DC and IIC is so bad it’s likely they were not appropriately adapted to the problem – can this be explained? Are there better apples to apples comparisons? This is especially true given that with the supervised component, we know there are FDA approved devices with arrhythmia classification (AliveCor). I'm not convinced these are the best examples to compare to.

Why were methods removed transitioning from table 1 to table 2.  

Minor Comments:
I found it strange that the information retrieval task as introduced first throughout, but then the clustering results were shown first. 

I think you need to be careful about using the term clinical prototype, as it has a collision with the medical devices space. Given this work is using ECG data this will be confusing to the clinical audience. I admit, this may be a subjective preference as I’ve never liked the term cluster prototype, because it changes the traditional definition of the word prototype (a first instance other are copied from). 


limitations_and_societal_impact: The authors acknowledge several key limitations, mostly the result of the data that were available to them. I think the authors should discuss several other key limitations: 1.) in the evaluation of the methods, particularly a.) the re-implementation of prior methods, and b.) clinical evaluation of the information retrieval task. It should be straightforward to test whether a clinician believes the information retrieval matches their training (by showing a matched ECG and a further ECG and testing whether they identify the matched ECG). If they do not, these examples can be further explored – either the clinician is wrong, or the model could be improved. 2.) the lack of external validation or transfer to alternate datasets (e.g. training on Chapman and application to PTB-XL - even if only for Age, Sex)

This work has a potential to contribute to important societal impacts, particularly around fair machine learning for health. It may allow researchers to study underrepresented groups that are not clearly defined by binary categorizations of race, sex etc. but will need to be further studied to demonstrate this.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper proposes a clustering and information retrieval algorithm for unannotated large databases of physiological signals (in this case ECG signals). The method is based on a contrastive learning approach, which has recently been suggested for semi-supervised learning on ECG data.
The proposed approaches are evaluated on two large ECG datasets and compared with state-of-the-art approaches.

main_review: The paper is really innovative with two new applications (clustering and information retrieval) based on a recently suggested representation learning approaches.
The research is quite significant, given the availability of large databases of EHR data, containing clinical physiological signals (for instance the MIMIC III database). Applying such approach on such a dataset, with many meta-data available, could help uncover specific clinical protoytpes of physiological signlas and be of the utmost importance.
The paper is clearly written and of good quality. The proposed approach is tested in several settings, using two available large sets of ECG signals, and the technique performance is compared to other state-of-the-art approaches.

limitations_and_societal_impact: The main concerns are the following:
1.	The explanation of using soft assignment instead of har-assignment is discussed in the description of the methodology, but it is unclear to me whether the authors have reported an experiment allowing to prove their conjecture.
2.	The authors should discuss the fact that CP CROCS seem to be underperforming the other clustering approaches based on the age attribute. This result is quite strange and an outlier, but it would be interesting to have a discussion on a possible explanation of this phenomenon.
3.	Could the authors discuss the main difference of the tow datasets, and especially as depicted in figure 4 why the separability of the classes is so much lower on PTB-XL compared to Chapman
4.	IN terms of quality of the information retrieval, cold the authors comment why evaluating soft assignment (and performance if only one attribute is correct) is clinical significant? It seems to me quite important that pathology and should be well retrieved, extracting an example of AF when asking for Normal rhythm, would have higher impact than a mistake on age or sex
5.	Can the authors discuss whether age and sex are good attributes for ECG signals? Heart Rate variability is affected by age, but I am not aware of morphological changes in the ECG signals due to age (or even sex)? Figure 3 seem to be indicating that the representation of the ECG signal is not that influenced by either age or sec, although there seem to be a trend or evolution with age, and separation of sex on the proposed full framework.
6.	The order in sections 2 related work and 3 background should be consistent clustering then IR.
7.	The acronym HER is not introduced properly, that is at the first appearance of the term
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: Cardiac signals when unlabeled are difficult to query for given a set of attributes of interest such as disease class, age and sex.  The paper proposes a cluster-based retrieval system for this purpose. The algorithm proposed is of two-fold - (1) Learn a function to map a cardiac signal to an embedding closer in representation to a clinical prototype (embedding representative of a unique attribute set)   - this is done using a contrastive-learning framework assuming an initial-set of labeled data where cardiac-signals are annotated with class, age and sex and (2) Given a query attribute set - use its clinical prototype embedding to rank the  cardiac signals. Here the function from step 1 is used to get an embedding representation for cardiac signals and then their Euclidean distance from clinical prototypes is considered for ranking in the retrieval. The paper evaluates clustering and retrieval tasks on two datasets Chapman and PTB-XL.
main_review: Briefly my overall review was primarily based on the following - the ideas presented in the paper are interesting to me, however I found that the setting considered is unrealistic and the evaluation provided is insufficient.  Please find my detailed review below - 

There are several things to like about the paper,

Originality

(1) I liked the idea of extending multiple attribute information into unique clusters with their own clinical prototypes.

(2) I liked the regularization term to introduce semantic meaning - i.e prototypes representing similar attribute sets (e.g., adjacent age groups) should be closer to one another. The way this is modeled in equation 3 (using hamming distance of original attribute sets for supervision) is very interesting to me.

(3) Overall I found the paper easy to read (except the description on retrieval setting which I think can be illustrated with an algorithm and I think the intuition of the metric (in equation 5) should be discussed)

There are several shortcomings as well -

Originality

 (1) I think the setting considered is slightly unrealistic. Typically the number of unlabeled data is orders of magnitude higher than the labeled data in such clinical retrieval tasks. Also the unlabeled data can be subjected to distribution shifts as compared to the labeled data distribution. Owing to both these difficulties existing literature has not explored the direction of using supervised clustering  (as this paper did) - but rather tried weak-supervision or unsupervised techniques of clustering.  This changes the setting in several ways - (a) In the paper the retrieval evaluation is done on the validation dataset (20%) which is smaller than the training data (60%). This will change the retrieval metrics - meaning the metrics reported in the paper can be overestimating the actual performance (b) Supervised learning may not be an ideal framework any more and semi-supervised learning would need to be considered to address the above challenges. 

(2) How will the algorithm perform as we decrease the labeled data and increase the unlabeled data? Can you provide retrieval and clustering metrics for the same and also compare this approach with semi-supervised/self-supervised clustering algorithms? (currently retrieval metrics are only compared against DTC a supervised learning method.)

(3) Although the application considering multiple patient attributes for retrieval is new - this comes with several limitations that needs to be discussed in the paper (a) the complexity of algorithm or the clusters considered will increase exponentially with increase in the attributes and the number of values taken by each attribute. This will also limit the amount of discretization that can be done in attributes such as age.  (b)  This will also mean assuming sufficient labeled data covering all these attribute information.

Quality/Clarity - 

The retrieval metrics (equation 5) are not reflective of precision at K and are much relaxed versions - the metric is not average P@K across all M clusters (or queries) but rather average of the times P@K is non-zero across the M clusters. Can you provide a discussion on this metric?

I have several questions - can you please help by answering them?

(1) In figure 3 - Is this representative of the real data experiments or is this a cartoon to visualize the algorithm? If it is indeed a zoomed-in figure for experimental data - please provide more details. If it is not can you validate that the model has learnt semantically meaningful representations?

(2) In Figure 4 right image - points from following classes - HYP and NORM seem to be indistinguishable. So is a part of STTC and MI. Considering this the accuracy reported in Table 1 (76%) seems a little high especially considering HYP and NORM are majority classes.

(3) Does DTC in Table 1 really have a standard deviation of 15.0 in Accuracy and 20 in AMI? How is this possible - please provide an explanation?

Significance -

It is difficult to evaluate the significance of the paper considering insufficient evaluation and unrealistic setting as compared with existing work.

limitations_and_societal_impact: The paper discusses some of the limitations of the algorithm. Please find more detailed comments above to extend on this.
review_title: null

Meta-review: This presents a well written paper which presents a framework for information retrieval and clustering of patients based on unannotated ECG/ physiological data . In discussion, there were mixed reviews with regards to novelty, experimental evaluation and baseline comparisons to related work. However, after evaluating the paper and the discussion and reviewers' comments,  I agree that the proposed approach is presents a supervised technique for learning a compact representation which respect the attributes (or weak labelling) available. Such a representation could be used for other task for which access to a large database of (strong) annotated data is quite difficult. This is a non-trivial task in the healthcare domain and highlights an important application of ML for real-world impact, which is potentially generalisable to other data settings. Although there were concerns around over-fitting this model to a very stringent problem, the novelty of the approach and problem formulation from an ML perspective far outweighs any potential downsides. As both the authors and one of the reviewers highlights, this work may have a potentially positive impact on multitude of stakeholders within healthcare and could inform researchers working on fairness and thus better serve patients from underrepresented backgrounds.
Decision: Accept (Poster)