Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: ##########################  
Summary:

This paper proposes to tackle the problem of retrieving and clustering physiological signals by learning clinical prototypes via supervised contrastive learning. Three readily available patient attributes, disease, age, and sex, are used to assist the learning. A hard assignment of samples to prototype is proposed, and further relaxed to a soft one to utilize the samples that do not have exactly matched attribute set. A regularization term is also proposed to encourage intra-cluster distances. Two ECG datasets are used to evaluate the proposed model.

##########################  
Pros:
The target problem to tackle is important and the idea of using prototype learning approach is interesting.


##########################  
Cons/Questions: 
1. Overall, the writing can be improved. I find several places confusing and many details are not fully described. For example, in Section 3.1, "To that end, we propose to learn a set of embeddings, $p\in P$", what is space $P$?
2. It's unclear to me what needs to be learned, $p$ or $v$, or both?
3. In Eq (4), what is $\text{class}_i$? is it already known?
4. In Eq (8), the $\delta(\cdot)$ function is undefined. Is it the indicator function?
5. For the retrieval task, why the CPs are used as the query set to retrieve instances from the held-out set, instead of the reverse (given unseen sample as query and retrieve samples from database)?
6. For the retrieval task, it is unclear to me if the attributes are available for the held-out set. If yes, then the evaluation metric of the retrieval task seems weird to me: If the task is to simply retrieve samples with similar attributes, it is straightforward to just query the attributes and just retrieve the samples with a maximal number of matched attributes. Why do we need to learn embeddings for such task? If the attributes are not available, what is the clinical motivation for retrieving such instances?
7. Usually the ECG signals are processed with CNN or RNN models, it's unclear if and what models are used by the authors to generate the feature $x$ for the input ECG.

review_title: Missing many details and flawed retrieval experiment design

final_score: rating: 4: Ok but not good enough - rejection
review_content: ##########################################################################
Reasons for score: 

The authors propose a novel contribution of designing a contrastive learning based attribute specific retrieval system for physiological signals, but the experiments in the paper aren’t sufficient in showing the proposal’s utility relative to appropriate baselines for the two tasks specified ( IR and clustering ) and  don’t discuss the prior work the physiological signal representation mechanism is based on which is central to the work.  

The work could be greatly improved by clarifying some important aspects of the proposed system and by conducting more appropriate experiments to back the result claims made.

 
##########################################################################
Pros:
1) The authors propose a novel contribution of designing a contrastive learning based attribute specific retrieval system for physiological signals  and do a fairly good job of showing how it differs in regards to existing works ( with the exception of 1 or 2 important ones )
2) The contrastive learning approach of the contrastive-soft loss is motivated and explained well and shown to improve 
upon solely using K-Means on representations learned via the L_contrastive−soft + L_regression  combined loss objective.
 
##########################################################################
Cons: 

1) There are no comparisons for how the system performs in the IR case relative to other works and the baselines for the clustering case are not appropriate ( as they are all unsupervised ) and this method explicitly needs supervision via attributes to derive attribute specific prototypes for learning.
	
   Also, in the retrieval section the use of R@K doesn’t seem appropriate as used.
   Usually R@K means does your result (document / entity / etc ) appear in the top K results, but here its different.

   For instance, if we look at the sex attribute, we have a set of M total attributes one per prototype (either 32 or 40) that is split 
   evenly between male and female [ M, F, M , F … ]  =  [a1, a2, a3, a4 .. aM ] for all M combinations of attributes.
   Given my understanding of the formulation in 4.2, we take a particular CP (lets say its the 15th of 32 ), then find its K nearest neighbors by euclidean distance, and then compare the true sex of those K returned neighbors with that of the CP.
   However this “sex” attribute is either M or F and doesn’t pertain to the exact sex prototype of CP 15 ( ie, the task isn’t find the correct 1 out of 32, but rather find any of the correct 16 out of 32 ).. Is this correct?  
   
   In the medical field, the need for exact matches seems like it would preclude just expanding the number of returned items K in hopes of finding one correct results especially when there is a simple possibility for querying with structured data anyways ( ie filter signals by age / sex / class first )  and then just use retrieval of similar signals that way.  Is that incorrect? 
	
    In section 5.1 60% at K=5 is hard to assess for quality without comparing it against other retrieval methods.  

    Specific to the clustering section, the external baselines described are all unsupervised and not directly comparable to the 
    supervised DROPS or K-means Combined methods as such.  Effectively the work shows that doing k-means on representations learned by the paper is not as good in terms of ACC/AMI as compared with those learned from two proposed proposed variants of contrastive learning.  

  The two proposed variants of attribute specific contrastive learning are interesting, but hard to size up against other methods.
    For instance  taking latent space reps from the  autoencoder + prototypical classifier network  from Gee et al19 and comparing how K-Means on those representations does with K-means Combined or DROPS would be directly comparable and appropriate ( there are probably other supervised learning methods to construct representations that you could use from the prior works section as well ).


2) Kiyasseh et al., 2020 “ Clocs: Contrastive learning of cardiac signals”  is not mentioned in the prior work section and mentioned once in section 4.5 to discuss the use of a hyper parameter.  
	This work seems quite similar in the methods, datasets, etc used particularly in regards to contrastive learning of cardiac signals for representation learning so in the background section the authors should clarify and say if this is an extension to that work or explain the differences. 

3) Points that were unclear to me:  
	3.1 The construction of D_hat and the ground truth D in section 3.3 in “Arrangement of clinical prototypes” .  It seems that in order to regularize with respect to the CPs, in order to calculate pairwise distances between them to make Dhat, you need to be given the prototypes initially ( otherwise where does the ground truth D come from ).  Its very possible this is a misunderstanding on my end, but this section could have been clearer.  

       3.2 Similarly, is C set to 4 and 5 for the two datasets specifically or are they set to 8 in both cases ( sex |2| x age |4| = |8| )?  The wording left that a little unclear to me and specifying that in section 4.5 would resolve that. 
          
	3.3 How are the "v" representations and prototypes actually learned?   The architecture given in section C of the supplement outputs an E x C, which seems its intended for the prototypes, but its not entirely clear.
          
       3.4  There are two different architectures used for the Chapman and PLB-XL datasets and the reason given is the size of the later, 
            but the first has ~10k examples ( each of size 2500 ) and 4 classes whereas the later has 18k each with size 2500 and 5 classes so they seem to be relatively similar in size no?   

4)  The work Gee et al 19 “Explaining Deep Classification of Time-Series Data with Learned Prototypes”  ( ICML 19 https://arxiv.org/abs/1904.08935 ) seems that it should possibly be cited in the prior works section since it learns prototypes on clinical signals and shows how the t-sne representations of the prototypes lie with respect to the learned latent representations of the patient clinical signals.  The work doesn't explicitly describe a retrieval mechanism, but it does assign classes to the prototypes by finding the nearest neighbor in euclidean distance over the training set and giving the prototype its class which is similar in nature to the retrieval mechanism proposed here ( though with a different and non-contrastive learning mechanism ).

5)  If the Beta parameter is distance metric dependent then it would be useful to show an experiment showing what occurs in terms of ACC / AMI by adjusting this parameter to justify setting it to .2.

6) Table 1 this table should really be  split to be attribute specific  ( have additional columns for age / sex / class for each K=1 , K=5 and K=10 for the two datasets)
    It seems like the approach would do best at the class task and then do worse ( as is 5.1 ) at getting the sex or age attributes, but without that information we can’t discern this for the retrieval task ( though its shown for the clustering task in Table 2 ) 

7) In Table 2 , the Combined method doesn’t improve performance in (a) or (b) for the PTB-XL case and it would be useful to hear discussion on that.

8) Below Table 2, I disagree with the statement “Note that this approach does not yet exploit clinical prototypes” since the training setup which learns these representations in fact does leverage CPs for learning purposes  since they are explicitly used in the contrastive soft loss objective.

9) K-Means could be used to provide clustering by different attributes ( though possibly not simultaneously ) and adding that to Table 2 (b) could help elucidate if its advantageous or not to learn the clusters simultaneously in fact.  

10) In Section 5.3, the contrastive learning representations are separated pretty well in the Chapman set, but it seems to a much lesser extent in the PTB-XL data.  Why is that?  
   What does plotting the k-means centroids of these representations with T-SNE give?   
   How would this differ from setting K=32 ( for the Chapman case ) and then using the average of the representations fo each 
   example in a specific attribute combination grouping ( sex - age - class ) as the centroid for each K?  
       
##########################################################################

Questions during rebuttal period: 
Please address and clarify the cons above 
 
#########################################################################

review_title: This work proposes to use a contrastive learning method to learn attribute specific prototypes of clinical physiological signals and show their utility in the tasks of information retrieval and clustering  while also showing these learned representations capture attribute specific semantic relationships .

final_score: rating: 2: Strong rejection
review_content: The manuscript focuses on clinical natural language processing of electronic health records. More precisely, it addresses a text classification task called information extraction or named entity recognition from these clinical records. Its contributions include developing an embedding model to capture clinical prototypes (CPs), via supervised contrastive learning, and presenting experimental evidence of these learnt CPs capturing attribute-specific semantic relationships and being helpful in subsequent clinical natural language processing task of information retrieval and clustering of clustering of physiological signals. I find this text processing methodology interesting, carefully described, and supplementary to other studies.

However, unfortunately, the authors demonstrate limited understanding of the related literature. First, the second and third paragraph of the Introduction section have many sentences that require references to be inserted. Second, and more importantly, the Related work section seems to not capture the key papers and trends of the field (I suggest reading some systematic reviews or surveys on clinical natural language processing, information extraction, and information extraction by, for example, Wendy Chapman, Carol Friedman, and Pierre Zweigenbaum), and, for example, as illustrated by the ImageCLEF and CLEFeHealth evaluation labs, computer vision tends to proceed faster than text analytics (see, e.g., https://www.researchprotocols.org/2018/7/e10961/), and not the other way around as claimed by this section.

In addition, to feel convinced of the presented experimental evidence, I would have wanted to see statistical significance tests, confidence intervals, effect sizes, or similar presented. I could not find this methodology described in the manuscript or its outcomes, although the narrative repeatedly referred to significant performance gains. Please clarify.

As my main minor comment, I would like to see a clearer separation of the materials, methods, and experiments sections from results, as well as including clearer justifications of this study design. For example, the aforementioned significance topic has not been addressed sufficiently. Another illustration of somewhat difficult task for the reader is to understand the experimental design as a whole and be convinced of this study being rigorous is the Experimental Results section including a lot of methodological details as opposed to only obtained outcomes.

I also suggest including a conclusion statement as well as embedding more evidence (e.g., evaluation materials and methods plus obtained indicators of performance gains, such as measure values and effect sizes) to convince the reader in the abstract. To continue, please remember to punctuate equations and formulae. 

Finally, typically the Methods, Experiments, and Experimental Results sections would have been written using a past tense to emphasise a finished (as opposed to an ongoing) study where materials, methods, and experiments have already been chosen, justified, and completed. Most importantly, please avoid having inconsistent tense in these sections (see, e.g., Section 4.5. using a past tense whereas almost all others are having a present tense).
review_title: Review of the submission called DROPS: Deep Retrieval of Physiological Signals via Attribute-specific Clinical Prototypes

Meta-review: This paper proposes to learn clinical prototypes via supervised contrastive learning to facilitate the reliable retrieval of clinical information and clustering in large datasets. The presentation of the paper could be substantially improved – e.g., the overview and motivation of the paper, the definition of clinical prototypes, selection of certain evaluation criteria, clarification of terminology in equations, the description of the motivations and settings of the experiments, etc.  In addition to the need to substantial improvement in clarity, major concerns include lack of comparison with more supervised approaches and discussion of relevant literatures raised by reviewers.  
Decision: Reject