CROCS: Clustering and Retrieval of Cardiac Signals Based on Patient Disease Class, Sex, and Age
Authors: Dani Kiyasseh
Tingting Zhu
David A. Clifton
Venue: NeurIPS 2021 Poster (2021)
Keywords: supervised contrastive learning
cardiac signals
clustering
retrieval

Abstract: The process of manually searching for relevant instances in, and extracting information from, clinical databases underpin a multitude of clinical tasks. Such tasks include disease diagnosis, clinical trial recruitment, and continuing medical education. This manual search-and-extract process, however, has been hampered by the growth of large-scale clinical databases and the increased prevalence of unlabelled instances. To address this challenge, we propose a supervised contrastive learning framework, CROCS, where representations of cardiac signals associated with a set of patient-specific attributes (e.g., disease class, sex, age) are attracted to learnable embeddings entitled clinical prototypes. We exploit such prototypes for both the clustering and retrieval of unlabelled cardiac signals based on multiple patient attributes. We show that CROCS outperforms the state-of-the-art method, DTC, when clustering and also retrieves relevant cardiac signals from a large database. We also show that clinical prototypes adopt a semantically meaningful arrangement based on patient attributes and thus confer a high degree of interpretability.

CROCS:
Clustering and Retrieval of Cardiac Signals Based on Patient Disease Class, Sex, and Age
Dani Kiyasseh, Tingting Zhu, & David Clifton
Department of Engineering Science
University of Oxford Oxford, UK
{dani minaiskiysseh, tingting
wwwpuniinggnguu.iai zhu, david. wwwftllnieeeeeeeoo teentennnnnneeeeeeeeeeeeeeeeeete oxpenngox.... ox. ac.uk
Abstract
The process of manually searching for relevant instances in, and extracting infor- mation from, clinical databases underpin a multitude of clinical tasks. Such tasks include disease diagnosis, clinical trial recruitment, and continuing medical educa- tion. This manual search-and-extrac process, however, has been hampered by the growth of large-scale clinical databases and the increased prevalence of unlabelled instances. To address this challenge, we propose supervised contrastive learning framework, CROCS, where representations of cardiac signals associated with a set of patient-specific attributes (e.g.. disease class, sex, age) are attracted to learnable embeddings entitled clinical prototypes. We exploit such prototypes for both the clustering and retrieval of unlabelled cardiac signals based on multiple patient attributes. We show that CROCS outperforms the state-of-the-arr method, DTC, when clustering and also retrieves relevant cardiac signals from large database. We also show that clinical prototypes adopt a semantically meaningful arrangement based on patient attributes and thus confer high degree of interpretability.
Introduction
Clinical databases comprise instances that are either unlabelled or labelled with patient attribute information, such as disease class, sex, and age. The process of manually searching for relevant instances in, and extracting information from, such databases underpin a multitude of tasks [[]. For example, clinicians extract disease diagnosis from patient data, researchers involved in clinical trials search for and recruit patients satisfying specific inclusion criteria (2). and educators retrieve relevant information as part of the continuing medical education scheme 3. This manual search-and-ed -ndtrxt process, however, is hampered by the rapid growth of large-scale clinical databases and the increased prevalence of unlabelled instances; those for which patient attribute information is unavailable.
Given such a setting, in this paper, we are interested in addressing two questions: given large, unlabelled clinical database, (1) how do we extract altribute information from such unlabelled instances? and (2) how do we reliably search for and retrieve relevant instances? To address the former, the task of clustering holds value. In this setting, a centroid groups together instances that share some similarities. Recent research has focused such as k-mmans, to group similar patients from electronic heath record (EHR) data [ 5) Such methods, however, are exclusively unsupervised; they do not exploit patient attribute information. To address the second question, the task of information retrieval holds promise. this setting, a query associated with set of desired attributes is exploited retrieve a relevant instance. Recent research has focused predominantly on retrieving medical images (6. clinical text 77), and EHR data g. with minimal emphasis medical time-series data These methods do not extend to cardiac
exploiting existing clustering algorithms,
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

time-series data nor do they account for search based on multiple patient attributes. Most notably, previous work performs either clustering or retrieval, and not both.
In this work, we address both questions while exploiting large-scale electrocardiogram (ECG)
databases comprising patient attribute information. Our contributions the following: (1)
propose a supervised contrastive learning framework, CROCS, in which we attract representations of cardiac signals associated with unique set of patient attributes to embeddings, entitled clinical prototypes. Such attribute-ppecific prototypes, which create "islands" of similar representations [T
allow for both the clustering and retrieval of cardiac signals based on multiple patient attributes. (2)
We show that CROCS outperforms the tate-of-the-aa method, DTC, in the clustering setting and retrieves relevant cardiac signals from a large database. At the same time, clinical prototypes adopt a semantically meaningful arrangement and thus confer a high degree of interpretability.
2 Related work
Clinical representation learning and clustering Learning meaningful representations of clinical data is an ongoing research endeavour. Recent research has focused on learning representations from 14 [5] and via auto-encoders, which are then clustered using existing methods, 4 5). As for time-series data, auto-encoders are learned with [16] or without [17] an auxiliary clustering objective, salient features (shapelets) are identified in an unsupervised manner
EHR data [ 12
13
such k-means
[1 [9]. and vatient-specineg representations are learned via contrastive learning [20] Li et al. (21)
learn prototypes, or representative embeddings, via the ProtoNCE loss and cluster instances using k-means. Their work builds upon recent research in the contrastive learning literature [22 23 24).
Similar to our work is that of Gee el al. 11 where prototypes are learned for the clustering of time-series signals. Their prototypes, however, cannot cluster instances based on multiple patient attributes and do not extend to the retrieval setting.
Clinical information retrieval (IR) Retrieving clinical data from a large database has been a longstanding goal of researchers within healthcare 255 Such research has involved the retrieval
of clinical documents 226 7 27 (8) where, for example, Avolio et al. [29] map text queries to an
ontology known as SNOMED, before retrieving relevant clinical documents. Recent research has
focused on the retrieval of biomedical images [6, 30), and EHR data [31 32 to discover patient cohorts in clinical database [8) Goodwin al. 9 implement unsupervised patient cohort retrieval system by exploiting clinical text and time-series data. These approaches, however, do not explore cardiac signals, cannot account for multiple patient attributes, and unable to also cluster instances. To the best of our knowledge, we are the first to design a learning framework that allows for both the clustering and retrieval of cardiac signals based multiple patient attributes.
Background
Supervised clustering We learn a function, fo x E R S E R parameterized by 0, that maps a D-dimensional input, x, to an E-dimensional representation, v. We also have labelled dataset, D, = Az),A,)) m where each instance, 2G, associated with a set of discrete patient attributes, A; {a a" ) where a, disease class, a₂ sex and a, = age. Supervised clustering can involve learning M < Nı centroids with each representing a unique set of attributes, (aL, {g) { and grouping similar instances together. Given unlabelled instances, {xa) 1N the centroid closest to each representation, vn = fo(x.). is used to infer the latter's attributes. In this work, we learn cluster centroids which are more formally introduced in Sec. 4.1
1M
Information retrieval IR involves searching through large, unlabelled dataset, {xu 1. 1. and retrieving a relevant instance, Fu- However, relevance, defined based on whether an instance satisfies some criteria, is difficult to ascertain when instances are unlabelled. Typically, query in the form of an embedding which represents a desired set of attributes, A,, retrieves its closest (and most relevant) representation, V. [o(w),,, and infers the latter's attributes. In this work, we learn set of query embeddings. As will become apparent in Sec. 5] these embeddings can also be treated as centroids, like those outlined in supervised clustering, and will thus serve a dual purpose.

---

Methods
4.1 Attribute-specific clinical prototypes
We aim to learn embeddings, referred to as clinical prototypes, that can be exploited for both the clustering and retrieval of cardiac signals based on multiple patient attributes. In the clustering setting. the goal is to annotate unlabelled instances with a set of patient attributes. To that end, we exploit clinical prototypes as centroids of clusters which such unlabelled instances are assigned (see Fig. left). In the retrieval setting, the goal is to retrieve unlabelled instances based on set of patient attributes. To that end, we exploit each clinical prototype as query to search through an unlabelled database and retrieve instances to which it is most similar (see Fig. rriht).
In designing clinical prototypes. we take inspiration from the field of natural language processing (NLP) where Icarnable word embedding represents a unique word. In our case, each prototype represents unique combination of discrete patient attributes. Formally, given the attributes, dg. and Oas we would have M |ozl x las11 x |oal such unique combinations denoted by 1-=1- We associate each combination, A₁ E A, with learnable prototype, PA, R for a set of M prototypes, P {PAy)/ 1=1- Note that this framework extends to any number of attributes. In the next section, outline how learn these clinical prototypes.
fae, ag, a f-
Clustering
Retrieval
assign
clinical prototypes
search
unlabelled data
sinus rhythm male under 25
clinical prototype
wnlabelled data
4
rhythm
male under
extract
retrieve
Figure Clinical prototypes are exploited for attribute-specific clustering and retrieval of cardiac signals. For clustering, we exploit prototypes as centroids of clusters to which unlabelled instances are assigned. Such assignment is associated with set of attributes, such disease class, sex, and age. For retrieval, we exploit each prototype query, associated with set of attributes, to search through an unlabelled database and retrieve instances which it most similar.
4.2 Learning attribute-specifie clinical prototypes
Clinical prototypes will serve dual purpose of suribute-speciinn clustering and retrieval. As such, prototypes will need to be in proximity to a subset of representations of instances associated with a specific of patient attributes. To achieve this proximity, we leverage the contrastive learning framework which involves a sequence of attractions and repulsions, explained next.
Hard assignment We encourage the representation, e; = fo (m:), of an instance, ma associated with set of attributes, A; E A, to be similar to the single clinical prototype, PAy that shares the exact same set of attributes (i.e.. A; A,), and dissimilar to the remaining clinical prototypes, {PAx Jaye. To achieve this, we optimize ENCE-hord for mini-batch of B instances (Eq, [ Intuitively, heavily penalizes the leamer if less probability mass is placed on the similarity of us and PA, than on the similarity of other representation and and- poottrottyyee pairs. We choose quantify the cosine similarity
of pairs, s(V;, PA,), alongside a temperature parameter, T., as is done by Kiyassch et al. (20).
PtPPA, 1
LNCE- -ard- = B 1=1
log
8(v..
(I)
10:
PA,
T
We refer to this many-to-one mapping from representations to clinical prototype as a hard assignment. Such an assignment, however, implics that prototype is unlikely extract potentially useful

---

information from a representation whose attributes are not exactly the same as those of the prototype. We quantify this limitation in Sec. 6.4 and propose an alternative assignment next.
Soft assignment To overcome the limitation of a hard assignment, we encourage the representation, Vi, to be similar to a subset of clinical prototypes, C P (see Fig. 2). We must take caution, however, to avoid erroneously attracting representations clinical prototypes from a different class. Doing so would reduce class-specific margins and thus hinder the downstream clustering and retrieval tasks,
ECG Attributes
attraction
Clinical
male
Similarity Probability Mass
Key
Force
Extractor
Features
repulsion
Protutype
female
discase class
X1
fe
Vi
discase class 2
disease
A
male
disease class
under
Figure 2: Clinical prototypes are learned via supervised contrastive learning framework referred to as CROCS. The representation, tt,, of an instance, Dg associated with a set of attributes, A, is strongly attracted to the clinical prototype which represents the same attributes, weakly attracted to others within the same disease class (colour), and repelled from those representing different classes. These attractions result in the shown similarity probability mass function. To avoid clutter, we have omitted the age attribute associated with clinical prototypes.
Uniform attraction. We chose the subset to include prototypes, L L= (PAi) 1|1 that share the same disease class, of with the representation, Ug, implying that A E {a a², aaa/=1 C. Note that the clinical prototypes in the subset, L, continue to represent attributes that vary along the dimensions of patient sex and age (a,, Qa)). Therefore, attracting i; to prototypes in L uniformly will likely cause the latter to become minimally distinguishable across sex and age. This is an undesired outcome in light of our goal of learning atfribute-specifi prototypes. We avoid this behaviour by modulating the degree of attraction between iú and all prototypes in the set, P, as outlined next.
Modulated attraction. The attractive force between 8i and PA, is reflected by the corresponding LNCE term (Eq. [.. By placing less probability mass on s(V;, PAy) (i.e., less similarity) than on 8(v5, PA.) Vk. F j. the learner incurs a higher loss and thus attracts the pair. We extend this logic all prototypes obtain M -had- terms per representation. To modulate these attractions, introduce a weight, Wij E {wij 1= as a coefficient of the j-th loss term (Eq. 2). Each weight, waj- quantifies the degree of matching between attributes of the representation, A, {a a oz J. and those of the clinical prototype, A, = {or, ay ). as reflected by q( A,) E R. We define as the indicator function and Tw temperature parameter that determines how soft the representation-ant prototype attraction is. For example, as Tu - oo, this approach reverts to the uniform attraction setup. The intuition is that stronger attraction (1 way) should exist for a representation-ande ppooottt pair that shares more attributes. We also avoid the erroneous attraction of pairs from different classes (i.c., of A a²) by setting Wiy O. When visualizing the UMAP projection 33 of prototypes learned with a uniform attraction (T. = 20) (Fig. B left) and those learned with a modulated attraction (Tw * oo) (Fig. centtre), we show that the latter become more linearly separable across sex.
B
M
s(Ui.Da) Σ M calue.pe))
LNCE- =
Wij log
aaAA Σ [. culAziAgT
if o = ag otherwise
Wij

---

1 [1 o.) 1 o².) + 1(a = ag)
AA,,, A,)= =
Tw
Arrangement of clinical prototypes Clinical prototypes would confer high degree of inter- pretability if they also captured the semantic relationships between attributes. Concretely, prototypes representing similar attribute sets (e.g.. adjacent age groups) should be similar to another. This is analogous to the high similarity of word embeddings representing semantically similar words [34]. To capture these semantic relationships, we encourage class-specific prototypes to maintain some desired distance between one another. As such, each pair of M clinical prototypes, PA,, PAs Vj. k E [1, M] associated with an empirical and ground-truth (desired) distance. For the former, normalize the prototypes (I2 norm) and calculate their Euclidean distance, dyx- |IAA, PAull2 Vj, E M.. For the latter, we define the ground-truit distance as dj. = B x dæ E R, where d Ax) z is the Hamming distance between pair of discrete attribute sets. Intuitively, the Hamming distance counts the number attribute mismatches and 3 E IR penalizes each mismatch. Therefore, we can generate an empirical sct, {dzx ) and ground-truth set, {dzt) [d of distance values. By minimizing the mean-squared error between these two sets, we learn clinical prototypes that adopt semantically meaningful arrangement (see Fig. righht) Since we are only interested in adopting this arrangement for prototypes the same class (i.e., of = ox)), we incorporate the regularization term, Lreg. into the final objective function, Liot.
2,k=1
M
Lreg Σ J,k=1
(dge ddx)²² + 0lc = =
Liot = LNCCE-SOft + freg
(3)
LNCE-soft
To
LNCE-Soft
To #
LNcE-soft + freg
dl55
STTC
NORM
HYP
Figure 3: Projection of the set of clinical prototypes, P, learned with variants of the CROCS framework. These prototypes lower-dimensional space via UMAP 33 The legend indicates the attributes of sex (colour), age (size of marker), and disease class (marker symbol). With -NCE-soft -ooft Tw oc, representations attracted to class-specific prototypes uniformly. With LNCE-soft & oc, prototypes become more linearly separable across sex. Our full framework, LNCE-soft Creg. leads to prototypes that adopt a semantically meaningful arrangement. We investigate the marginal impact of these design choices on performance in Sec. 6.4
derived through empirical experiments and projected onto
Experimental design
Datasets We use Chapman 35 which consists of 12-lead ECG recordings from 10,646 patients alongside cardiac arrhythmia (disease) labels which we group into 4 major classes. 2) PTB-XL 36 consists of 12-lead ECG recordings from 18,885 patients alongside discase labels which group into 5 major classes 337. Each dataset contains patient sex and age information and is split, the patient level, into training, validation, and test sets using 60 20 20 configuration. Each time-series recording is split into non-overian non-overlapping segments of 2500 samples (== 5s in duration), as this is common for in-hospital recordings. Further details are provided in Appendix
1 Description of clustering setting During inference, we treat the clinical prototypes, {PA, Jj=1 as of cluster centroids. We calculate the Euclidean distance between the i-th representation and each of the M prototypes, identify the closest prototype, PAs and assign the representation to Aş which now denote by A, { at, a Repeating this process for N unseen instances results in
5

---

a set of assigned attribute values, = f f Ii=1, for particular attribute, á E A (e.g.. disease class). Such unseen instances would typically be unlabelled. For evaluation, however, we assume access to the ground-truth attribute values, oi = {a*) Ji=1 with which we calculate the accuracy, Acc(c), and the adjusted mutual information, AMI(()) E [0, 1]. between ā and ã.
N Σ
(M) = MI â) ((MI(( -M(MI.)
Acc(a)
1(a = aa)
(4)
(a)( H())) ((MI((,
where MI â) denotes the mutual information between the ground cound-truth and assigned set of attribute values, and (a)) denotes the entropy of the attribute values.
Description of retrieval setting During inference, we treat the clinical prototypes, (PA, } as a query set. We calculate the Euclidean distance between the j-th clinical prototype and representations of N unseen instances, retrieve the K closest instances, and then assign them to Aj = {az ag og Note that such instances would typically be unlabelled, thus precluding simple SQL search. For evaluation, however, we assume access to the ground-truie attributes, {a a',, a² [ with which we calculate variant of the precision at K metric (Eq. 5 It checks whether at least one of the retrieved instances is relevant, where relevance is based on a partial or exact match of query and instance attributes (# attribute matches). This value is then averaged across all M prototypes.
K
PeK-
1 M
Σ j=1
1
Σ 1 (1a2
n |a - ad) n [a - - 21
(5)
relevance attribute matches
Ideally, retrieved instance would match all of the query's attributes (discase class, sex, and age). In our context, however, the motivation for breaking down the evaluation of the retrieval setting based on the number of attribute matches is twofold. First, it allows to evaluate our framework granular level and thus detect subtle changes in performance. Second, the importance of such an evaluation the attribute level arguably depends on the clinical context. For example, cardiologist diagnosing a disease might be most interested in the pathology (i.e., disease class) attribute. On the other hand, pharmaceutical company looking recruit patients within particular age group for clinical trial might be most interested in the age attribute. Moreover, biomedical researchers looking to stratify treatment outcomes based sex would be most interested in the sex attribute.
more
Baseline methods We compare clinical prototypes learned via the CROCS framework (CP CROCS) to the following methods. For retrieval, Deep Transfer Cluster (DTC) 38 learns cluster prototypes by minimizing the KL divergence between a target distribution and one based on the distance between prototypes and representations. TP CROCS involves traditional prototypes where each prototype, UA₁- = ΣTÁ.aa, Σi=1 vi 1(A, = A₁), is simply an average of representations, U4. associated with the same set of attributes. Ay. Such representations are also learned via CROCS.
For the clustering task, we compare several state-of- the-art clustering methods, in addition to those mentioned above. k-means identifies cluster centroids based on the input instances, æ (KM raw), representations, learned via the CROCS (KM CROCS) explainable prototypes (KM EP) 11 framework. DeepCluster (DC) 39 iteratively applies k-means to representations, pseudo- labels them according their assigned cluster, and then exploits such labels for supervised training. Deep Temporal Clustering Representation (DTCR) [16] optimizes an objective function with a reconstruction, k-means, and classifier loss that determines whether instances real. Information Invariant Clustering (IIC) [40] maximizes the mutual information between class probabilities of an instance and its perturbed counterpart. SeLA [4] implements Sinkhorn-Knn -nopp to pseudo-label instances in a supervised manner. For further details, see Appendix D
During optimization, we chose the temperature parameter, т, 0.1 20), and Tu = 1. We specified B = 0.2, in the regularization term, after experimenting with several values (see Appendix [T). Too small a value of 3 would decrease the distance between class-specific clinical prototypes. Too large value of 3 would cause clinical prototypes from different classes to overlap with one another and thus reduce class separability. For Chapman and PTB-XI, sex E (M, F), age
Hyperparameters
converted to quartiles, and |class| = 4 and 5, respectively. Therefore, M cclass| X |sex x Jage

---

32 and 40, for the two datasets, respectively. The network, fe. comprises ID convolutional operators and we chose the embedding dimension 128 and 256 for Chapman and PTB-XL, respeciveil We show that E has a minimal effect on performance (see Appendix E. implementation details can be found Appendix C
Further network and
Experimental results
6.1 Visualizing clinical prototypes
We begin by qualitatively validating the claim that clinical prototypes are attribute-specific. In other words, can prototypes be delineated along the dimensions of disease class, sex, and age? To address this, we illustrate, in Fig. 4 two dimensional UMAP projections of the class-specific clinical prototypes (large, coloured shapes), traditional prototypes (large, black shapes), and representations of instances in the validation set of Chapman and PTB-XL.
We show that clinical prototypes are indeed disease class-specific, evident by the high degree of class separability of such prototypes. We also find that clinical prototypes are distinct from traditional prototypes, distinction whose importance will become evident in later sections. Second, the consistency of the class labels of the prototypes with those of the representations is a harbinger of how prototypes may perform in the clustering and retrieval settings, These findings complement the delineation of the prototypes along the dimensions of sex and age, and their adoption of a semantically meaningful arrangement, as was shown in Section
show in the next section.
STTC
HYP
NORM
SR
GSVI
Figure 4: Projection of class-specific clinical prototypes p (large, coloured shapes), traditional prototypes i, (large, black shapes), and representations, in the validation set of (left) Chap- man and (right) PTB-XL. We show that clinical prototypes are class-specific, consistent with the class labels of representations, and distinct from traditional prototypes. This bodes well for their use as centroids for clustering (see Sec. 6.2) and as queries for retrieval (see Sec. 6.3
6.2 Deploying clinical prototypes in clustering setting
In the clustering setting, we assign cardiac signals in a held-out dataset to set of patient attributes associated with the cluster of the closest clinical prototype. We evaluate these assignments based on the three patient-specific attributes (disease class, sex, and age) and present the results in Table
In Table [ find that CROCS outperforms both generic and domain-specific state-of-the-- art clustering methods. For example, on Chapman, CP CROCS, KM EP, and DTC achieve Ace(class) 90.3, 65.6, and 53.4% respectively. Along the dimension of sex, and on PTB-XL, CP CROCS and DTC achieve Acc(sex) 73.5 and 58.6%, respectively. Second, we find that CROCS leads to rich representation learning that facilitates elustering. This evident when compar- ing the performance of k-means applied to representations that are learned via different methods. For example, on Chapman, KM raw, KM EP, and KM CROCS achieve Ace(class) = 28.4, 65.6, and 73.4%, respectively. We also find that clinical prototypes, when exploited as centroids, are preferable to traditional prototypes, and centroids learned via k-means. For example, on PTB-XL, KM CROCS, TP CROCS, and CP CROCS achieve Acc(class) 47.6, 53.6, and 76.0%, respectively. These findings, which hold across datasets and evaluation metrics, point the overall utility of the CROCS framework and clinical prototypes for attribute- clustering.
7

---

Table 1: Clustering performance on the validation set of Chapman and PTB-XL. Evaluation is based on (a) class and (b) sex and age attributes. Results are averaged across five random seeds. Brackets indicate standard deviation and bold reflects the top-performing method. We show that CP CROCS outperforms the remaining methods regardless of patient attribute.
(a) Cardiac arrhythmia class attribute
Chapman
PTB-XL
(b) Sex and age attributes
Method
Acc
AMI
Acc
AMI
21.0 m.11
(.2
10.5 (1))
(0.5)
Method
Chapman
PTB XL
DC 59
21.0 m.1)
3.0
10.5 (.1))
4.9 m.)))
sex
age
six
ago
27.0 m2))
0.2:
22.0 (0:)
0.5 (.))
DTCR
(0.9
25. 9 (0.1)
51.0 (0.9)
25. 1 (0.5)
DTCR [0
11.3
0.2
38.4 (4.3)
1.4 (0:3
DTC 3
64.5
26.4 00.3
58.6 (1.9)
25. 7 (0T)
DTC B
23.4 (154)
67.3
(6.7
KM EP M
56.1 cum
31.0
54.1
29.2 (3.0)
KM ruw
28.4 (1.2
(0.0
KM CROCS
(4.9
32.3 (0.5)
51.8 (1.2)
31.6 (1.5)
KM EP 11
65.6 (40)
(2.6)
48.9 (1.1
(1.9
TP CROCS
(1.8
31.1 (1.
69.7
39.4 (04)
KM CROCS
T3.4 (7.1)
58.6 (2.8)
-7..6 (3.9)
25.9 (1.3)
CP CROCS
(124
38.0 (0.8)
73.5 (0.)
19.5 (0.2)
TP CROCS
(144
05.0 (0.)
53.6 (0:7)
29.1 (0.7)
CROCS
90.3 (0:3)
72.8 (1.5)
76.0 (0.3) 35.9 (0.4)
6.3 Deploying clinical prototypes in the retrieval setting
Up until now, we have shown that CROCS leads to accurate clustering. In this section, we show that CROCS can also be independently exploited for retrieval. Specifically, query retrieves the closest K == [1,5, 5, 10] previously unseen cardiac signals, and assigns them to its associated set of patient we evaluate these assignments based on both partial and exact matches of the attributes (# attribute matches) represented by the query and retrieved cardiac signals.
attributes. In Table 2
In T we find that CROCS outperforms the baseline retrieval method, DTC. For example, on Chapman, at K and when # attribute matches 2 1, CP CROCS and DTC achieve a precision of 95.6 and 71.9%, respectively. This indicates that, on average, 95.6% of the cardiac signals retrieved by the clinical prototypes are relevant. Relevance, in this case, implies that the retrieved cardiac signals share at least one attribute with the query. Such finding points to the utility of clinical prototypes as queries in the retrieval setting. We also find that CROCS leads to rich representation learning that facilitates retrieval. This is evident by the strong performance of TP CROCS which depends directly on representations learned via our CROCS framework. For example, on PTB-XI, at
K = 1, and when # attribute matches 2 1, DTC, TP CROCS, and CP CROCS achieve precision of 70.0, 99.0, and 92.5%. respectively. In this particular case, the lower performance of CP CROCS relative to TP CROCS is hypothesized to stem from clinical prototypes acting instead as archetypes (extreme representative data points) 422 which may occasionally hinder retrieval along multiple attributes. Evidence of such extreme embeddings can be found in Fig. 4 We also show that CROCS continues to perform well even when provided with only 10% of the labelled training data (see Appendix G
Table 2: Precision of K retrieved representations, v. in the validation set of Chapman and PTB- XL, that are closest to the query. Results are shown for partial and exact matches of the attributes (# attribute matches) represented by the query and retrieved cardiac signals, and are averaged across five random seeds. Brackets indicate standard deviation and bold reflects the top-performing method. The strong performance of CP CROCS provides evidence in support of our CROCS framework.
# atrribute
Chapman 5
PTB-XI
matches
Query
K
10
5
10
DTC 3 TP CROCS
71.9 (a.
100.0 cam
100.0 c.m
70.0 (uo)
90.0 (8.)
100.0 (6.0)
(1.9
97.5 (2.)
100.0 (0m)
99.0 (.0)
100.0 (0.))
100.0 (0:0)
CP CROCS
95.6 (6.)
100.0 co.
100.0 co.
92.5 (00)
100.0 (0:0)
100.0 (6.0)
DTC [8
25.0 (0.0)
71.9 (9.))
90.0 (7.0)
22.5 (00))
52.5 (16.0)
80.5 (4.0)
TP CROCS
35.0 (38)
79.4 (7.6)
90.0 (0.1)
71.5 (2.9)
91.55 (1.0)
100.0 (0.0)
CP CROCS
61.3 (10.0
86.3 (10.9)
93.8 (7.9)
63.D (1.9)
96.5 (2.00
99.5 (1.0)
DTC [8
3.1 (0.0)
15.0 (3.1)
23.8 (0.1)
2.5 (0.0)
0.5
16.5 (0:))
TP CROCS
(..6
23.8
36.9 (2.00
15.5
32.0
43.5 (0.))
CP CROCS
11.3 (2.)
33.1 (6.1)
16.3 (6.7)
12.5 (000)
33.5 (2.00
13.0 (4.0)
To qualitatively evaluate the retrieval performance, we first randomly choose query representing set of attributes and calculate its Euclidean distance to the representations in a validation set. We present
8

---

distributions of such distance values in Fig. (top row), for DTC query and a CP CROCS query, coloured based on the ground-truth class of the representations (other queries shown in Appendix HH. In Fig- 5 (bottom row), we illustrate the six cardiac signals (K 6) that are closest to each query, with a green border indicating signals whose class attribute matches that of the query. We find that the CP CROCS query is closer to representations of the same class (SR) than to those of different class. For example, in Fig. [ (top row), the average distance between the CP CROCS query representing (SR, male, under 49} and representations with and without the class attribute SR 0.6 and > 1.5, respectively. Such separability, which not exhibited by the DTC query, points the improved reliability of the CP CROCS query in distinguishing between the relevance of cardiac signals. Further evidence in support of this reliability is shown in Fig. 5 (bottom row) where find that DTC and CP CROCS query retrieve relevant cardiac signals 0% and 50% of the time, respectively. This finding also extends to the PTB-XI. dataset (Appendix H
class
0.4
class
SA
5R
Density
1.00
SB
GSVT
Density
0.3
GSVT
56
0.75
AF1B
AF1
0.2
0.1
0.25
0.0
1.4
1.5
1.8
2.0
2.2
0.5
1.0
1.5
2.0
2.5
3.0
3.5
Euclidean distance
Buclideon distance
M0 1 M
(a) DTC query (SR, male, under 49)
(b) CP CROCS query (SR, male, under 49)
Figure 5: Qualitative retrieval performance of two distinct query prototypes. (top row) Eu- clidean distance from (a) DTC query or (b) CP CROCS query to representations, v, in the validation set of Chapman. (bottom row) Six closest cardiac signals to the query which is associated with set of patient attributes { (discaso, sex, age}. Retrieved cardiac signals with a green border indicate those whose class attribute matches that of the query. We show that the CP CROCS query is closer representations of the same class (SR) and thus retrieves relevant cardiac signals.
6.4 Investigating the marginal impact of design choices
We have shown that CROCS reliably allows for both clustering and retrieval. In this section, conduct several ablation studies to better understand the root cause of this reliability (see Table We find that, average, the soft assignment of representations to prototypes is preferable to the hard
Table 3: Marginal impact of design choices of CROCS on clustering performance. Evaluation is based on (a) class and (b) sex and age attributes. Results are averaged across five random seeds. Brackets indicate standard deviation and bold reflects the top- pop-prforrinng method. We show that our full framework (LNCE saft + freg) is preferable to other variants regardless of attribute.
(a) Cardiac arrhythmia class attribute
(b) Sex and age attributes
Method
Chapman
PTB-XL
Method
Chapman
PTB-XL
Acc
AMI
Acc
AMI
SCx
ago
50%
age
DNCS-hard
86.8 (0.)
67.5
66.5
35.0
ENcS-Mard
(0.9
26.2
76.3 (07))
19.8
EncE-soft
LNCR-aofe
Tu
Tu $ oo 1Arey
89.8 (1.7)
72.1 12.8)
76.1 (0.2)
36.0 (0.4)
Tu # 2o
56.8 (1.8)
37.4 (1.00
74.3 (0.0)
19.2 (0.9)
90.3 (033
72.8
76.0 (0.3)
35.9 (1.4)
Cress
(7.4
38.0 (0:)
73.5 (0.))
19.5 (0.2)

---

assignment. For example, on PTB-XL, LNCE-soft and LNCE-hard achieve Acc(class) = 76.0 and 66.5%, respectively. We also find that our full framework (LNCE-soft + Ereg) performs better than, or par with, other variants. For example, on Chapman, LNCE- hard and LNCE-soft Tw 0o, and Tu + oo achieve (MM(ssss = 67.5, 68.2, and 72.1 1%, respectively, whereas LNCE-soff + Greg achieves AMI(class) = 72.8%. This is positive outcome given that the regularization term's main purpose was simply to improve the interpretability of prototypes by allowing them to capture the semantic relationships between attributes. These findings extend to the retrieval setting (Appendix IT.
7 Discussion
In this paper, we proposed a supervised contrastive learning framework, entitled CROCS, for the clustering and retrieval of cardiac signals based on multiple patient attributes. In the process, we attracted representations associated with a set of attributes to learnable embeddings, termed clinical prototypes, that share such attributes and repelled them from prototypes with different attributes. We showed that CROCS outperforms the state-of- of-the-art method, DTC, when clustering, and retrieves relevant cardiac signals while lending itself to a higher degree of interpretability.
We acknowledge several limitations that can be addressed in future work. We made the design choice of discretizing patient attributes (e.g., and age). However, these attributes, and indeed other clinical parameters, can be continuous. Therefore, extending clinical prototypes to capture this continuity could offer researchers finer control over the clustering and retrieval process. In doing so, we envision two main challenges. First, continuous attribute values imply that the number of prototypes be learned will grow. This may pose computational bottleneck. To alleviate this bottleneck, researchers may benefit from advancements in the field of NLP and textual representation learning. Second, to learn useful prototypes in this continuous setting, one may require sufficient labelled data associated with each attribute combination. In practice, this may be difficult to achieve, particularly for under-represented attribute groups. Furthermore, in this work, we were limited to three discrete patient attributes. Such a decision, although primarily motivated by the availability of patient meta-data alongside ECG signals, does not exploit the diverse and abundant meta-data typically available within healthcare.
Acknowledgements
We thank the anonymous reviewers and Antong Chen for their insightful feedback. We also thank Nagat Al-Saghira and Mohammed Abdel Wahab for lending their voice. David Clifton was supported by the EPSRC under Grants EP/P009824/1and EP/N020774/1, and by the National Institute for Health Research (NIHR) Oxford Biomedical Research Centre (BRC). The views expressed those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health. Tingting Zhu was supported by the Engineering for Development Research Fellowship provided by the Royal Academy of Engineering.
References
[1] Chaitanya Shivade, Preethi Raghavan, Eric Fosler-Lussier, Peter J Embi, Noemie Elhadad, Stephen B Johnson, and Albert M Lai. A review of approaches to identifying patient pheno- type cohorts using electronic health records. Journal of the American Medical Informatics Association, 21(2):221-230 2014.
[2] Vivek H Murthy, Harlan M Krumholz, and Cary P Gross. Participation in cancer clinical trials: race-, sex-, and based disparities. JAMA, 291 2004.
[3] Ali Pourmand, Mary Tanski, Steven Davis, Hamid Shokoohi, Raymond Lucas, and Fareen Zaver. Educational technology improves eeg interpretation of acute myocardial infarction among medical students and emergency medicine residents. Western Journal of Emergency Medicine, 16(1):133, 2015.
[4] Li Huang, Andrew L Shea, Huining Qian, Aditya Masurkar, Hao Deng, and Dianbo Liu. Patient clustering improves efficiency of federated machine learning to predict mortality and hospital stay time using distributed electronic medical records. Journal of Biomedical Informatics, 99:103291, 2019.
10

---

[5] Isotta Landi, Benjamin S Glicksberg, Hao-Chih Lee, Sarah Cherng, Giulia Landi, Matteo Danieletto, Joel T Dudley, Cesare Furlanello, and Riccardo Miotto. Deep representation learning of electronic health records to unlock patient stratification at scale. NPJ Digital Medicine, 1(1):1-11, 2020.
[6] R Rani Saritha, Varghese Paul, and P Ganesh Kumar. Content based image retrieval using deep learning process. Cluster Computing, 22(2):418--4200, 2019.
Haolin Wang, Qingpeng Zhang, and Jiahu Yuan. Semantically enhanced medical information
retrieval system: a tensor factorization based approach. IEEE Access, 5:7584-7593, 2017.
[8] Steven R Chamberlin, Steven D Bedrick, Aaron M Cohen, Yanshan Wang, Andrew Wen, Sijia Liu, Hongfang Liu, and William R Hersh. Evaluation of patient-level retrieval from electronic health record data for cohort discovery task. JAMIA Open, 3(3):395-44 2020.
[9] Travis R Goodwin and Sanda M Harabagiu. Multi-modal patient cohort identification from eeg report and signal data. In AMIA Annual Symposium Proceedings, volume 2016, page 1794. American Medical Informatics Association, 2016.
[10] Geoffrey Hinton. How to represent phrt-whole hierarchies in a neural network. arXiv preprint arXiv:2102.12622 2021.
[11] Alan H Gee, Diego Garcia-Olano, Joydeep Ghosh, and David Paydarfar. Explaining deep classification of time-series data with learned prototypes. In CEUR workshop proceedings. volume 2429, page 15. NIH Public Access, 2019.
[12] Dianbo Liu, Dmitriy Dligach, and Timothy Miller. Two-stage federated phenotyping and patient representation learning. In Association for Computational Linguistics, volume 2019, page 283.
NIH Public Access, 2019.
[13] Yue Li, Pratheeksha Nair, Xing Han Lu, Zhi Wen, Yuening Wang, Amir Ardalan Kalantari Dehaghi, Yan Miao, Weiqi Liu, Tamas Ordog, Joanna M Biernacka, et al. Inferring multimodal latent topics from electronic health records. Nature Communications, 11(1):1-17, 2020.
[14] Siddharth Biswal, Cao Xiao, Lucas M Glass, Elizabeth Milkovits, and Jimeng Sun. Doctor2vec: Dynamic doctor representation learning for clinical trial recruitment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 557-564, 2020.
[15] Sajad Darabi, Mohammad Kachuee, Shayan Fazeli, and Majid Sarrafzadch. Taper: Time-aware patient ehr representation. IEEE Journal of Biomedical and Health Informatics, 2020.
[16] Qianli Ma, Jiawei Zheng, Sen Li, and Gary W Cottrell. Learning representations for time series clustering. Advances in Neural Information Processing Systems, 32:3781-3791, 2019.
[17] Naveen Sai Madiraju. Deep temporal clustering: Fully unsupervised learning of time-domain features. PhD thesis, Arizona State University, 2018.
[18) Josif Grabocka, Nicolas Schilling, Martin Wistuba, and Lars Schmidt-Thient Learning time-series shapelets. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 392-401, 2014.
[19] Qin Zhang, Jia Wu. Peng Zhang, Guodong Long, and Chengqi Zhang. Salient subsequence learning for time series clustering. IEEE Transactions on Pattern Analysis and Machine Intelligence, (199):2133-2007 2018.
[20] Dani Kiyasseh, Tingting Zhu, and David A Clifton. CLOCS: Contrastive learning of cardiac signals across space, time, and patients. In International Conference on Machine Learning, pages 5606-5615. PMLR, 2021.
[21] Junnan Li, Pan Zhou, Caiming Xiong, and Steven Hoi. Prototypical contrastive learning of unsupervised representations. In International Conference on Learning Representations, 2021.
[22] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, pages 1597-1607. PMLR, 2020.
[23] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap your own latent a new approach to self-supervised learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 11271-21280 Curran Associates, Inc., 2020.

---

[24] Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis P Langlotz. Contrastive learning of medical visual representations from paired images and text. arXiv preprint arXiv:2010.00747, 2020.
[25] William R Hersh and Robert A Greenes. Information retrieval in medicine: state of the art. MD Computing: Computers in Medical Practice, 7(5):302-31,, 1990.
[26] Harsha Gurulingappa, Luca Toldo, Claudia Schepers, Alexander Bauer, and Gerard Megaro. Semi-supervised information retrieval system for clinical decision support. In Text Retrieval Conference (TREC), 2016.
[27]
Adam M. Rhine. Information Retrieval for Clinical Decision Support. PhD thesis, 2017.
[28]
Byron Wallace, Joël Kuiper, Aakash Sharma, Mingxi Zhu, and lain Marshall. Extracting pico sentences from clinical trial reports using supervised distant supervision. The Journal of Machine Learning Research, 17(1):4572-4566, 2016.
[29] Leonard W D' Avolio, Thien M Nguyen, Wildon R Farwell, Yongming Chen, Felicia Fitzmeyer, Owen M Harris, and Louis D Fiore. Evaluation of a generalizable approach to clinical infor- mation retrieval using the automated retrieval console (arc). Journal of the American Medical Informatics Association, 17(4):375-382 2010.
[30] Deepak Roy Chittajallu, Bo Dong, Paul Tunison, Roddy Collins, Katerina Wells, James Flesh- man, Ganesh Samkaranarayanan, Steven Schwaitzberg. Lora Cavuoto, and Andinet Enquobahrie. Xai-cbir: Explainable ai system for content based retrieval of video frames from minimally invasive surgery videos. In International Symposium on Biomedical Imaging, pages 66-69. IEEE, 2019.
[31] Travis R Goodwin and Sanda M Harabagiu. Learning relevance models for patient cohort retrieval. JAMIA Open, 1(2):265-275, 2018.
[32] Yanshan Wang, Andrew Wen, Sijia Liu, William Hersh, Steven Bedrick, and Hongfang Liu. Test collections for electronic health record-based clinical information retrieval. JAMIA Open, 2(3):360-368, 2019.
[33] Leland MeInnes, John Healy, Nathaniel Saul, and Lukas Großberger. Umap: Uniform manifold approximation and projection. Journal of Open Source Software, 3(29):861, 2018
[34] Alan F Smeaton. Using NLP or NLP resources for information retrieval tasks. In Natural Language Information Retrieval, pages 99-111. Springer, 1999.
[35] Jianwei Zheng, Jianming Zhang, Sidy Danioko, Hai Yao, Hangyuan Guo, and Cyril Rakovski. A 12-lead electrocardiogram database for arrhythmia research covering more than 10,000 patients. Scientific Data, 7(1):188, 2020.
[36] Patrick Wagner, Nils Strodthoff, Ralf-Dieter Bousseljot, Wojciech Samek, and Tobias Schaeffter. PTB-XL, a large publicly available electrocardiography dataset, 2020.
[37] Nils Strodthoff, Patrick Wagner, Tobias Schaeffter, and Wojeiech Samek. Deep learning for ecg analysis: Benehmarks and insights from ptb-xl. IEEE Journal of Biomedical and Health 2020.
Informatics, 25(5):
[38] Kai Han, Andrea Vedaldi, and Andrew Zisserman. Learning to discover novel visual categories via deep transfer clustering. In Proceedings of the IEEE International Conference on Computer Vision, pages -401, 2019.
[39] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In Proceedings of the European Conference on Computer Vision (ECCV), pages 132-149, 2018.
[40] Xu Ji, João F Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. In Proceedings of the IEEE International Conference on Computer Vision, pages 9865-9874, 2019.
[41] Yuki Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultaneous clus- tering and representation learning. In International Conference on Learning Representutions, 2020.
[42] Morten Mørup and Lars Kai Hansen. Archetypal analysis for machine learning and data mining. Neurocomputing, 80:54-63, 2012.
12

---

[43] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-peeformance deep learning library. In Advances in Neural Information Processing Systems, pages 8024-8035, 2019.
13


Reviews:
[{"review_id": "xF7ZsL110el", "reviewer": "Reviewer_mzV8", "summary": "Cardiac signals when unlabeled are difficult to query for given a set of attributes of interest such as disease class, age and sex.  The paper proposes a cluster-based retrieval system for this purpose. The algorithm proposed is of two-fold - (1) Learn a function to map a cardiac signal to an embedding closer in representation to a clinical prototype (embedding representative of a unique attribute set)   - this is done using a contrastive-learning framework assuming an initial-set of labeled data where cardiac-signals are annotated with class, age and sex and (2) Given a query attribute set - use its clinical prototype embedding to rank the  cardiac signals. Here the function from step 1 is used to get an embedding representation for cardiac signals and then their Euclidean distance from clinical prototypes is considered for ranking in the retrieval. The paper evaluates clustering and retrieval tasks on two datasets Chapman and PTB-XL.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Briefly my overall review was primarily based on the following - the ideas presented in the paper are interesting to me, however I found that the setting considered is unrealistic and the evaluation provided is insufficient.  Please find my detailed review below - \n\nThere are several things to like about the paper,\n\nOriginality\n\n(1) I liked the idea of extending multiple attribute information into unique clusters with their own clinical prototypes.\n\n(2) I liked the regularization term to introduce semantic meaning - i.e prototypes representing similar attribute sets (e.g., adjacent age groups) should be closer to one another. The way this is modeled in equation 3 (using hamming distance of original attribute sets for supervision) is very interesting to me.\n\n(3) Overall I found the paper easy to read (except the description on retrieval setting which I think can be illustrated with an algorithm and I think the intuition of the metric (in equation 5) should be discussed)\n\nThere are several shortcomings as well -\n\nOriginality\n\n (1) I think the setting considered is slightly unrealistic. Typically the number of unlabeled data is orders of magnitude higher than the labeled data in such clinical retrieval tasks. Also the unlabeled data can be subjected to distribution shifts as compared to the labeled data distribution. Owing to both these difficulties existing literature has not explored the direction of using supervised clustering  (as this paper did) - but rather tried weak-supervision or unsupervised techniques of clustering.  This changes the setting in several ways - (a) In the paper the retrieval evaluation is done on the validation dataset (20%) which is smaller than the training data (60%). This will change the retrieval metrics - meaning the metrics reported in the paper can be overestimating the actual performance (b) Supervised learning may not be an ideal framework any more and semi-supervised learning would need to be considered to address the above challenges. \n\n(2) How will the algorithm perform as we decrease the labeled data and increase the unlabeled data? Can you provide retrieval and clustering metrics for the same and also compare this approach with semi-supervised/self-supervised clustering algorithms? (currently retrieval metrics are only compared against DTC a supervised learning method.)\n\n(3) Although the application considering multiple patient attributes for retrieval is new - this comes with several limitations that needs to be discussed in the paper (a) the complexity of algorithm or the clusters considered will increase exponentially with increase in the attributes and the number of values taken by each attribute. This will also limit the amount of discretization that can be done in attributes such as age.  (b)  This will also mean assuming sufficient labeled data covering all these attribute information.\n\nQuality/Clarity - \n\nThe retrieval metrics (equation 5) are not reflective of precision at K and are much relaxed versions - the metric is not average P@K across all M clusters (or queries) but rather average of the times P@K is non-zero across the M clusters. Can you provide a discussion on this metric?\n\nI have several questions - can you please help by answering them?\n\n(1) In figure 3 - Is this representative of the real data experiments or is this a cartoon to visualize the algorithm? If it is indeed a zoomed-in figure for experimental data - please provide more details. If it is not can you validate that the model has learnt semantically meaningful representations?\n\n(2) In Figure 4 right image - points from following classes - HYP and NORM seem to be indistinguishable. So is a part of STTC and MI. Considering this the accuracy reported in Table 1 (76%) seems a little high especially considering HYP and NORM are majority classes.\n\n(3) Does DTC in Table 1 really have a standard deviation of 15.0 in Accuracy and 20 in AMI? How is this possible - please provide an explanation?\n\nSignificance -\n\nIt is difficult to evaluate the significance of the paper considering insufficient evaluation and unrealistic setting as compared with existing work.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper discusses some of the limitations of the algorithm. Please find more detailed comments above to extend on this.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8-9 hrs"}, "extra_text": {}}, {"review_id": "n8HXTAiO-e", "reviewer": "Reviewer_2o6W", "summary": "This paper proposes a clustering and information retrieval algorithm for unannotated large databases of physiological signals (in this case ECG signals). The method is based on a contrastive learning approach, which has recently been suggested for semi-supervised learning on ECG data.\nThe proposed approaches are evaluated on two large ECG datasets and compared with state-of-the-art approaches.\n", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is really innovative with two new applications (clustering and information retrieval) based on a recently suggested representation learning approaches.\nThe research is quite significant, given the availability of large databases of EHR data, containing clinical physiological signals (for instance the MIMIC III database). Applying such approach on such a dataset, with many meta-data available, could help uncover specific clinical protoytpes of physiological signlas and be of the utmost importance.\nThe paper is clearly written and of good quality. The proposed approach is tested in several settings, using two available large sets of ECG signals, and the technique performance is compared to other state-of-the-art approaches.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The main concerns are the following:\n1.\tThe explanation of using soft assignment instead of har-assignment is discussed in the description of the methodology, but it is unclear to me whether the authors have reported an experiment allowing to prove their conjecture.\n2.\tThe authors should discuss the fact that CP CROCS seem to be underperforming the other clustering approaches based on the age attribute. This result is quite strange and an outlier, but it would be interesting to have a discussion on a possible explanation of this phenomenon.\n3.\tCould the authors discuss the main difference of the tow datasets, and especially as depicted in figure 4 why the separability of the classes is so much lower on PTB-XL compared to Chapman\n4.\tIN terms of quality of the information retrieval, cold the authors comment why evaluating soft assignment (and performance if only one attribute is correct) is clinical significant? It seems to me quite important that pathology and should be well retrieved, extracting an example of AF when asking for Normal rhythm, would have higher impact than a mistake on age or sex\n5.\tCan the authors discuss whether age and sex are good attributes for ECG signals? Heart Rate variability is affected by age, but I am not aware of morphological changes in the ECG signals due to age (or even sex)? Figure 3 seem to be indicating that the representation of the ECG signal is not that influenced by either age or sec, although there seem to be a trend or evolution with age, and separation of sex on the proposed full framework.\n6.\tThe order in sections 2 related work and 3 background should be consistent clustering then IR.\n7.\tThe acronym HER is not introduced properly, that is at the first appearance of the term\n", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "k8iy9k_Obm", "reviewer": "Reviewer_vkP4", "summary": "This work presents a framework for information retrieval and clustering of patients based on their ECG’s. To do this, the authors framed the problem as a supervised contrastive learning problem – this is a very hot topic in machine learning for healthcare and I believe the framework and experimental design they propose is technically appropriate. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, my opinion was that the work was technically sound, and my primary concerns are about novelty and the chosen benchmarks for comparison. If these two points can be clarified I believe this can be a suitable paper for this venue.\n\nI had trouble differentiating the authors technical contribution and from prior work. I believe the authors have substantially built upon past work and have novel contributions but they should make it more clear what their key technical contributions are. I believe this is likely the key major point to evaluate this work for a venue like NeurIPS.\n\nIn line with this comment – the authors may want to break out a section from the clinical representation learning to include more of the recent contrastive learning papers. \n\nFigure 3 is confusing - what data were used? is this a result or part of the methods? (simulated?). The legend also isn't explained (e.g. sttc). \n\nI believe there are better metrics to show in table 1 (e.g. traditional classification metrics + confidence intervals). \n\nThe performance of examples such as SeLA, DC and IIC is so bad it’s likely they were not appropriately adapted to the problem – can this be explained? Are there better apples to apples comparisons? This is especially true given that with the supervised component, we know there are FDA approved devices with arrhythmia classification (AliveCor). I'm not convinced these are the best examples to compare to.\n\nWhy were methods removed transitioning from table 1 to table 2.  \n\nMinor Comments:\nI found it strange that the information retrieval task as introduced first throughout, but then the clustering results were shown first. \n\nI think you need to be careful about using the term clinical prototype, as it has a collision with the medical devices space. Given this work is using ECG data this will be confusing to the clinical audience. I admit, this may be a subjective preference as I’ve never liked the term cluster prototype, because it changes the traditional definition of the word prototype (a first instance other are copied from). \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors acknowledge several key limitations, mostly the result of the data that were available to them. I think the authors should discuss several other key limitations: 1.) in the evaluation of the methods, particularly a.) the re-implementation of prior methods, and b.) clinical evaluation of the information retrieval task. It should be straightforward to test whether a clinician believes the information retrieval matches their training (by showing a matched ECG and a further ECG and testing whether they identify the matched ECG). If they do not, these examples can be further explored – either the clinician is wrong, or the model could be improved. 2.) the lack of external validation or transfer to alternate datasets (e.g. training on Chapman and application to PTB-XL - even if only for Age, Sex)\n\nThis work has a potential to contribute to important societal impacts, particularly around fair machine learning for health. It may allow researchers to study underrepresented groups that are not clearly defined by binary categorizations of race, sex etc. but will need to be further studied to demonstrate this.\n", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "9g2mInckqCR", "reviewer": "Reviewer_XjBr", "summary": "Paper presents for retrieving cardiac signal data corresponding to a patient described  by a set (3) of attributes (Sex, age, disease type). This is built on a model constructed by clustering such cardiac signals representations and learning their corresponding patient attributes.\n\tAlthough well written, the paper suffers, in our opinion, from targeting a very simplified problem (number of attributes, attribute sampling) that lends it to building a bespoke solution that overfits the used data and thus overperforms other methods.  The novelty was not clearly apparent to us from other clustering/retrieval methods.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- \"by the rapid growth of large-scale clinical databases and the increased prevalence of unlabeled instances; those for which patient attribute information is unavailable.\"\n\t- This covers a multitude of possible cases (some diagnostics data missing, variability in acquired diagnostics data from different origins (equipment, institutions, countries etc.) from different patients, missing patient data). A few examples of the ones you specifically target would be welcome at this stage.\n- \"Such methods, however, are exclusively unsupervised; they do not exploit patient attribute information.\"\n\t- Clustering approaches are unsupervised by design and meant to be. What do they cluster if they do not exploit patient attribute information? This part needs clarification. At this stage should define what data you are using and how you are segmenting it (patient attributes (disease, age, sex etc., other data (diagnostics?)).\n- Figure 2: \n\t- Need to explain what the attributes are, what sinus rhythm/Atrial fib./ Bradycardia are (disease, disease class etc. for non domain expert. (later one learns of 5/6 disease classes used). \n\t- Is that a \"dual\" repulsion line towards the triangles of atrial fib.? \n\t- For the Similarity probability mass, can you provide the domain knowledge supporting it? Why/Is gender/sex that more indicative of cardiac signal discrepancies? How do the triangle/+ symbols used relate to the age, disease class attributes?\n \nNotes:\n- Please introduce your acronyms at first use to avoid confusions. \n    - Example: Electronic Health Record (EHR) data. \n- \"... a centroid groups together instances that share some similarities.\": \n    - Isn't the centroid rather a single point used to represent a group of instances that share some similarities?\n\nAfter rebuttal:\n-----------------\nThanks to the authors for their responses.\nAfter reading the rebuttals, many unclear points were explained. \nI am still not convinced by the adequacy of the chosen experiment attributes and the generalization power of the approach. \nTherefore, I am raising my evaluation to \"Marginally below the acceptance threshold\". \nRegards.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No specific ones come to mind.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {"ethical_concerns": "Even if nominative information is not used, could have explained any ethical rules governing establishments or country where data is sourced from regarding use of medical or personal information records and the legislated limitations."}}]

Author rebuttal:

Meta-review: This presents a well written paper which presents a framework for information retrieval and clustering of patients based on unannotated ECG/ physiological data . In discussion, there were mixed reviews with regards to novelty, experimental evaluation and baseline comparisons to related work. However, after evaluating the paper and the discussion and reviewers' comments,  I agree that the proposed approach is presents a supervised technique for learning a compact representation which respect the attributes (or weak labelling) available. Such a representation could be used for other task for which access to a large database of (strong) annotated data is quite difficult. This is a non-trivial task in the healthcare domain and highlights an important application of ML for real-world impact, which is potentially generalisable to other data settings. Although there were concerns around over-fitting this model to a very stringent problem, the novelty of the approach and problem formulation from an ML perspective far outweighs any potential downsides. As both the authors and one of the reviewers highlights, this work may have a potentially positive impact on multitude of stakeholders within healthcare and could inform researchers working on fairness and thus better serve patients from underrepresented backgrounds.
Decision: Accept (Poster)