{COMPANYNAME}11K: An Unsupervised Representation Learning Dataset for Arrhythmia Subtype Discovery
Authors: Shawn Tan
Guillaume Androz
Ahmad Chamseddine
Pierre Fecteau
Aaron Courville
Yoshua Bengio
Joseph Paul Cohen
Venue:  (2020)
Keywords: representation learning
healthcare
medical
clinical
dataset
ecg
cardiology
heart
discovery
anomaly detection
out of distribution

Abstract: We release the largest public ECG dataset of continuous raw signals for representation learning containing over 11k patients and 2 billion labelled beats. Our goal is to enable semi-supervised ECG models to be made as well as to discover unknown subtypes of arrhythmia and anomalous ECG signal events. To this end, we propose an unsupervised representation learning task, evaluated in a semi-supervised fashion.  We provide a set of baselines for different feature extractors that can be built upon.  Additionally, we perform qualitative evaluations on results from PCA embeddings, where we identify some clustering of known subtypes indicating the potential for representation learning in arrhythmia sub-type discovery.

Under review as a conference paper at ICLR 2020
[COMPANYNAME] I1K: AN UNSUPERVISED REP- RESENTATION LEARNING DATASET FOR ARRHYTHMIA SUBTYPE DISCOVERY
Anonymous authors Paper under double-blind review
ABSTRACT
We release the largest public electrocardiogram (ECG) dataset of continuous raw signals for representation learning containing 11 thousand patients and billion labelled beats. Our goal to enable semi-supervised ECG models to be made as well to discover unknown subtypes of arrhythmia and anomalous ECG sig- nal events. To this end, we propose an unsupervised representation learning task, evaluated semi-supervised fashion. We provide set of baselines for differ- ent feature extractors that can be built upon. Additionally, we perform qualitative evaluations on results from PCA embeddings, where we identify some clustering of known subtypes indicating the potential for representation learning in arrhyth- mia sub-type discovery.
INTRODUCTION
Arrhythmia detection presently performed by cardiologists or technologists familiar with ECG readings. Recently, supervised machine learning has been successfully applied to perform detection of certain types of arrhythmia (Hannun al., 2019: Yildırım et al. 2018 Mincholé & Rodriguez,
2019 Porumb et al. 2020).
However, there may be ECG anomalies that warrant further investigation because they do not fit the morphology of presently known arrhythmia. We seek to use a data driven approach to finding these differences that cardiologists have anecdotally observed, which motivates the representation learning potential of this data.
Our data is collected by the [DEVICENAME) (COMPANYNAME) (MMANAueM)PPapueeettet tion and sampled at 250Hz with the (DEVICENAME) a modified lead 1 position. The wealth improve on the techniques currently used by the medical
single-lead heart monitor device from (Paquet et al. 2019). The signals were recorded with 16-bit resolu-
of data this provides us allow industry to process days worth of ECG data, and perhaps to catch anomalous events earlier than currently possible. All data is made publicu
The ethics institutional review boards at the (UNIVERSITY) approved the study and release of data #(STUDYID)
1.1 OBJECTIVE
We want improve the state-of-the-ard of automated arrhythmia detection via representation learn- ing. Ideally, this representation should preserve as much information about the underlying true heart function possible. Such representations and learned feature extractors can improve downstream tasks which require more complicated features than what is typically extracted to predict major cardiac issues. More concretely, we are proposing semi-supervised challenge on ECG data.
While an objective method to evaluate such a representation would be to measure its performance on tasks of interest, the way to perform best on such an evaluation would be directly run su- pervised learning task on those objectives. However, in certain circumstances, like training a neural
Data available: URI

---

Under review as a conference paper at ICLR 2020
Sami ceni-speeriied
Subtype discovery
Morpho analysis
Thansfor leaming
Device worm by over 11k patients
Datesel conisining over billion labolled beats (Released free the public)
Unsupervised
representation leaming
Figure 1 Overview of the project.
network, for example, doing so results loss of information about the input (Tishby Zaslavsky, 2015). The process may remove information vital to the discovery of new sub-types. We will see an example this in $5.2
Extracting features which can predict outcomes of not just arrhythmia is also an existing field of study (Lerma Glass, 2016; Karpagachelvi et al. 2010), and can benefit from learned feature extractors based on this data.
RELATED WORK
ECG (or sometimes known EKG) signals collected by electrocardiograph machines. These machines traditionally have 10 electrodes, resulting in 12-lead ECG data. These can be thought of 12 channel signal that provides additional data about the heartbear, but allows for only short periods of data capture due to the cumbersome nature of these machines, and are not sufficient for capturing rarer events that happen over time.
One of the first open dataset of ECG signals was the MIT-BIH dataset, created in 1979 (Moody & Mark 200T). They "exxected that the availability of a common database would foster rapid and quantiñable improvements in the technology of automated arrhythmia analysis." The MIT-BIH is still use today with just subjects. However, |hhah & Rubin (2007); (ugliin & Thatai (2006) found that computer predictions during that time were fraught with errors.
Later, data collection efforts improved leading to the creation of many small specific datasets (Goldberger et al. 2000). The MIMIC- III Waveform Database (Johnson et al., 2016) contains 67,830 waveform records from 30,000 ICU patients. These samples a higher sampling rate and with more leads. However, they are only recorded for short periods of time. The ECG-ViEW II dataset (Kim et al. 2017) aims to be a freely available dataset of ECG records together with clinical data for 461,178 patients. Instead of raw sig- nals, only beat information is included: RR interval, PR interval, QRS duration, etc Figure 2 shows the basic ECG form where a letter identifies each aspect of the beat: P. Q. R, S, and T. The STAFF III Database (Pablo Martinez et al., 2017) contains 104 pa- tients under an acutely induced myocardial ischemia. This includes pre, during, and post catheter insertion.
at
Figure QRS regions for an ECG heartbeat signal.
More recently, single-lead wearable devices provided much larger amounts of data than before. As these devices could worn throughout the day, over a period couple of weeks, machine learning had much more data to work with. RRajurkar al. (2017) created annotated training dataset ECG signals consisting of 30,000 patients (Turakhia et al. 2013). The authors' approach, and the follow up work claim that their automated models perform at the level of trained cardiologists (Hannun et al. 2019). However, their data has not been made publicly available.
TThe letters indicate the interval between the two events. For example, PR interval is the time in hetween the and R event, while RR interval is the length of time between Rs in different heartbeat.
2

---

Under review as a conference paper at ICLR 2020
505A
4130
0016
Kde
(a) Duration wear
(b) Age
(c) Patient sex
Figure 3: Demographics of the patients in the data.
3 PRIVACY CONCERNS
Our data has been made completely anonymous so that no reidentification is possible. For the sake of precision, no name or serial number any kind associated with a particular data set.
3.1 HEARTBEATS AS BIOMETRICS
There are attempts to use ECG signal data as biometrics to identify someone. This brings risk of re-identification from our anonymized signal data.
A paper by Salloum & Kuo (0117) claims high performance but the evaluation does not seem very controlled. A company called Nymi also aims to use wearable ECG as method authenticate users.
When viewed in the context of other literature, the claim that ECG is a reliable method of authenti- cation seems to be diminished. For example, Song et al. (2017) explores alternative ways to sense cardiac motion (movement the heart, which they say is the identifying aspect), stating that ECG "biosignals are not related to cardiac motion, in which case indirect or incomplete cardiac charac- terization will compromise the advantages of cardiac motion as biometric".
Israel & Irvine (2012) state "Unlike fingerprint and face, the heartbeat data could contain health- related information as well as the personal identification information. This suggests a need for greater care in the collection, storage, and transmission of such data." Additionally, they say that ECG has several limitations that must be overcome before they can be used as a biometric. Specifi- cally, that (I) it requires sufficient number of samples to identify individual because the signal does not contain much information, (2) the combination of varying environments and individual yields unique signal, (3) A target's emotional state also requires intra-individual normalisation, and (4) a change in the contact location can reduce the ability to identify someone.
[COMPANYNAME) 1K DATASET
The dataset is processed from data provided by 11,000 patients who used the [DEVICENAME)TM.R devicee predominantly in Ontario, Canada, from various medical centers. While the device captures ECG data for up to two weeks, the majority of the prescribed duration of wear was week. Figure Bal shows the distribution over duration of wear in the unprocessed data.
It should be noted that since the people who wear the device are patients, the dataset does not represent a true random sample the global population. For one, the average age the patient is 62.2+17.4 years of age. Furthermore, whereas the (DEVICENAME) TM be worn by any patient, is mostly used for third line exanf] so the majority of records the dataset exhibit arrhythmias. No particular effort has been done patient selection except data collection has been conducted over years 2017 and 2018. Figure shows the distribution over age and gender.
The data is analysed by [COMPANYNAMEL on proprietary analysis tools. When the data first extracted from the device, beat detection is performed automatically. A first technologist looks at the record as soon as possible to quickly send
of 20 technologists who performed annotation
Most patients were prescribed (DEVICENAME) TMyy tertiary referral hospital

---

Under review as a conference paper at ICLR 2020
Patient
20000000 4D00000O
G0000000
50000000
100000000
120000000
340000000
1048576
Frame
2048
Figure 4: ECG data at different levels of the hierarchy. From top to bottom, full patient record, segment, and a frame.
feedback on the severity of the case. A second technologist then analyses the record labelling beats and rhythms (these will be further elaborated in Section 5.I] performing a full disclosure analysis i.e. he / she sees the whole recording. The types of labels are described in more detail in Section Finally, the analysis is approved by a senior technologist.
5.1
To prepare the data, segment each patient record into segments of 20 + 1 signal samples = 70 minutes). This longer time context was informed by discussions with technologists: the context useful for rhythm detection. We made it power of two with middle sample to allow for easier convolution stack parameterisation. From this, we randomly select 50 of the segments and their respective labels from the list of segments. The goal here is reduce the size of the dataset while maintaining fair representation of each patient. In the training data we remove the labels for 80% of the patients. For the remaining 20%, half will be kept for the semi-supervised task, while another half will remain as test data for evaluation. Further details of nomenclature and statistics of the unprocessed and processed data can be found in Table
We describe in further detail the different levels of hierarchy we have separated the data into:
Patient level (3-14 days) At this level, the data can capture features which vary in a systematic way and not isolated events, like the placement of the probes or patient specific noise.
Segment level (approximately hour) A cardiologist can look at specific segment and identify patterns which indicate disease while ignoring noise from the signal such as unique signal ampli- tude. Looking trends in the segment help to correctly identify arrhythmia as half hour provides the necessary context to observe the stress of specific activity.
Frame level (approximately 8 seconds) At this level, the data can capture features about the beat as well as the rhythm.
While we have provided baselines only for frame-level features this paper, we believe that pro- cessing the data with these levels of hierarchy results in some grouping information that could be leveraged to attain better results.
UNSUPERVISED REPRESENTATION LEARNING TASK
While the processed data includes labelled beat and arrhythmia information, we propose an imsu- pervised representation learning challenge to the community.
The goal of this data is to develop unsupervised representations of the ECG signal which can aid in two aspects:
1. Improve the performance of supervised tasks by using the learned representations.
2. Identify unknown subtypes of disease by studying the clustering of the representations.
4

---

Under review as a conference paper at ICLR 2020
Training data
Learn feature extractor
ECG signals
Festure extractor
Evaluation data
signals
Sample subeets
ECG
Beat annotation and labels
Labels
Features
Evaluation results
Classification method
Figure 5:
Diagram detailing the training and evaluation pipeline for the representation learning task. We have provide different methods in this paper for the blocks colored in green.
These issues are addressed in quantitative and qualitative evaluations in the next two sections. The focus of this section studies the frame level embeddings which are typically enough for cardiologists interpret.
5.1 QUANTITATIVE EVALUATION
For the quantitative evaluation we will benchmark common unsupervised algorithms in a semi- supervised setting establish base quality. We make all code and models public order facilitate reproducibility and future work[]
The evaluation consists of predicting the beat and rhythm for each frame hold id's > 10, 000). The beat task is to predict if frame contains all normal beats or contains at least one premature ventricular contractions (PVC) or premature atrial contraction (PAC) anywhere in frame. Classifying a beat alone regardless of its surrounding beats can be challenging example, a PAC is an abnormal beat only because appears too soon and disrupts the rhythm (frequency). Furthermore, PAC beat has the same shape as normal beat, taken alone, you can nearly not make the difference with a normal beat. The model will need to construct features about the nearby beats well.
(samples
for
The second task is to predict the rhythm type given a frame. For a given frame the classification method must predict if the rhythm is normal, atrial fibrillation (AFib atrial flutter, based on the input representation. AFib indicated by irregular RR intervals, no distinct waves and usually variable intervals between two atrial activations (Vollmer et al. (018). Flutter appears saw-tooth pattern of R waves. Recognising both patterns require contexts larger than a single beat. These labels
https://github.. htaerluu..oom/hhaeee com/shawwntan/ccnttie
AFib a controversal rhythm as cardiologists not agree on the minimum duration. 8 second frames might not be sufficient to make such a decision.
Term
Definition
Statistic
# (units)
Segment
Fixed length contiguous region of signal.
Number of Patients
11,000
Number of laheled heats
2.774,054,987
Sample
As used in signal process- ing: A scalar value rep- resenting the amplitude of the signal in time.
Sample Rate
250Hz
Frame size
2 +1
2, 049 samples
Segment size
2 + 1
1, 048, 577 samples
Total number of frames
1,084.314
Event
A specific arrhythmia oc- curring.
Total number of segments
542.157
Datasct Size
271.27GB
(a) Glossary of terms
(b) Dataset Statistics
Table 1: Reference tables
5

---

Under review as a conference paper at ICLR 2020
Beat labels
Count
Rhythm Labels
Count
Normal
174,249
NSR (Normal Sinusal Rhythm)
261,377
Premature Atrial Contractions
58,780
AFib (Atrial Fibrillation)
13,056
Premature Ventricular contractions
44,835
AFlutter (Atrial Flutter)
3,330
(a) Beat labels in the test set
(h) Rhythm labels in the test set
Table Label counts in the test subset (patients 1000-10999). Each frame has label. Only 2 types of labels are provided. Only these meaningful labels are used for evaluation and presented to the classifier.
annotated the beat level. If beat is beat-level anomaly, this will be labelled at the beat where the event occured. If a beat within an anomalous rhythm period, the beats within the rhythm would all be labelled with the corresponding rhythm type.
Both these tasks used in a supervised classification problem as a proxy for evaluating the usefulness of extracted features for detecting such events.
Figure 3 shows the pipeline for our evaluation method. Code to perform this evaluation in a con- sistent fashion is made available online for replicating the results and implementing new methods. The bulk of the training data does not come with beat annotation and labels, and can be used to train or feature extraction method. The evaluation consists of sampling N frames from the test set and computing representations using the feature extractor. 50% of the data is then used to train classification method and then evaluated on the held out 50%. Two classification models are used: (I) A k-nearest neighbors (KNN) method with A = 3, and (2)an MLP method, which consists of layers of dimensions 1024, 1024, 512, and 512. The MLP model was trained for 10 epochs with Adam optimizer. We applied dropout (Srivastava al. 2014) prevent overfitting.
Each representation is learned without knowledge the tasks - the feature extraction model not updated during training of the classifier. We provide the evaluation results for the following baseline feature extraction methods:
Principal Components Analysis (PCA) computes the principal components from 30k examples from the training data. Then projects the test data onto 100, 50, or 10 principal components.
Fast Fourier Transform (FFT) computes a Fourier transform representing the magnitude of fre- quencies between 1Hz and 125Hz (Cooley & TTukey 1965).
Periodogram computes an estimate of the power spectral density using Welch's 's method (Welch 1967..
BioSPPy identifies each beat using the detection algorithm by Carreiras et al. 2015) and computes the mean and standard deviation then concatenates them together to form the representation.
Autoencoder (Hinton) 1990) comprises of 2 MLPs, an encoder with an input size of 2049, hidden layer of dimension 200, and bottleneck representation of 100 dimensions. The decoder has the same architecture in reverse. There are residual connections before each non-linearity and a batch normalization (loffe & Szegedy| 2015) is performed at the bottleneck layer. The model is trained for epochs with Adam (Kingma Ba 2014) a learning rate 10 with the L2 loss.
Our hope is that evaluation using semi-supervised setting on known arrythmia labels (e.g. prema- ture atrial contraction, premature ventricular contraction) and the various rhythm labels (e.g. atrial fibrillation, atrial flutter) is sufficient proxy for the quality of representation - that these rep- resentations will prove useful for discovering unknown disease subtypes. Two models are used to evaluate the representations. We utilize small numbers of samples (N 1000 and N 20000) for evaluation to simulate the situation where a small cohort of patients augmented using the unla- belled data provide. Balanced accuracy is used to compute performance because there is large imbalance between classes. If a model is to predict the same class for all samples the maximum balanced accuracy will be 0.33. We expect that this also becomes source of noise at N 1000

---

Under review as a conference paper at ICLR 2020
KNN
MLP
N
1000
N
20000
N
1000
N
20000
Model
Beat
Rhythhn
Beat
Rhythm
Beat
Rhythn
Beat
Rhythm
Random
0.33=0002
0.33+0000
0.33:1:.01
0.33::0.01
0.33:10.01
0.33+00.11
0.33=0.02
0.33::0.00
Raw Sequence
0.44=:0.03
0.3320.00
0.611:0.01
0.341:0.00
0.54:10.05
0.38:00.88
0.67=0.01
0.33::0.01
PCA 1e
0.50=0004
0.33::0.01
0.65
0.34::0.01
0.55:10.04
0.36::0.07
0.67=0.01
0,33::0.00
PCA 1
0.51=0.03
0.33++0.00
0.64400.01
0.34+00.00
0.55:40.44
0.34:40.06
0.64=0.01
0.33++0.00
PCA R
0.46=0.02
0.34+0.01
0.524:0.11
0.344:0.00
0.47:40.05
0,40:+0.06
0.50=0.01
0,33+00.00
FFT
0.48=00.22
0.37=0.03
0.53::0.01
0.36:20.01
0.50=10.04
0.41=0009
0,54=0.01
0.33+0.00
Periodogram
0.43=00.22
0.35=#0.03
0.47+00.01
0,36+t0.01
0.49::0.03
0.44:10.10
0.53=0.01
0.33++0.00
BioSPPy beat
0.35=0.02
0.34=0.01
0.38+:0.1
0.39:00.01
0.40:10.33
0.34:0.08
0.40=0.01
0.33+0..0
AE (Random init)
0.41=00.03
0.3320.00
0.53
0,3410.00
0.45:10.02
0.36::0.88
0.55=0.00
0.33+0..00
0.51=0.04
0.34#0.01
0.64+0.01
0.34+0.01
0.56:10.02
0.38+0.07
0.66=0.01
0,33+0.00
Table 3: Performance on semi-supervised task computed as balanced accuracy. Given a random
subset of labels from the training set predict the labels in the test set. Evaluated over 10 random subsets.
Figure 6: An analysis of the specific clusters resulting from the PCA features of 100 dimensions visualized with t-SNE. 40,000 example frames were randomly sampled from the test data.
because an underrepresented class has large impact in the performance random predictions get few samples right by chance.
The results are shown in Table 3 Currently autoencoders are not able to perform as well as we expected. PCA is able to perform the best beat detection when using the KNN model while the MLP is able to predict better using the raw signal. One surprise is that rhythm detection is difficult. possible that, because the Periodogram and FFT captures periodicity in the signal, it performs better than the other feature extraction methods. Work by Vollmer et al. (2018) has shown that it is possible in a supervised setting.
The results also shows the issues with using MLPs as classification method for this task. MLPs typically requires more data points for training, and this issue shows up the where there is a higher variance in the accuracy for each subset. The effect is even larger in rhythm classification, where the classes are imbalanced, resulting in huge variations the balanced accu- racy. When more data is available (N 20000), variance is lower. As the ultimate purpose of this task to learn better representation of the ECG signal, having powerful parametric models like an MLP that works well only on higher instance counts may be offloading the representation learning to the classification method, which, as we alluded to before, is not favourable in our setting.
1000 case,
5.2 QUALITATIVE EVALUATION
Medical literature has discussed multiple types of PVC (Kanei et al., 2008 Phibbs 2006). PVCs can be monomorphic or multimorphic (have different morphologies). Additionally. PVCs can also be multifocal and manifest in a different shape. In a multi-lead setting, when arising from the right ventricle, it has a dominant S wave in one particular lead but has a dominant wave if generated from the left ventricle Phibbs 2006).

---

Under review as a conference paper at ICLR 2020
ONT
(a) Raw
(b) FFT
(c) PCA
(d) BioSPPy
(e) AE
Figure 7: t-SNE plot of embeddings produced by different frame-level encoders. Colors represent the three basic labels. Each plot is computed using the same 20,000 frame examples encoded using each method and then having t-SNE applied.
We investigate the clustering of the signals by looking at the PCA encoding of 40,000 frames using t-SNE Figure The plots clearly show two clusters of PVC that we can interpret as two different morphologies of this arrhythmia. We note that these are easy to see because of the different colors we use to highlight the points, but there seems to be remaining clusters that have not been analysed. The correlation between having two clusters for PVCs and PVCs being multimorphic aspect may be of interest to medical researchers to further explore clusters in this space created by different feature extractors.
Many other encoding methods, shown in Figure 7 also show clustering related to PVC and PAC. Notably FFT and BioSppy do not break the PVCs into two clusters. Although we can observe rhythm having some grouping it does not appear significant in the quantitative evaluation.
Such analysis is similar what done by Kachuee al. (2018). However, in that work the features were constructed using supervised task.
CONCLUSION
Single-lead heart monitors like the (DEVICENAME) are increasingly common, and have the potential for cardiologists to learn much more about arrhythmia and related heart diseases. However, this amount of data means manual analysis no longer practical.
Machine learning has been widely deployed in the medical field by training model to predict the right diagnosis based on human expert labels. Supervised learning serves well as an assistant in medical field; however, it hardly provides information beyond human knowledge. Additionally, cer- tain human body signals can very complex and imply non- non-linear features that cannot be easily identifiable manually. At present, representation learning methods have a potential in disentangling complex features, and potentially, unveil new signal structures of certain diseases which can corre- late with clinical presentations.
By releasing this dataset, we believe that we can leverage unsupervised representation learning ex- pertise to not only help to enable training models with lower number of samples, but potentially find new diseases and identify patterns associated with them.
We have proposed an evaluation pipeline for learning feature extractor and evaluating extracted features using known arrhythmia as a proxy to measure the usefulness of the features. In addition, have provided baseline results for frame-level representations under different feature extraction methods. Our data preparation makes three level hierarchy available the segment and patient level grouping of data. While did not provide baselines that exploit this, future work that can
8

---

Under review as a conference paper at ICLR 2020
Whe Necel Vhy in M
Figure 8: Reconstructions using the AE and PCA 100 Two samples are shown, one for each column. The input is shown on the top followed by the AE and then PCA.
take advantage of this context to extract better representations, and perhaps, find more interesting structure in the representation space. We also believe that this dataset can serve as a benchmark in other areas of machine learning, such as anomaly and outlier detection and hierarchical sequence modelling.
REFERENCES
Carlos Carreiras, Ana Priscila Alves, André Lourenço, Filipe Canento, Hugo Silva, Ana Fred, and Others. BioSPPy: Biosignal Processing in Python, 2015.
James W Cooley and John W Tukey. An Algorithm for the Machine Calculation of Complex Fourier Series. Mathematics of Computation, 1965. doi: 10.2307/2003354.
Ary L. Goldberger, Luis A. N. Amaral, Leon Glass, Jeffrey M. Hausdorff, Plamen Ch. Ivanov, Roger G. Mark, Joseph E. Mietus, George B. Moody, Chung- Peng, and H. Eugene Stanley. PhysioBank, Tookio and PhysioNet. 2000 Circulation. doi:
Maya E Guglin and Deepak Thatai. Common errors in computer electrocardiogram interpretation. International journal of cardiology, 2006.
Awni Y. Hannun, Pranav Rajpurkar, Masoumeh Haghpanahi, Geoffrey H. Tison, Codie Bourn, Mintu P. Turakhia, and Andrew Y. Ng. Cardiologist-level arrhythmia detection and classifica- tion in ambulatory electrocardiograms using a deep neural network. Nature Medicine, 2019. doi: 00.10888/41591-01--02266
Geoffrey E Hinton. Connectionist learning procedures. In Machine learning. 1990.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning. 2015.
Steven A. Israel and John M. Irvine. Heartbeat biometrics: a sensing system perspective. Interna- tional Journal of Cognitive Biometrics, 2012. doi: 10. 10.1504/jcb.2012.44611
Alistair E. E.W. Johnson, Tom Pollard, Lu Shen, Li-wei H. Lehman, Mengling Feng. Mohammad Ghassemi, Benjamin Moody, Peter Szolovits. Leo Anthony Celi, and Roger G. Mark. MIMIC-III, a freely accessible critical care database. Scientific Data, 2016. doi: 10. 1..1038/data.2016.55.
Mohammad Kachuee, Shayan Fazeli, and Majid Sarrafzadeh. ECG heartbeat classification: A deep transferable representation. 2018 In International Conference on Healthcare Informatics. doi: N0.1109//CH1.0018.000
Yumiko Kanei, Meir Friedman, Naomi Ogawa, Sam Hanon, Patrick Lam, and Paul Schweitzer. Frequent premature ventricular complexes originating from the right ventricular outflow tract are associated with left ventricular dysfunction. Annals of Noninvasive Electrocundiology. 2008.
Karpagachelvi, M. Arthanari, and M. Sivakumar, ECG Feature Extraction Techniques Survey Approach. In International Journal of Computer Science and Information Security, 2010.

---

Under review as a conference paper at ICLR 2020
Young- Sunng-Gunna Kim, Dahye Shin, Man Young Park, Sukhoon Lee, Min Seok Jeon, Dukyong Yoon, and Rae Woong Park. ECG- ViEW II, a freely accessible electrocardiogram database. PloS one, 2017. doi: 10.137
017622..
Diederik Kingma and Jimmy Ba. Adam: Method for Stochastic Optimization. International
Conference Learning Representations, 2014. doi: 10.1 10.1145/1830483,1 1830503.
Claudia Lerma and Leon Glass. Predicting the risk of sudden cardiac death. 2016 The Journal of Physiology. doi: 10.1 1./313352275355.
Ana Mincholé and Blanca Rodriguez. Artificial intelligence for the electrocardiogram. 2019 Nature Medicine. doi: 00.103//41191-008-000000
G B Moody and R G Mark. The impact of the MIT-BIH arrhythmia database. IEEE engineering in medicine and biology magazine the quarterly magazine of the Engineering Medicine & Biology Society, 2001.
Juan Pablo Martínez, Olle Pahlm, Michael Ringhorn, Stafford Warren, Pablo Laguna, and Leif Sömmoo The STAFF III Database: ECGs Recorded During Acutely Induced Myocardial Is- chemia. In Computing in Cardiology (CinC), 2017. doi: 10.13026/C20P4H.
Pierre Paquet, David Levesque, and Pierre Fecteau. 2019 Adhesive extender for medical electrode anduse thereof with wearable monitor. US Patent App. 16/093,151.
Brendan Phibbs. Advanced ECG: boards and beyond. Elsevier Health Sciences, 2006.
Mihaela Porumb, Ernesto ladanza, Sebastiano Massaro, and Leandro Pecchia. convolutional neural network approach detect congestive heart failure. Biomedical Signal Processing and Control, 2020. doi: (0.1016..BSPP..219101155
Pranav Rajpurkar, Awni Y Hannun, Masoumeh Haghpanahi, Codie Bourn, and Andrew Y Ng- Cardiologist-level arrhythmia detection with convolutional neural networks. arXiv preprint vrXiv:1707.01366. 2017.
Ronald Salloum and C.-C. Jay Kuo. ECG-based biometrics using recurrent neural networks. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2017 IEEE. doi: 10. 10.1100ICASSPP22177795555
Atman Shah and Stanley A. Rubin. Errors in the computerized electrocardiogram interpretation of cardiac rhythm. 2007 Journal of Electrocardiology. doi: 10. 100166fjeeccctoocar..2000.000..0
Chen Song, Feng Lin, Yan Zhuang, Wenyao Xu, Changzhi Li, and Kui Ren. Cardiac Scan: A Non- Contact and Continuous Heart-Based User Authentication System. In International Conference on Mobile Computing and Networking, 2017.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learn- ing Research, 2014.
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 Information Theory Workshop (ITW), 2015.
Mintu P Turakhia, Donald D Hoang, Peter Zimetbaum. Jared D Miller, Victor F Froelicher, Uday N Kumar, Xiangyan Xu, Felix Yang. and Paul A Heidenreich. Diagnostic utility of a novel leadless arrhythmia monitoring device. The American journal of cardiology, 2013.
Mareus Vollmer, Philipp Sodmann, Leonard Caanitz, Neetika Nath, and Lars Kaderali. Can Su- pervised Learning Be Used to Classify Cardiac Rhythms? In 2017 Computing in Cardiology Conference (CinC), volume 44, 2018. doi: 10.22489/cine.2017..47--
Peter D. Welch. The Use of Fast Fourier Transform for the Estimation of Power Spectra: A Method Based on Time Averaging Over Short, Modified Periodograms. Transactions on Audio and Elec- froacoustics, 1967. doi: 10. /09/TAU.1967, 161901.
Özal Yıldımm, Pawel Plawiak, Ru-San Tan, and U Rajendra Acharya. Arrhythmia detection using deep convolutional neural network with long duration ecg signals. Computers in biology and medicine, 2018
10


Reviews:
[{"review_id": "ryg6feKotr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall the paper is well written and organized and is an application of various paradigms to extract most relevant features for Arrhythmia Subtype classification.\nPlan to release the largest (claimed) public ECG data set of continuous raw signals for representation learning containing 11 thousand patients and 2 billion labelled beats.\nOur goal is to enable semi-supervised ECG models to be made as well as to discover unknown sub types of arrhythmia and anomalous ECG signal events.\n\nThe stated intended goal though is the discovery of new  Arrhythmia Sub types (title), “automated arrhythmia detection” (1.1 Objective). \nA definition/example of a regular ECG signal, one presenting arrhythmia, various types of arrhythmia signal forms, would be welcome for the paper to fit this audience and be self-contained.\n\nThis work suffers from many drawbacks:\n-\tA definition/example of a regular ECG signal, one presenting arrhythmia, various types of arrhythmia signal forms, would be welcome for the paper to fit this audience and be self-contained.\n\n-\tGiven the intended audience in ICLR, the application domains specificities are not well explained. Many concepts are not introduced clearly:\n      o\tECG models, sub types of arrhythmia, anomalous ECG signal events.\n      o\tIn related work section: \n            \tParagraph 2: more “leads”, RR interval, PR interval, QRS duration, induced myocardial ischemia.\n            \tParagraph 3: single-lead wearable devices.\n      o\tIn 3 privacy concerns: heartbeats as biometrics section/ Paragraph 2:\n            \talternative ways to sense “cardiac motion”.\n      o\tIn 4 {companyname} 11k data set section:\n            \tParagraph 2: “third line exam”.\n            \tParagraph 3: “beats and rhythms”, these are fundamental to understanding graphical results provided. \n      o\tIn 5.1 quantitative evaluation section:\n            \tParagraph 3: “ .. irregular RR intervals, no distinct P waves and usually variable intervals between two atrial activations. \n            \tPage 6:  “ .. high-level abnormality labels”\n      o\t5.2 qualitative evaluation section\n            \tParagraph 1:  “ .. has a dominant S wave in V1 lead.”\n      o\tIn 6 conclusion section:\n            \t“ Single-lead heart monitors” .\n\n-\tFigures are barely decipherable\n      o\tIn Figure 6: \n            \tWhat are ESSV and ESV, btype and rtype. Could you give a list of known subtypes?\n            \tIs this for PVC data only?\n\n-\tMany paragraphs are not clear to us:\n      o\tIntroduction/Paragraph 2: \n            \t“While cardiologists are able to see these differences, it is hard to conclude that they are ‘real’ by finding the same anomalous signal across multiple time points and patients without a data driven approach.”\n      o\tIn 3 privacy concerns: heartbeats as biometrics section/ Paragraph 3:\n            \tContrary to 1), 3) & 4), is (2) “.. the expression of environmental variables on the heartbeat data is unique to the individual”, a limitation to ECG to being considered a biometric measure?\n      o\tIn 4 {companyname} 11k dataset section:\n            \tParagraph 4:  “we segment each patient record into segments of 220 +1 signal samples ( ~70 minutes). Care to explain the rationale?\n            \tLast paragraph: “ .. we believe that processing the data with these levels of hierarchy results in some grouping information that could be leveraged to attain better results.”. This is a multi-scale approach. But do you have any medical (application domain) knowledge that would justify/hint to using such approach?\n      o\tFigure 5:\n            \tOnly Beats are labelled. No Rhythm labelling?\n      o\tTable 2:\n            \t“Only 2 types of labels are provided”. Are these beats and Rhythms? What are anomalies considered in the study then?\n      o\tIn 5.1 quantitative evaluation section:\n            \tParagraph 2: “..PAC is an abnormal beat only because it appears too soon and disrupts the rhythm (frequency). Furthermore, a PAC beat has the same shape as a normal beat, so taken alone, you can nearly not make the difference with a normal beat.”. A graphical representation would be welcome.\n            \tParagraph 3: “..Both require a representation that will compose a representation showing the difference between beats over time.”.\n      o\tqualitative evaluation section\n            \tParagraph 2: “We note that these are easy to see because of the different colors we use to highlight the points, but there seems to be remaining clusters that have not been analysed.”. The clusters are not evident to us even after purposefully colored.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HygH-rostB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes a large-scale ECG dataset that the authors intend to publish. Along with the to-be-released dataset, the authors also provide some unsupervised analysis and visualization of the dataset. \n\nThe data is collected using the author’s company’s single lead ECG devices. The dataset is collected from 11,000 patients and contains over 2.7 billion peaks. According to the cited works in section 2, the scale of this proposed dataset is unprecedented and will be beneficial to the ECG community.\n\nI think this is an interesting dataset for the ECG machine learning community. Some recent advances in this field are based on non-public dataset. However, this dataset seems to require additional curation before it is ready.\n\n\n\nQuestions. \nIn section 3, the authors discuss the potential concerns regarding privacy, what are the measures taken by the authors when collecting & distributing this dataset? I didn’t find this question addressed in section 3.\n\nLike the authors stated in 2nd paragraph under section 4, this dataset is biased in the sense that it is collected from patients. The subjects are in the age group of 62.2 +/- 17.4. Do the authors recognized various chronicle conditions that often appear in that age group? Is that information part of the to-be-released dataset?\n\nSince this work focuses on releasing  a dataset, I find the current experiment section unnecessary & somewhat unrelated. What would be interesting to see is whether the authors use ML techniques to facilitate the curation of the dataset. For example, in paragraph 3 under section 4, it says each segment data is labeled by 3 technologists. How many technologist are involved in total? and what measure are taken to address the variances among different technologists?\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "N/A", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a new ECG dataset. While I appreciate the efforts to clarify several points raised by the reviewers, I still believe this contribution to be of limited interest to the broad ICLR community. As such, I suggest this paper to be submitted to a more specialised venue.
Decision: Reject