Attention Based Joint Learning for Supervised Electrocardiogram Arrhythmia Differentiation with Unsupervised Abnormal Beat Segmentation
Authors: Xinrong Hu
long wen
shushui wang
Dongpo Liang
Jian Zhuang
Yiyu Shi
Venue:  (2021)
Keywords: interpretability
multitask learning
attention mechanism
electrocardiography

Abstract: Deep learning has shown great promise in arrhythmia classification in electrocar- diogram (ECG). Existing works, when classifying an ECG segment with multiple beats, do not identify the locations of the anomalies, which reduces clinical inter- pretability. On the other hand, segmenting abnormal beats by deep learning usu- ally requires annotation for a large number of regular and irregular beats, which can be laborious, sometimes even challenging, with strong inter-observer variabil- ity between experts. In this work, we propose a method capable of not only dif- ferentiating arrhythmia but also segmenting the associated abnormal beats in the ECG segment. The only annotation used in the training is the type of abnormal beats and no segmentation labels are needed. Imitating human’s perception of an ECG signal, the framework consists of a segmenter and classifier. The segmenter outputs an attention map, which aims to highlight the abnormal sections in the ECG by element-wise modulation. Afterwards, the signals are sent to a classifier for arrhythmia differentiation. Though the training data is only labeled to super- vise the classifier, the segmenter and the classifier are trained in an end-to-end manner so that optimizing classification performance also adjusts how the abnor- mal beats are segmented. Validation of our method is conducted on two dataset. We observe that involving the unsupervised segmentation in fact boosts the clas- sification performance. Meanwhile, a grade study performed by experts suggests that the segmenter also achieves satisfactory quality in identifying abnormal beats, which significantly enhances the interpretability of the classification results.

Under review as a conference paper at ICLR 2021
ATTENTION BASED JOINT LEARNING FOR SUPER- VISED ELECTROCARDIOGRAM ARRHYTHMIA DIFFER- ENTIATION WITH UNSUPERVISED ABNORMAL BEAT
SEGMENTATION
Anonymous authors Paper under double-blind review
ABSTRACT
Deep learning has shown great promise in arrhythmia classification in electrocar- diogram (ECG). Existing works, when classifying ECG segment with multiple beats, do not identify the locations of the anomalies, which reduces clinical inter- pretability. On the other hand, segmenting abnormal beats by deep learning usu- ally requires annotation for a large number of regular and irregular beats, which can be laborious, sometimes even challenging, with strong inter-observer variabil- ity hetween experts. In this work, we propose a method capable of not only dif- ferentiating arrhythmia but also segmenting the associated abnormal beats the ECG segment. The only annotation used in the training is the type of abnormal beats and no segmentation labels are needed. Imitating human's 's perception of an ECG signal, the framework consists of segmenter and classifier. The segmenter outputs an attention map, which aims to highlight the abnormal sections the ECG by element- wise modulation. Afterwards, the signals are sent to classifier for arrhythmia differentiation. Though the training data is only labeled super- vise the classifier, the segmenter and the classifier are trained in an end-to-end manner so that optimizing classification performance also adjusts how the abnor- mal beats are segmented. Validation of our method is conducted on two dataset. We observe that involving the unsupervised segmentation in fact boosts the clas- sification performance. Meanwhile. a grade study performed by experts suggests that the segmenter also achieves satisfactory quality in identifying abnormal beats, which significantly enhances the interpretability of the classification results.
INTRODUCTION
Arrhythmia in electrocardiogram (ECG) is a reflection of heart conduction abnormality and occurs randomly among normal beats. Deep learning based methods have demonstrated strong power in classifying different types of arrhythmia. There are plenty of works on classifying a single beat, involving convolutional neural networks (CNN) (Acharya et al. 2017b] Zubair et al. 2016). long short-term memory (LSTM) (Yildirim) 2018). and generative adversarial networks (GAN) (Golany & Radinsky 2019). For these methods work in clinical setting, however, good segmenter needed to accurately extract single beat from an ECG segment, which may be hard when abnormal beats are present. Alternatively, other works (Acharya et al.] 2017a [Hannun et al., 2019) try to directly identify the genres of arrhythmia present in an ECG segment. The limitation of these works that they work as black-box and fail to provide cardiologists with any clue on how the prediction is made such as the location of the associated abnormal beats.
In terms of ECG segmentation, there are different tasks such as segmenting ECG records into beats or into P wave, QRS complexity, and T wave. On one hand, some existing works take advantage of signal processing techniques to locate some fiducial points of PQRST complex so that the ECG signals can be divided. For example, Pan- Tompkins algorithm (Pan & Tompkins 1985) uses
combination of filters, squaring, and moving window integration to detect QRS complexity. The shortcomings of these methods are that handcraft selection of filter parameters and threshold is

---

Under review as a conference paper at ICLR 2021
needed. More importantly, they unable to distinguish abnormal heartbeats from normal ones. To address these issues, Moskalenko et al. (2019): Oh et al. (2019) deploy CNNs for automatic beat segmentation. However, the quality of these methods highly depends the labels for fiducial points of ECG signals, the annotation process of which can be laborious and sometimes very hard. Besides, due the high morphological variation of arrhythmia, strong variations exist even between annotations from experienced cardiologists. As such, unsupervised learning based approaches might be a better choice.
Inspired by human's 's perception of ECG signals, our proposed framework firstly locates the abnormal beats in an ECG segment in the form of attention map and then does abnormal beats classification by focusing on these abnormal beats. Thus, the framework not only differentiates arrhythmia types but also identifies the location of the associated abnormal beats for better interpretability of the result. It is worth noting that, in our workflow, we only make use of annotation for the type of abnormality in each ECG segment without abnormal beat localization information during training, given the difficulty and tedious effort in obtaining the latter.
We validate our methods on two datasets from different sources. The first one contains 508 12-lead ECG records of Premature Ventricular Contraction patients, which are categorized into different classes by the origin of premature contraction (e.g., left ventricle (LV) or right ventricle (RV)). For the other dataset, we process signals the MIT-BIH Arrhythmia dataset into segments of standard length. This dataset includes various types of abnormal beats, and we select 2627 segments with PVC present and 356 segemnts with Atrial Premature Beat (APB) present. Experiments on both two dataset show quantitative evidence that introducing the segmentation of abnormal beats through an attention map, although unsupervised, can in fact benefit the arrhythmia classification performance as measured by accuracy, sensitivity, specificity, and area under Receiver Operating Characteristic (ROC) curve. At the same time, grade study by experts qualitatively demonstrates our method's promising capability to segment abnormal beats among normal ones, which can provide useful in- sight into the classification result. Our code and dataset, which is the first for the challenging PVC differentiation problem, will be released to the public.
RELATED WORKS
Multitask learning There are many works devoted to training one deep learning models for multi- tasks rather than one specific task, like simultaneous segmentation and classification. (Yang al., 2017) solves skin lesion segmentation and classification at the same time by utilizing similarities and differences across tasks. In the area of ECG signals, (Oh et al., 2019) modifies UNet to output the localization of r peaks and arrhythmia prediction simultancously l. What those two works have in common is that different tasks share certain layers in feature extraction. In contrast, our segmenter and classifier are independent models and there is no layer sharing between them. As can be seen in Figure| [ we use attention maps as a bridge connecting the two models. (Mehta et al. 2018) segments different types of issues in breast biopsy images with UNet and apply discriminative map generated by subbranch of the UNet to the segmentation result as input to a MLP for diag- nosis. However, their segmentation and classification tasks are not trained end-to-end, (Zhou et al. 2019) proposes method for collaborative learning of disease grading and lesion segmentation. They first perform a traditional semantic segmentation task with a small portion of annotated labels, and then they jointly train the segmenter and classifier for fine-tuning with an attention mechanism, which is applicd on the latent features in the classification model, different from our method. An- other difference is that for most existing multitask learning works, labels for each task are necessary, i... all tasks are supervised. Our method, on the other hand, only requires the labels of one task (classification), leading to a joint supervised/unssperviis scheme.
Attention mechanism After firstly proposed for machine translation (Bahdanau et al., 2014), at- tention model became a prevalent concept in deep learning and leads to improved performance in various tasks in natural language processing and computer visions. (Vaswani et al.) 2017) exploits self-attention in their encoder-decoder architecture to draw dependency between input and output sentences. (Wang et al.) 2017) builds very deep network with attention modules which generates attention-awarr features for image classification and (Oktay et al.) 2018) integrates attention gates into U-Net (Ronneberger al. 2015) to highlight latent channels informative for segmentation task.
2

---

Under review as a conference paper at ICLR 2021
spoor
pooting
22-lned
Sogmentation
Classifier
output
Seamenter
ComV
1d
Block
Block 2
Block
Block
GAP
MIP
softmax
autput
30.11
16.11
16.11
(b)
Figure 1: (a) Overview of our framework consisting of a segmenter and classifier. The segmenter is a modified version of U-Net. The input of the classifier is the element-wiss production of the original ECG segment and the attention map generated by the segmenter. The tuple alongside every cuboid represents (number of channels, length of data) for the feature maps. (b) The detailed architecture of the classifier. Following ID point-wise convolutional layer are four convolutional blocks, each containing two combinations of Conv + BN + ReLU, GAP stands for global average pooling while MLP stands for a two-layer perceptron. The elements in the tuple represent channel number and downsample ratio respectively.
When it comes to ECG, (Hong et al., 2019) proposes multilevel knowledge guided attention net- work to discriminate Atrial fibrillation (AF) patients, making the learned models explainable at beat level, rhythm level, and frequency level, which is highly related to work. Our method and theirs however are quite different the way attention weights are derived and applied, as well as the output of attention network. First, in that work, the attention weights are obtained from the outputs of hid- den layers, while ours are directly from the input. Second, domain knowledge about AF is needed to help the attention extraction, the process weakly supervised, while do not any external information and is fully unsupervised. Third, their attention weights are applied to latent features in that work while put ECG segment is divided into equal-length segments in advance and the attention network output only indicates which segment contains the target arrhythmia. The quality highly depends on how the segment is divided, and it does not provide the exact locations of abnormal beats. On the other hand, our method directly locates the abnormal beats on the entire input ECG, offering potentially better interpretability and robustness.
applied to the input for better interpretability. Finally, in that work, the in-
3 METHOD
3.1
OVERVIEW OF THE FRAMEWORK
Here we briefly introduce the workflow of our joint learning frameworks for supervised classification and unsupervised segmentation. Firstly, in this work, we choose to model the input signal as a one-dimensional signal D E RM where M is the number of leads and N is the length of the input ECG segment (number of samples over time). We then use one- mme-iimensional (ID) fully convolutional network called segmenter S to output feature map S(D) E IR MxN After that,

---

Under review as a conference paper at ICLR 2021
apply a pooling layer to generate window-style element- attention A R RMN containing weights directly for every sample in the input ECG. The interrattention signal X = A E RMM where @ represents element- wise production, is then fed into multi-layer CNN called classifier C, in which the outermost fully connected layer gives the prediction of the arrhythmia types. After training, the abnormal areas highlighted in X, thus achieving the goal of segmenting abnormal beats from normal ones. Moreover, I, which indicates those beats that are highly associated with the differentiation task, also serves an explanation for C's decision. The architecture of our framework is illustrated in Fig 1.
3.2 SEGMENTER AND CLASSIFIER
In most existing works, the attention map is fused with the deep features in neural network. How- ever, for our specific purposes of enhancing intepretability of the classification results as well as unsupervised segmentation, the best result would be obtained by directly applying it to the input signal D. In order to generate attention weights of the same length as D. we choose to utilize U- Net (Ronneberger et al.) 2015), fully convolutional network highlighted by the skip connection on different stages. Encoding path extract features recursively and decoding path reconstruct the data as instructed by loss function. Note that the output of S has only channel and expand it channel-wise so that it matches the channel dimension of the ECG signal and at the same time each channel gets the same attention. The reason is that the 12 leads are measured synchronously and the abnormal beats occur at the same time across all the leads.
Both recurrent neural networks (RNN) and CNN are candidate architectures for many arrhythmia classification works. RNN takes an ECG signal as sequential data and is good at dealing with the temporal relationship. CNN focuses on recognition of shapes and patterns in ECG, thus less sensitive to the relative position of abnormal beats with respect to normal ones. Because abnormal beats may occur randomly among normal beats, we decide to use CNN as the backbone of our classifier. The detailed implementation of C' is shown in Pb).
3.3 POOLING FOR WINDOW- STYLE ATTENTION
We do not use the output of the segmenter L as the attention map directly but instead perform pooling with large kernel size first. This is out of considerations for both interpretability and perfor- mance. Regarding interpretability, it desirable that each abnormal beat is uniformly highlighted, i.c. the attention weights should be almost constant and smooth for all the samples within each abnormal beat. Regarding performance, it is desirable that the attention map does not distort the shape of abnormal beats after it is applied to the input X. Pooling layer is the easiest way to achieve this goal, functioning as a sliding window over multiple samples in an BCG signal for global in- formation extraction. Max pooling outputs the same value around a local maximum, and average pooling reduces fluctuation by averaging over multiple samples. The kernel size cannot be too large, which may fuse sharp changes from neighboring areas and lead to the loss of local information. Therefore, deciding the proper pooling kernel size is essentially finding a balance between local information and global information preservation. Through experiments be shown in Section 5.3 we find that setting the kernel size as nearly half the length of a normal beat yields the best balance between performance and interpretability. Padding of zeros on both sides of the segmenter output is implemented to keep the length of the resulting attention map A after pooling to be the same as the input X.
Meanwhile, the polarization of QRS complex is a critical feature for ECG signal, while the com- monly used pooling layers, like max pooling and average pooling, fail control the sign of out- put, leading to differentiation performance degradation. Rectified linear unit (ReLu) (la.m) = mar(0, l,.m), where e and m denote channel number and spatial position in L. respectively, is usu- ally performed as an activation function to add non- non-linearity to neural network for stronger represen- tation ability. In this work, we can apply ReLU on before pooling so that the all the weights in A generated by the following max pooling are positive. Alternatively, we replace average pooling with L2 norm pooling that takes square root of the L2 norm of input. In that case, ReLU is not needed. The two pooling implementations can be expressed as:
Og maz PMMX (T)), -max (o(le,m), (ll,m 1), -r. a (lem+t))
4

---

Under review as a conference paper at ICLR 2021
m++k
acm
L2
(L)
Σ
P.
(2)
where, de,m is the mith data point in channel e of A and k is the kernel size for the pooling.
3.4 JOINT LEARNING
Compared to traditional segmentation network, our segmenter does not give prediction on ev- ery point in the input ECG, instead we generate attention map A and distinguish heartbeats by the amplitude weights in A. The segmentation result is reflected in the after-attention signal X. During training. with only annotation for the arrhythmia differentiation task, the segmentation task actually unsupervised. Unlike clustering or mutual learning, the popular unsupervised segmen- tation methods, train the segmenter and classifier in end-to-end manner and the gradient of classification loss is backpropagated to S for updating how the signal is segmented.
EXPERIMENTS
4.1 DATASET
Our experiments are conducted on two datasets from different sources, The first dataset is collected by MAC5500 machine sample rate of 500Hz. They are all from patients diagnosed with PVC. Further catheter ablation test is performed to confirm the origins of PVC, so all the labels are accurate. For every patient, experienced cardiologists exam the long ECG record and grabs a ECG segment that contains the PVC arrhythmia with fixed length of 5000 samples. In the dataset, there are totally 508 segments, including 135 cases of left ventricle (LV) and 373 cases of right ventricle (RV). Moreover, within left ventricle patients, 91 are left ventricle outflow origins (LVOT) and among right ventricle patients 332 are right ventricle outflow origins (RVOT). It is of clinical interest to classify between LV and RV, and between LVOT and RVOT, problems in our experiments. We will release this dataset to the public, which will be the first for the challenging problem of PVC differentiation.
will explore both
The other dataset is derived from the public MIT-BIH Arrhythmia dataset (Moody & Mark 2001), which includes 48 recordings of 47 patients, all sampled 360 Hz. There two leads for every records. All heart beats in those recordings types include PVC, atrial premature beat (APB), left/right bundle branch block beat, etc. We prepro- cess those ECG signals into segment with standard 2000 samples. More specifically, we accumula- tively add beat of interest to a segment until its length will exceeds 2000 if the next beat is added. Then we pad zeros for that segment to the target size. Among those segments, we focus on 356 segments with only normal beats and APB and 2627 segments with only normal and PVC beats.
annotated by expert cardiologists and arrhythmia
As for preprocessing, we adopt series of filters and remove the high frequency noise and baseline drift. Besides, we apply normalization to each lead independently so that the voltage ranges are all the same for the 12 leads.
4.2 EXPERIMENT SETTING
All our codes are based on the open source machine learning library PyTorch. Architecture details of our segmenter and classifier are shown in Figure [ As for the classifier, inside each block are two serialized Conv + BN + ReLU combinations. The dimensions of weights in the two-layer perceptron are 128 x and 128 optimizer with initial learning rate set to 0.00001. The training epoch is set to 120 as we observe lowest validation loss and highest accuracy can be achieved by then. The loss function for the classification is negative log likelihood loss and add weights for different classes due to the imbalanced distribution of the dataset. We apply five-fold cross-vas vallidvaao with different classes evenly distributed between folds, and the average performance is reported.
respectively. We choose Adam algorithm (Kingma & Baj
2014)
We implement our method with L2 norm pooling as well as a combination of ReLu and max pooling at the output of the segmenter, as discussed in Section 3.3 All the hyperparameters remain the same
5

---

Under review as a conference paper at ICLR 2021
training loss
validation loss
validation accuracy
classfter
0.5
cestatted
0.7
0.4
0.3
0.2
cascaded
Max
Max
norm
1.2 morm
0.3
250
200 250
309
350
60
100 130
65
130
130
(a)
(b)
(c)
Figure 2: (a) Training loss curve, (b) validation loss curve and (c) validation accuracy curve for the classifier only method (Classifier), cascaded segmenter and classifier (Cascaded), our methods with ReLu and max pooling (Max), and with L2 norm pooling (L2 norm) on (LVOT, RVOT) task. There is a clear overfitting for the cascaded segmenter and classifier method.
for both datasets, except for the kernel size which is set to be linearly proportional to the sampling frequency as different sample frequencies result in different beat duration. The kernel size for PVC origin dataset is 200 and that for MIT-BII BIH-BIH dataset 150, As none of the existing works addresses the problem of supervised arrhythmia classification with unsupervised abnormal beat segmentation, we also implement the following two baseline models for comparison: (1) classifier only: Keep the same classifier design, but without segmenter any attention.
(2) cascaded segmenter and
classifier: Feed A instead of A @ D into the classifier and train segmenter and classifier together. The resulting architecture will have the same number of layers as the original one in Figure]
RESULTS
Task
Method
Accuracy%
Speciffiity%
Sensitivity%
AUC
Classifier only
88.5 0.5
93.5
74.8 1.0
0.912 +: 0.005
Cuscaded segmenter/clasiffer
70.5 + 2.5
47.6 1.2
0.728 + 0.030
(RV, L.V
Ours (Rel max pooling)
89.9
+6.4+11.0
+2.0+28
0.918 0.014
Ours (12 norn pooling)
90.2 + 0.6
96.5 0.4
72.7
0.922 + 0.010
Classitier only
8..3
916208
10.1
0.875 = 0.019
Cascuded segmentericlassficc
70.2 70.2
+9.2
37.4
0.618 + 0.066
(RVOT, LOVT)
Ours (Rela max pooling)
$9.6 0.3
96.0+11.3
7.0
0.884 + 0.010
Ours (12 norm pooling)
01.3
9.94
70.3
0.913 + 0.010
Classitier only
98.1
9.9:0.55
919
0.085 = 0.00+
Cascaded segmenter/chassine fier
8-.1+-2.1
9..3+44.4
60.4 8.1
0.928 + 0.050
(PVC, APB)
Ours (ReLa + max pooling)
98.8
9..4
93.0 1.6
0.990 + 0.007
Ours (1.2 norm pooling)
0.1
99.6
93.4 1.3
0.993 + 0.004
Comparison of different methods' performance on two tasks of PVC orgin differentiation and the arrhythmia classification task on MIT-BIH. Our method with 1.2 norm pooling almost always attains the best scores.
Table 1:
5.1 COMPARISON PVC DIFFERENTIATION PERFORMANCE
The metrics we select for comparison include overall accuracy, specificity, sensitivity, and AUC (area under curve) of ROC (Receiver Operating Characteristic) curve. We evaluate the performance of all the methods two tasks: differentiating PVC originating in LV and RV ((VVLV) task), as well as PVC originating in LVOT and RVOT ((VOT, LVOT) task). For (RV, LV) task, the specificity calculated with regards to RV, for (RVOT, LVOT), it is with regards to RVOT, and
and sensitivity it is PVC for (PVC, APB) task. For all tasks, we calculate the AUC of ROC curve for each class and record the average value.
Table lists the results for three baseline methods and our methods with two different pooling. A few meaningful observations be made out of the table. Firstly, in general methods show higher

---

Under review as a conference paper at ICLR 2021
Original ECG
HHM
Hu HigH
Attention map
After attention
(a)
Original ECG
Attention map
1
After attention
(b)
Figure 3: Visual examples of the segmentation results on (a) our dataset and (b) MIT-BIH Arrhyth- mia dataset. The columns represent different cases. For (a), in the original ECG, the abnormal beats occur simultaneously all the 12 leads and are only marked on the first one. Regarding the heat map of the attention, the warmer an area is, the larger the attention weight is.
score in almost all benchmarks including accuracy, specificity, sensitivity and AUC, than the baseline methods. This proves that our attention mechanism indeed improves the classifier's 's capability of PVC origin classification. Secondly, using I.2 norm pooling has better performance than using the combination of ReLU and max pooling, which implies the limitation of ReLU which may lose information in negative values. In contrast, L2 norm pooling preserves the negative information. Finally, the cascaded segmenter and classifier method, which has the same number of layers as our methods, has poor performance. This confirms that the better classification performance of methods actually comes from the attention mechanism instead of deeper architecture. Actually, from Figure we can see that there is large gap between the training loss and the validation loss for the cascaded segmenter and classifier method after several epochs, suggesting apparent overfitting.

---

Under review as a conference paper at ICLR 2021
Class
Original ECG
1000
2000
3000
4000
5000
2000
2000
3030
4000
5000
1000
2000
2000
4000
attention
1000 2000
3000
4000
5000
Figure 4: Illustration of the three classes in grade study. In class I, all abnormal beats identified and all normal beats are removed. In Class II, all abnormal beats are detected, though some normal beats are also included. Class III, no significant difference can be found between abnormal and normal beats.
Moreover, the benefits of adding segmenter to the MIT-BIH dataset seems not large, but it is due to that distinguishing between PVC and APB is much less challenging than differentiating various origins of PVC. Also, the baseline already achieves good performance.
5.2 EVALUATION OF SEGMENTATION AND INTERPRETABILITY
Three visual examples of the segmentation results are shown in Figure 3 From the figure we can see that after applying the attention map to the original ECG signal, the location of the abnormal beats can be easily identified.
We design an independent and blind grade study by an experienced cardiologist to qualitatively evaluate our segmenter's ability to detect abnormal beats for the (RVOT, LVOT) task. In general, qualitative evaluation widely used in attention mechanism related works due to simplicity and visualization (Hul 2019). It is also most suitable to judge the intepretability perspective of the results.
Focusing the after-attention signal X, categorize the segmentation result into three classes by the contrast between abnormal beats and normal beats, as shown in Figure Class I: all normal beats eliminated, and all abnormal beats are kept. Best interpretability is attained. Class II: some normal beats still remain, and all abnormal beats are kept. In this case, interpretability reduced but all abnormal beats can still be identified in X. Class III: there is no significant difference between abnormal beats and normal beats. There is little interpretability.
We randomly select the after-attention signals of 100 ECG segments and the blind grade study result shows that the number of cases in the three classes are 50:27:23 (Class I: Class II: Class III), which implies superior performance of our segmenter. cardiologists quick understanding of why the prediction is made.
practice, these segmentation results provide
5.3 INFLUENCE OF KERNEL SIZE
Fig 5 shows the comparison of classification performance and segmentation result of our framework with different kernel sizes in the I.2 norm pooling layer the output of the segmenter regarding the (RVOT, LVOT) task. We can see apparent degradation of accuracy, specificity, and AUC when the kernel size is increased to 300 while the performance difference between kernel size 100 and 200 is minor. Regarding the segmentation result, additional grade studies are conducted on kernel size 100 and kernel size 300. The ratios between the number cases in the three classes when the kernel size is 200 is close to those when kernel size is 300, showing comparable ability of abnormal beats detection. There a much higher number of class III cases for kemel size 100, suggesting that too small kernels may suffer from poor segmentation and low intrepretability in accordance with the
8

---

Under review as a conference paper at ICLR 2021
30.0%
GLO
21.%
Chaiil
Choo11
1.0
accuracy specificity
0.9
sensitivity
0.8
0.7
0.6
200
300
Figure 5: Comparison classification performance and grade study result for different kernel size with respect to differentiating RVOT and LVOT
analysis in Section 3.3 of 200.
After weighing interpretbility and performance, we choose the kernel size
CONCLUSION AND DISCUSSION
In this paper, we propose novel framework combining unsupervised abnormal beats segmentation and supervised arrhythmia differentiation. The key to the multitask learning is applying an attention map generated by segmenter directly to input data before the classification task. In addition, perform a large-kernel pooling layer to constrain the shape of attention map for better performance and easier interpretability. We use premature ventricular contraction differentiating, one of the most challenging problems in arrhythmia classification as a case study to evaluate effectiveness of our method. On hand, experiment result demonstrates better accuracy with the help of attention map. On the other hand, we observe obvious discrimination between abnormal beats and normal beats from after-attention signal. Indicating enhanced interpretability in clinical practice.
In the future, we expect to extend our method to high-dimension data such as images and videos. In our opinion, the difficulties in applying our framework on 2-D /3-D data are more complicated background information and the need for fine-graide fin-ggained constraint on the attention map shape. When doing arrhythmia classification, the "background" in ECG signal is just normal beats. As for image classification, the "background" can be more complex, like birds flying among flowers and cars driving through streets, etc. Learning the difference between target objects and environment may be harder for the segmenter if without labels. On the other hand, the target objects have higher variations in terms of size, shape and texture, even within the same class, which requires a more elaborate design of constraints on the attention map shape.
REFERENCES
U Rajendra Acharya, Hamido Fujita, Oh Shu Lih, Yuki Hagiwara, Jen Hong Tan, and Muhammad Adam. Automated detection arrhythmias using different intervals tachycardia eeg segments with convolutional neural network. Information sciences, 405:81-90, 2017a.
Rajendra Acharya, Shu Lih Oh, Yuki Hagiwara, Jen Hong Tan, Muhammad Adam, Arkadiusz Gertych, and Ru San Tan. A deep convolutional neural network model to classify heartbeats. Computers in biology and medicine, 89:389 3969 2017b.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning align and translate. arXiv preprint arXiv:1409.044 0473, 2014.
Tomer Golany and Kira Radinsky. Pgans: Personalized generative adversarial networks for ecg syn- thesis to improve -ptient-specific deep eeg classification. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 557-564, 2019.

---

Under review as a conference paper at ICLR 2021
Awni Y Hannun, Pranav Rajpurkar, Masoumeh Haghpanahi, Geoffrey H Tison, Codie Bour, Mintu Turakhia, and Andrew Y Ng. Cardiologist-level arrhtthmia detection and classifica- tion in ambulatory clectrocardiograms using deep neural network. Nature medicine, 25(1):65, 2019.
Shenda Hong, Cao Xiao, Tengfei Ma, Hongyan Li, and Jimeng Sun. Mina: Multilevel knowledge- guided attention for modeling electrocardiography signals. arXiv preprint arXiv: 11333, 2019.
Dichao Hu. An introductory survey on attention mechanisms in nlp problems. In Proceedings of SAI Intelligent Systems Conference, pp. 432-448. Springer, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6988. 2014.
Sachin Mehta, Ezgi Mercan, Jamen Bartlett, Donald Weaver, Joann G Elmore, and Linda Shapiro. Y-net: joint segmentation and classification for diagnosis of breast biopsy images. In Interna- tional Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 893- 901. Springer, 2018.
George B Moody and Roger G Mark. The impact of the mit-bih arrhythmia database. IEEE Engi- neering in Medicine and Biology Magazine, 20(3):45-50, 2001.
Viktor Moskalenko, Nikolai Zolotykh, and Grigory Osipov. Deep learning for eog segmentation. In International Conference Neuroinformatics, pp. 244-254. Springer, 2019.
Shu Lih Oh, Eddie YK Ng, Ru San Tan, and U Rajendra Acharya. Automated beat-wise arrhythmia diagnosis using modified u-net on extended electrocardiograapicc recordings with heterogeneous arrhythmia types. Computers biology and medicine, 105:92-101, 2019.
Ozan Oktay, Jo Schlemper, Loic Le Folgoc, Matthew Lee, Mattias Heinrich, Kazunari Misawa, Kensaku Mori, Steven McDonagh, Nils Y Hammerla, Bernhard Kainz, et al. Attention u-net: Learning where to look for the pancreas. arXiv preprint 1804vv1804.0399/ 2018.
Jiapu Pan and Willis Tompkins, A real-time qrs detection algorithm. JEEE transactions on biomed- ical engineering, (3):230-236, 1985.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234-241. Springer, 2015.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-590899 6008, 2017.
Fei Wang, Mengqing Jiang, Chen Qian, Shuo Yang. Cheng Li, Honggang Zhang, Xiaogang Wang. and Xiaoou Tang. Residual attention network for image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1166-3164, 2017.
Xulei Yang, Zeng Zeng, Si Yong Yeo, Colin Tan, Hong Liang Tey, and Yi Su. A novel multi-task deep learning model for skin lesion segmentation and classification. arXiv preprint arXiv:1703.01025, 2017.
Özal Yildirim. A novel wavelet sequence based deep bidirectional Istm network model for ecg signal classification. Computers in biology and medicine, 96: 06:1889 182-202, 2018.
Yi Zhou, Xiaodong He, Lei Huang, Li Liu, Fan Zhu, Shanshan Cui, and Ling Shao. Collaborative learning of superviee segmentation and classification for medical images. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2079-2088, 2019.
Muhammad Zubair, Jinsul Kim, and Changwoo Yoon. An automated eeg beat classification system using convolutional neural networks. In 2016 6th international conference on IT convergence and security (ICITCS), pp. 1-5. IEEE, 2016.
10


Reviews:
[{"review_id": "rFHOnxFFtzA", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a framework for the classification of arrhythmias in electrocardiogram (ECG) data. The proposed approach performs segmentation and classification of the ECG signal. The segmenter performs segmentation of the signal (also called attention map) even though the term segmentation is not quite correct. This attention-modulated signal is then classified to identify the origin of Premature Ventricular Contraction (PVC).  The proposed approach is evaluated on a dataset from a single machine consisting of 508 segments (I am not sure what “segments” means in this context). The results seem ok, but it is not clear to me what level of performance is required in order to achieve a similar level of performance as an expert.\n\nMain concern is that the proposed approach seems rather ad-hoc: The combination of segmentation (or attention) and classification in a joint fashion seems hardly new and while the results obtained are good, there is no systematic evaluation how the method compares to other state-of-the-art ECG classification methods. Another problem is that the writing in the paper is not always clear and it is often unclear what exactly the authors are doing. As a result, it is quite difficult to exactly assess what the authors have done or what they mean.\n\nDetailed comments:\n\n• What is the output of the classifier? Is this a binary label? Or a multi-class label?\n• The authors write “… the output of S has only 1 channel and we expand it channel-wise so that it matches the channel dimension of the ECG signal …” – What exactly is meant here? In Fig 1 it seems that the segmentation output has naturally 12 channels? Should the segmentation be identical for all channels?\n• “We do not use the output of the segmenter L as the attention map directly but instead perform a pooling with large kernel size first” – Why is this done? What does “large kernel” mean?\n• Where is the attention map in Fig. 1?\n• How are the Premature Ventricular Contraction (PVC) origin labels defined? Is that a single time point (per channel or common for all channels) or a time window? \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "nsYbfQVfIwZ", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method for segmentation and classification of ECG data applied to the task to segmenting and detecting Premature Ventricular Contractions (PVC). The taks is semi-supervised, in the sense that segmentation labels are not required by labels for the PVC events (classification) are used.\nThe authors motivate this application quite well and detecting abnormalities in ECG signals is an important task of clinical relevance.  I can understand why segmentation labels may be very laborious to collect and unsupervised methods would be desirable.\n\nThe proposed approach builds upon U-Net and introduces some task specific changes.  However, I would argue that this is primarily an application paper. I don't mean that as a criticism necessarily, I think that strong and well motivated applications of machine learning are important and informative. However, it would be helpful if the authors could discuss more about how their approach might generalize to other tasks, both the detection of other types of arrythmias and other temporal segmentation and classification tasks.  \n\nMy main comments regarding the paper are around the experimental evalutation.  The authors highlight that there are some published baselines for this task or at least similar related works (e.g., Moskalenko et al. (2019); Oh et al. (2019)) and/or the authors could have applied classification on top of features extracted using Pan-Tompkins - but that would be a more crude baseline.  While I recognize that these approaches might not enable unsuperivsed segmentation and so direct comparisons on that might be hard with the full approach they propose.  It might be possible to present a comparison of classification metrics on their own. Perhaps I am misunderstanding but it doesn't seem as though Table 1 includes such a comparison, rather the baselines are different from the previous published methods - is that correct?  I would almost describe Table 1 as ablation results rather than a comparison with other published baselines. I'd like to know the author's response to that and if Table 1 does show these results perhaps linking the rows to the previous approaches might be helpful?  Or justifying why it isn't appropriate to show these comparisons. I don't say this just because the authors should show better numbers, but rather to ground the chose baselines in the context of previous work in this space.\n\nBuilding from the previous point. I think this paper would be an excellent case for for showing transfer learning results, it seems to me that PhysioNet provides a large amount of available data for ECG classification.  A couple of question I'd like to hear the authors responses to:\n1) Why did they not do any experiments on these public datasets?  Is there a reason they are not appropriate?  Do they not have the right labels, are they not large enough, do you need full 12 lead recordings (I am not sure if they are avaiable on PhysioNet datasets - but I imagine so.)\n2) Even if training your method on your dataset is preferable, it would seem natural to test it on a set from PhysioNet, perhaps even with a different type of arrythmia, to see how much performance degrades? This I think would be most informative, both showing segmentation and classification results.\n\nFig. 3 is a nice illustration, but it is quite difficult to read.  I might suggest reorganizing it.  I am not sure showing multiple leads is necessary and maybe limiting to two columns might help.  I'd encourage the authors to leverage supplementary material to show more examples as I do think these help.  \n\nFinally, physiological signals are notorious for having large individual variation.  I'd be interested to have the authors discuss more about this. I couldn't find the information about how the train/val/test splits were organized and whether this was person independent etc.  The following sentence in Section 4.2 \"We apply five-fold cross-validation with different classes evenly distributed between folds, and the average performance is reported\" doesn't seem to mention that.  Knowing more about the splits would be very helpful.  This is perhaps another reason that performing experiments on at least one PhysioNet dataset would be helpful as the train, val, test splits could be released.  But I acknowledge that the authors say they will release their data which is good.", "extra_scores": {}, "extra_text": {}}, {"review_id": "lTXSA9uRWAe", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a deep neural network for Premature Ventricular Contraction (PVC) differentiation and segmentation from electrocardiogram (ECG) signals. The network is jointly trained as a segmenter and a classifier with a multitask learning manner. Differentiation is achieved by the classifier, and segmentation is achieved by pooling for window-style attention from segmenter’s output. Quantitative experiments show better performance than baselines on differentiation tasks. Qualitative experiments show the effectiveness of segmentation tasks. \n\nThe results look interesting, and it might have a broader impact on practical usage for AI models in the clinical environment. However, my concerns are: \n\n1) The topic seems too narrow for the computer science community. More likely a paper of the biomedical engineering community or computing cardiology community. The proposed method also lacks in-depth technical/theoretical analysis; thus the paper novelty is limited. \n\n2) The related works include multitask learning and attention mechanisms. But (image) segmentation works are also worth (or even more) investigating. Just a simple modification of image segmentation neural networks (such as Conv2D -> Conv1D) can make them suitable for ECG segmentation tasks. \n\n3) For the evaluation of segmentation, only several cases of qualitative evaluations are not convincing. At least, a comprehensive user study by a community of cardiologists is needed. \n\nSome questions:\n\n- Could you provide more details about data preprocessing? Which filters do you use? What are the cut-off frequencies for high-pass filter and low-pass filter? \n\n- In figure 3, are there duplicate attention maps in every column?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "7AM3ttMwsxI", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This manuscript contributes a neural architecture to classify arrhythmia type from ECG data. The signal treated as 1D, and the architecture performs joint segmentation-classification detecting the abnormal beats and then classifying them as a function of their origine. It uses U-nets for segmentation and, for classification CNN and one fully-connected layer. The unet segmentation generates weights that are considered as an attention map and multipled with the original time series after pooling on a window (which amounts to smoothing).\n\nCompared to the prior art, the central contribution put forward is the addition of the segmentation component of the architecture.\n\nThe work is light on theory and the contribution mostly resides on the empirical improvement. However, the evidence for this improvement is not rock solid, as it is shown on a single dataset, which has a rather small sample size. Also, I fear that hyper-parameters are not set fully independent of the final error measure.\n\nHow are hyper-parameters (such as learning rate or architecture parameters) chosen? Given the procedure exposed in section 5.2, it seems to me that some of the architecture parameters (kernel size) where not chosen independently of the test set. Such choice will incur a positive bias with regards to the actual expected generalization error.\n\nWith n=500 and an accuracy of 90%, the p=.05 confidence interval of a binomial model is 5%. Hence, the improvements observed by adding the segmentation on top of the classifier do not seem really significant.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper received 4 reviews with mixed initial ratings: 5, 6, 4, 4. The main concerns of R1, R4 and R2, who gave unfavorable scores, included: insufficient evaluation (lack of experiments on public datasets, small sample size), an ad-hoc nature and overall limited novelty of the method, a number of issues with the presentation. In response to that, the authors submitted a new revision and provided detailed answers to each of the reviews separately. After having read the rebuttals, the reviewers (including R3, who initially gave a positive rating) felt that this work overall lacks methodological novelty and does not meet the bar for ICLR.
As a result, the final recommendation is to reject.
Decision: Reject