Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This manuscript contributes a neural architecture to classify arrhythmia type from ECG data. The signal treated as 1D, and the architecture performs joint segmentation-classification detecting the abnormal beats and then classifying them as a function of their origine. It uses U-nets for segmentation and, for classification CNN and one fully-connected layer. The unet segmentation generates weights that are considered as an attention map and multipled with the original time series after pooling on a window (which amounts to smoothing).

Compared to the prior art, the central contribution put forward is the addition of the segmentation component of the architecture.

The work is light on theory and the contribution mostly resides on the empirical improvement. However, the evidence for this improvement is not rock solid, as it is shown on a single dataset, which has a rather small sample size. Also, I fear that hyper-parameters are not set fully independent of the final error measure.

How are hyper-parameters (such as learning rate or architecture parameters) chosen? Given the procedure exposed in section 5.2, it seems to me that some of the architecture parameters (kernel size) where not chosen independently of the test set. Such choice will incur a positive bias with regards to the actual expected generalization error.

With n=500 and an accuracy of 90%, the p=.05 confidence interval of a binomial model is 5%. Hence, the improvements observed by adding the segmentation on top of the classifier do not seem really significant.


review_title: Empirical evidence has some loopholes

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes a framework for the classification of arrhythmias in electrocardiogram (ECG) data. The proposed approach performs segmentation and classification of the ECG signal. The segmenter performs segmentation of the signal (also called attention map) even though the term segmentation is not quite correct. This attention-modulated signal is then classified to identify the origin of Premature Ventricular Contraction (PVC).  The proposed approach is evaluated on a dataset from a single machine consisting of 508 segments (I am not sure what “segments” means in this context). The results seem ok, but it is not clear to me what level of performance is required in order to achieve a similar level of performance as an expert.

Main concern is that the proposed approach seems rather ad-hoc: The combination of segmentation (or attention) and classification in a joint fashion seems hardly new and while the results obtained are good, there is no systematic evaluation how the method compares to other state-of-the-art ECG classification methods. Another problem is that the writing in the paper is not always clear and it is often unclear what exactly the authors are doing. As a result, it is quite difficult to exactly assess what the authors have done or what they mean.

Detailed comments:

• What is the output of the classifier? Is this a binary label? Or a multi-class label?
• The authors write “… the output of S has only 1 channel and we expand it channel-wise so that it matches the channel dimension of the ECG signal …” – What exactly is meant here? In Fig 1 it seems that the segmentation output has naturally 12 channels? Should the segmentation be identical for all channels?
• “We do not use the output of the segmenter L as the attention map directly but instead perform a pooling with large kernel size first” – Why is this done? What does “large kernel” mean?
• Where is the attention map in Fig. 1?
• How are the Premature Ventricular Contraction (PVC) origin labels defined? Is that a single time point (per channel or common for all channels) or a time window? 

review_title: CNN-based approach for segmentation and classification of ECG signals which is quite ad-hoc and limited novelty

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents a method for segmentation and classification of ECG data applied to the task to segmenting and detecting Premature Ventricular Contractions (PVC). The taks is semi-supervised, in the sense that segmentation labels are not required by labels for the PVC events (classification) are used.
The authors motivate this application quite well and detecting abnormalities in ECG signals is an important task of clinical relevance.  I can understand why segmentation labels may be very laborious to collect and unsupervised methods would be desirable.

The proposed approach builds upon U-Net and introduces some task specific changes.  However, I would argue that this is primarily an application paper. I don't mean that as a criticism necessarily, I think that strong and well motivated applications of machine learning are important and informative. However, it would be helpful if the authors could discuss more about how their approach might generalize to other tasks, both the detection of other types of arrythmias and other temporal segmentation and classification tasks.  

My main comments regarding the paper are around the experimental evalutation.  The authors highlight that there are some published baselines for this task or at least similar related works (e.g., Moskalenko et al. (2019); Oh et al. (2019)) and/or the authors could have applied classification on top of features extracted using Pan-Tompkins - but that would be a more crude baseline.  While I recognize that these approaches might not enable unsuperivsed segmentation and so direct comparisons on that might be hard with the full approach they propose.  It might be possible to present a comparison of classification metrics on their own. Perhaps I am misunderstanding but it doesn't seem as though Table 1 includes such a comparison, rather the baselines are different from the previous published methods - is that correct?  I would almost describe Table 1 as ablation results rather than a comparison with other published baselines. I'd like to know the author's response to that and if Table 1 does show these results perhaps linking the rows to the previous approaches might be helpful?  Or justifying why it isn't appropriate to show these comparisons. I don't say this just because the authors should show better numbers, but rather to ground the chose baselines in the context of previous work in this space.

Building from the previous point. I think this paper would be an excellent case for for showing transfer learning results, it seems to me that PhysioNet provides a large amount of available data for ECG classification.  A couple of question I'd like to hear the authors responses to:
1) Why did they not do any experiments on these public datasets?  Is there a reason they are not appropriate?  Do they not have the right labels, are they not large enough, do you need full 12 lead recordings (I am not sure if they are avaiable on PhysioNet datasets - but I imagine so.)
2) Even if training your method on your dataset is preferable, it would seem natural to test it on a set from PhysioNet, perhaps even with a different type of arrythmia, to see how much performance degrades? This I think would be most informative, both showing segmentation and classification results.

Fig. 3 is a nice illustration, but it is quite difficult to read.  I might suggest reorganizing it.  I am not sure showing multiple leads is necessary and maybe limiting to two columns might help.  I'd encourage the authors to leverage supplementary material to show more examples as I do think these help.  

Finally, physiological signals are notorious for having large individual variation.  I'd be interested to have the authors discuss more about this. I couldn't find the information about how the train/val/test splits were organized and whether this was person independent etc.  The following sentence in Section 4.2 "We apply five-fold cross-validation with different classes evenly distributed between folds, and the average performance is reported" doesn't seem to mention that.  Knowing more about the splits would be very helpful.  This is perhaps another reason that performing experiments on at least one PhysioNet dataset would be helpful as the train, val, test splits could be released.  But I acknowledge that the authors say they will release their data which is good.
review_title: This paper presents an semi-supervised approach for ECG segmentation and PVC classification. The application is well motivated. I have some concerns about the experimental evaluation and novelty described below.  I think it has the makings of a promising paper but would like to see responses to these questions.

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes a deep neural network for Premature Ventricular Contraction (PVC) differentiation and segmentation from electrocardiogram (ECG) signals. The network is jointly trained as a segmenter and a classifier with a multitask learning manner. Differentiation is achieved by the classifier, and segmentation is achieved by pooling for window-style attention from segmenter’s output. Quantitative experiments show better performance than baselines on differentiation tasks. Qualitative experiments show the effectiveness of segmentation tasks. 

The results look interesting, and it might have a broader impact on practical usage for AI models in the clinical environment. However, my concerns are: 

1) The topic seems too narrow for the computer science community. More likely a paper of the biomedical engineering community or computing cardiology community. The proposed method also lacks in-depth technical/theoretical analysis; thus the paper novelty is limited. 

2) The related works include multitask learning and attention mechanisms. But (image) segmentation works are also worth (or even more) investigating. Just a simple modification of image segmentation neural networks (such as Conv2D -> Conv1D) can make them suitable for ECG segmentation tasks. 

3) For the evaluation of segmentation, only several cases of qualitative evaluations are not convincing. At least, a comprehensive user study by a community of cardiologists is needed. 

Some questions:

- Could you provide more details about data preprocessing? Which filters do you use? What are the cut-off frequencies for high-pass filter and low-pass filter? 

- In figure 3, are there duplicate attention maps in every column?

review_title: Interesting paper, but the topic is too narrow; related image segmentation works needed

Meta-review: This paper received 4 reviews with mixed initial ratings: 5, 6, 4, 4. The main concerns of R1, R4 and R2, who gave unfavorable scores, included: insufficient evaluation (lack of experiments on public datasets, small sample size), an ad-hoc nature and overall limited novelty of the method, a number of issues with the presentation. In response to that, the authors submitted a new revision and provided detailed answers to each of the reviews separately. After having read the rebuttals, the reviewers (including R3, who initially gave a positive rating) felt that this work overall lacks methodological novelty and does not meet the bar for ICLR.
As a result, the final recommendation is to reject.
Decision: Reject