Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes an active learning framework called SoCal that is consistency-based and can decide between whether to make use of the oracle to provide a label or to make use of a pseudo-label generated by the algorithm itself instead. The proposed method hopes to address resource-constrained active learning scenarios where the oracle is not always available or we wish to make use of the oracle as infrequently as possible. Experimental results demonstrate reasonable performance on four publically available physiological datasets.  Experimental results when the oracle is noisy is also reported.

Overall, I  think the paper proposes a reasonable approach to tackle various challenges one might encounter in the resource-constrained active learning setting. I have the following concerns:

1. Perhaps the major concern I have is the novelty of this paper. While the paper proposes a framework that harnesses several elements in design that are helpful to its success, most of these elements are not new to the literature. Perturbation in samples and network parameters are widely used practices in achieving consistency. Deciding to query the oracle or not is also a problem studied in active learning.

2. Related to the first point, because of the lack of obvious novelty in the paper, providing insights and demonstrating why the proposed composition of design elements should work is all the more important. However, I find that the paper is not very insightful. There is a lack of justification and insights in explaining why the proposed method should work. As a remedy, theoretical justification of some sorts and ablation studies will be desirable.

Miscellaneous:
1. Why the Gaussian assumption made in the paper is reasonable?
2. Any thoughts on why MCP or MCD alone "can fail to identify instances that lie in the region of uncertainty"?
3. When there is no oracle, or when there is a noisy oracle, is it reasonable to compare the proposed method with methods that deal with noisy labels such as semi-supervised learning?
4. While I understand that learning from physiological signals can be an important use case for the proposed method, the proposed method also seems to be general enough to be applied to other non-healthcare datasets. Is there any consideration on why or why not running the proposed methods on these datasets?


=======Post Discussion==========

After reading the author's response and other reviewers' reviews, I still find the novelty of this paper somewhat insufficient. Therefore, I would like to maintain my initial evaluation.



review_title: Incremental in Novelty, Good Empirical Performance

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes an active learning framework which relies on a consistency-based acquisition function and a selective querying method. The idea is interesting and the paper is well written; however, I have the following comments and questions:

he motivation and intuition behind the proposed acquisition functions should be in the main text instead of appendix as this is one of the main contributions of the work (at least partially). Also Figure 6 in the Appendix does not really provide an intuition; it’s basically showing the description in the text by the figure. The terms NSR, AFib and LBBB are introduced in the figure without any definition in the text (I believe these are your e cardiac arrhythmia labels but the reader not familiar with these terms may have trouble figuring it out.). For a more intuitive figure, I suggest using a synthetic toy example. 

I believe the ‘selective query questioning’ module is quite similar to the approach proposed by Yoo and Kweon in ‘Learning Loss for Active Learning’ in CVPR 2019. The main difference is that the loss prediction module in the latter is more general and can support tasks other than classification. Can you elaborate on the advantages of your approach over theirs?

Can you also provide the same analysis that you’ve done for D2 and D1 in Fig 4(a, b) for other datasets and \beta values? I couldn’t find it in the appendix.  The results in the Appendix (I) do not provide a coherent story which makes it hard to judge the pros and cons of the method. Also it would be nice if you add some more standard datasets such as CIFAR-10 as it helps comparing the method with many other active learning baselines in the literature. 

It’s a bit unintuitive to think about examples that the acquisition function is considering informative but the ‘selective oracle questioning’ is preferring pseudo labels for them over oracle labels. In other words, why should these two modules disagree on an instance? If we are highly uncertain about the label of an instance under what scenarios the pseudo label for that instance helps the model? What’s the advantage of your approach over using an approach which only has an acquisition function based on loss prediction (e.g. ‘Learning Loss for Active Learning’ )? 

Experiments in 6.2: Do you observe the same behavior for other beta thresholds? 

Overall, I think the paper is generally well written and the idea is promising but I believe the strengths and weaknesses of the method need to be discussed in more details. It’s totally fine that the model fails on some datasets and shows significant improvement over baselines in some other datasets; however, in the current version of the paper the reader is not able to understand what are the success and failure chances of the method. The experiments in the appendix fail to provide a coherent story or a discussion on why the method is not performing well in some experiments (e.g. ~40% of the experiments in test set performance without an oracle).

-----------------------------------------------
Post-rebuttal comments:
I'd like to thank the authors for adding the experiments; the paper looks stronger now but unfortunately, the results on the new experiments are not that encouraging. Considering the results and also other reviews; I'd like to keep my score as marginally below acceptance threshold. 
review_title: Interesting idea but the empirical evidence is inconclusive 

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes the framework that dynamically determines whether to request a label from an oracle or to generate a
pseudo-label instead. It is claimed to remain strong performance with noisy oracle. In this work, the unlabelled instances in proximity to the decision boundary are considered more informative. After performing Monte Carlo Perturbations on unlabeled data and feeding into networks, the samples with significant different outputs are considered close to the boundary.  This paper proposes to perturb both instances and parameters as shown in Equation (1)(2).

Important technical details are missing. For example, based on the two Gaussian distributions of perturbed networks and the Hellinger distance of the distributions to determine the probability of requesting a label from an oracle as in Equation (5). It is not mentioned how to generate the pseudo-label. The connection of the probability in Equation (5) and previous sections are not close. 

My main concern is that the novel contribution of this work is not strong. Though this work proposed to consider both the instance perturbation, parameters perturbation, temporal information and so on. They are not strongly motivated and no related references to support the motivations. There are also no related empirical experiments to support the motivations.


review_title: paper is not self-contained, important technical details are missing, novel contributions are not clear

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors proposed a consistent-based active learning framework to annotate largely unlabeled physiological signals with the help of human annotators (oracles). The paper is well organized and easy to follow. It is somewhat novel to equipping active learning with consistency learning and selective classification. The experiments (along with the Appendix) give a comprehensive analysis of the method. 

However, I do have the following concerns: 

First, there might be a simple baseline to deal with unavailable labels: to augment all unavailable data with pseudo labels (such as predictions of the current model, or assign with nearest neighbor’s label), and train on [oracle labeled data] + [pseudo labeled data] together. How is this method compared with yours? 

Second, both title and abstract emphasize physiological signals. But the proposed method is not related to physiological signals. And it seems no special design for physiological signals. Only all four experimental datasets are physiological signals - ECG and PPG. However, if it is targeting physiological signals, others like blood pressure (BP), heart rate (HR), and electroencephalogram (EEG) are also important physiological signals. It is necessary to see how this method performs on those data. 

Third, datasets have different sampling rates, but this paper used the same model architecture (in Appendix H.1). And such architecture is not a first-choice model for any of the four datasets. For D1, D2, D3, one can find many other delicate designed models from the challenge papers. For D4, Hannun et al., 2019 also used a much deeper neural network. Thus, in Table 1, we can see that even 100% Oracle is much lower than reported numbers in other papers. 

Some minors from Table 1: For AUC, do you mean area under ROC-curve, precision-recall curve, or the others? If it is ROC-AUC, it is confusing that table 1 AUCs of most methods on D4 are lower than 0.5 - even worse than a random classifier. Besides, BALC seems no obvious positive effect than BALD. 

review_title: Better to see more types of datasets and network architectures, and how about pseudo labels augmentation?

Meta-review: The reviewers still have several concerns about the paper after the author feedback stage: the novelty of the paper is not sufficient; the experimental results are not very encouraging. We encourage the authors fixing these issues in the next revision.
Decision: Reject