SoCal: Selective Oracle Questioning for Consistency-based Active Learning of Cardiac Signals
Authors: Dani Kiyasseh
Tingting Zhu
David A. Clifton
Venue:  (2021)
Keywords: Active learning
consistency-training
cardiac signals
healthcare

Abstract: The ubiquity and rate of collection of cardiac signals produce large, unlabelled datasets. Active learning (AL) can exploit such datasets by incorporating human annotators (oracles) to improve generalization performance. However, the over-reliance of existing algorithms on oracles continues to burden physicians. To minimize this burden, we propose SoCal, a consistency-based AL framework that dynamically determines whether to request a label from an oracle or to generate a pseudo-label instead. We show that our framework decreases the labelling burden while maintaining strong performance, even in the presence of a noisy oracle.

Under review as a conference paper at ICLR 2021
SOCAL: SELECTIVE ORACLE QUESTIONING FOR CONSISTENCY-BASER ACTIVE LEARNING OF CAR- DIAC SIGNALS
Anonymous authors Paper under double-blind review
ABSTRACT
The ubiquity and rate of collection of cardiac signals produce large, unlabelled datasets. Active learning (AL) can exploit such datasets by incorporating human annotators (oracles) to improve generalization performance. However, the over- reliance of existing algorithms on oracles continues to burden physicians. To minimize this burden, we propose SoCal, consistency-based AL. framework that dynamically determines whether to request label from an oracle or to generate a pseudo-label instead. We show that our framework decreases the labelling burden while maintaining strong performance, even the presence of a noisy oracle.
INTRODUCTION
The success of modern-day deep learning algorithms in the medical domain has been contingent upon the availability of large, labelled datasets (Poplin et al., 20188 Tomašev et al. 2019 Attia et al. 2019). Curating such datasets, however, is a challenge due the tume-consuming nature of, and high costs associated with, labelling. This is particularly the case in the medical domain where the input of expert medical professionals is required. One way of overcoming this challenge and exploiting large, unlabelled datasets is via the active learning (AL) framework (Settles 2009). This framework iterates over three main steps: 1) a learner is tasked with acquiring unlabelled instances, usually through an acquisition function, 2) an oracle (e.g. physician) is tasked with labelling such acquired instances, and 3) the learner is trained on the existing and newly-labelled instances.
By altering the way in which acquisitions are performed and the degree of involvement of the oracle, the active learning framework aims to improve the performance of network while minimizing the burden of labelling on the oracle. One principal desideratum for an acquisition function is its ability reduce the size of the version space, the set of hypotheses (decision boundaries) consistent with the labelled training instances. This ability is highly dependent upon the approximation of version space, goal that Monte Carlo Dropout (MCD) attempts to achieve (see Fig, Tap. For example, state-of-the-art imcertainty-baser acquisition functions, such as BALD (Houlsby al.) 2011), used alongside MCD acquire instances that lie in a region of uncertainty, a region where there is high disagreement between the hypotheses about particular instance. In many scenarios, however, estimating this region uncertainty is nontrivial. Furthermore, existing AL frameworks overly reliant on the presence of oracle. Such over-reliance precludes the applicability of AL algorithms to certain environments, such as low-resource healthcare settings, where an oracle is either unavailable or ill-trained for the task hand.
Contributions. In this work, we aim to design an active learning framework that better estimates the region of uncertainty and decreases its reliance on an oracle. Our contributions are as follows:
Consistency-basee active learning framework: we propose a novel framework that stochastically perturbs inputs, network parameters, or both to guide the acquisition of unlabelled instances.
2. Selective oracle questioning: propose dynamic strategy which learns, for acquired unlabelled instance, whether to request label from an oracle or to generate pseudo-label instead.

---

Under review as a conference paper at ICLR 2021
(a) MCD
(b) MCP
(c) BALC
Figure I: Labelled instances from two classes and unlabelled instances (gray) alongside the Version space of (a) MCD where each MC sample is viewed as distinct hypothesis (decision boundary), (b) MCP where there is one hypothesis but several perturbations of the unlabelled instance, and (c) BALC where there are several hypotheses in addition to the unlabelled instance and its perturbed counterpart.
2 RELATED WORK
Actíve learning methdologies were recently reviewed by Settles (2009). In the healthcare domain, Gong et al. (2019) propose to acquire instances from an electronic health record (EHR) database using a Bayesian deep latent Gaussian model to improve mortality prediction. Smailagic et al. (2018) 2019) acquire unannotated medical images by measuring their distance in a latent space to images in the training set. The work of Wang et al. (2019) is similar to ours in that they focus the
cleetrocardiogram (ECG). Gal ct al. (2017) ) adopt BALD (Houlsby et al.) 2011) in the context of Monte Carlo Dropout to acquire datapoints that maximize the Jensen- -hassnn-Shanon divergence (JSD) across MC samples. Previous work attempts to learn from multiple or imperfect oracles (Dekel et al. 2012 Zhang & Chaudhuri] 2015 (inha al.) 2019). For example, Urer et al. (2012) propose choosing the oracle that should label a particular instance. Unlike our approach, they do not explore independence from an oracle. Yan al. (2016) consider oracle abstention in an AL setting. Instead, we place the decision of abstention under the control of the learner. To the best of our knowledge, previous work, in contrast to ours, has assumed the existence of an oracle and has not explored dynamic oracle selection strategy.
Consistency training in the context of semi-supervised learning helps enforce the smoothness as- sumption (005)) For example, Interpolation Consistency Training (Verma et al. (2119) penalizes networks for not generating linear combination of outputs in response to a linear combination of inputs. Similarly, Xie et al. (2019) penalizes networks for generating drastically different outputs response to perturbed instances. the process, networks learn perturbation-invart
representa- tions. McCallumzy & Nigamy (1998) introduce acquisition function that calculates the average Kullback- divergence, DKL, between the output of network and the consensus output across all networks in an ensemble. Unlike ours, their approach does not exploit perturbations. Similar to our work is that of Giao et al. (2019) which incorporates into the objective function a consistency-loss based on the DEL L and actively acquires instances using the variance of the probability assigned to each class by the network in response to perturbed versions of the same instance.
Selective classification imbues a network with the ability to abstain from making predictions. Chow (1970): Yaniv & Wiener (2010) introduce the risk-coverage trade-off whereby the empirical risk of model inversely related to its rate of abstentions. Wiener & El- El-Yaniv| (201I) use a support vector machine (SVM) to rank and reject instances based on the degree of disagreement between hypotheses. In some frameworks, these are the same instances that active learning views as most informative. Cortes et al. (2016) outline an objective function that penalizes abstentions that are inappropriate and frequent. Most recently, Liu et al. (2019) propose the gambler's loss to learn a selection function that determines whether instances are rejected. However, this approach is not implemented in the context of AL. Most similar to our work is SelectiveNet (Geifman & El- Yaniv 2019) where multi-heal architecture is used alongside an empirical selective risk objective function and percentile threshold. However, their work assumes the presence of ground truth labels and, therefore, does not extend to unlabelled instances.
2

---

Under review as a conference paper at ICLR 2021
3 BACKGROUND
3.1 ACTIVE LEARNING
Consider learner fo E R + E IR,, parameterized by w, that maps mn-dimensional input, I, to a dddimenssionl representation, u. Further consider Io v E R + y E RC that maps d- dimensional representation, to C-dimensionall output, y, where C is the number of classes. After training on a pool of labelled data I. = (X f. Yt. for T epochs, the learner is tasked with querying the unlabelled pool of data U (Xu- ; Yu) and acquiring the top fraction of instances, 36 ~ Xy. that it deems to be most informative. The degree of informativenes: of an instance is determined by an acquisition function, a, such BALD (Houlsby al. 2011). Additional acquisition functions can be found in Appendix A These are typically used in conjunction with Monte Carlo Dropout (Gal Ghahramani 2016) identify instances that lie in the region of uncertainty, a region in which hypotheses disagree the most about instances.
4
METHODS
4.1
CONSISTENCY-BASER ACTIVE LEARNING
4.1.1
MONTE CARLO PERTURBATIONS
Unlabelled instances in proximity to the decision boundary are likely be more informative for training than those further away. To identify such instances, we stochastically perturb them and observe the network's outputs. The intuition is that such outputs will differ significantly across the perturbations for instances close to the decision boundary (see Fig. [b]. We refer to this setup as Monte Carlo Perturbations (MCP) and illustrate its derivation in Appendix
4.1.2 BAYESIAN ACTIVE LEARNING BY CONSISTRNCY
Acquisition functions dependent upon perturbations applied to either the inputs (MCP) or the network parameters (MCD) alone can fail to identify instances that lie in the region of uncertainty. We illustrate this point with the following example: without loss of generality, let us assume an unlabelled instance is in proximity to some decision boundary A and is classified by the network as belonging to some arbitrary class 3. Such proximity should deem the instance informative for the training process (Settles) 2009). In the MCD setting, perturbations generating various decision boundaries, which in turn influence the network outputs. In Fig. 2 (red rectangle), visualize such outputs for three arbitrary classes. If these parameter perturbations happen to be too small in magnitude, for example, then the network will continue to classify the instance as belonging the same class. At this stage, regardless of whether an uncertainty-bassd consistency- based acquisition function is used, the instance would be deemed uninformative, and thus not acquired. As result, an instance that should have been acquired (due to its proximity the decision boundary) was erroneously deemed uninformative. A similar argument can be extended to MCP.
applied parameters,
By applying perturbations to both instances and network parameters, we aim to leverage the smooth- ness assumption (Zhu) 2005) to better identify instances that lie the region uncertainty and thus avoid missing their acquisition. Motivated by this, we propose framework, entitled Bayesian Active Learning by Consistency (BALC) (see Fig. Ic]. that consists of three main steps: 1) we perturb an instance, x, generate 2, 2) we perturb the network parameters, w, to generate w'. and 3) we pass both instances through the perturbed network, generating outputs, p(y xz, w²) and p(y| 2 w') ) RCC respectively. We perform these steps for T stochastic perturbations and generate two matrices of network outputs, G(z), E T We visualize such network outputs in Fig. where T = 3 and C 3.
To leverage G and G', we propose two divergence-based acquisition functions that acquire instances that the network is least robust to. In BAL.CKLD, calculate the DKL. between two C-dimensional Gaussians that are empirically fit to G and G'.
BALCKLD DRI E(()) II (V((())
3

---

Under review as a conference paper at ICLR 2021
Scenario 1
Scenario 2
Across Input Perturbations
Across Input Perturbations
Across Parameter Perturbations
Perturbations
B
A
6
BALCCLLD
BALC
BALCKLDD
BALC
Figure 2: (Scenario 1) network output variations (for three classes, A, B, and C) caused primarily by input perturbations. The rectangle illustrates a potential limitation MCD. Given that these parameter perturbations do not result in network output variations and MCD is dependent upon said perturbations, unlabelled instances can be erroneously deemed uninformative. (Scenario 2) network output variations caused by both input and parameter perturbations. We show that while BALCKLD is likely to acquire instances due to input perturbations, BALCJSD considers both input and parameter perturbations when performing acquisitions.
where y- and = (G- - (1) represent the empirical mean vector and covariance matrix of the network outputs, respectively. BALCKLD is likely to detect output variations due to input perturbations. We support this claim in Fig. 2 by illustrating two scenarios. In scenario 1, network output variations caused solely by input perturbations. In contrast, in scenario 2, network output variations are caused by both input and parameter perturbations. We show that BALCKLD R 1 and 0 in these scenarios, respectively. Since the higher the value of an acquisition function, the more informative an instance is, these scenarios illustrate BALCKLD" 's preference for input perturbations. To detect variations due both input and parameter perturbations, we introduce BALCJsD whose full derivation can be found in Appendix
across parameter perturbations
across input perturbations
BALC1sD = EEr (DkI(G.(z) Il G,())) Dx (K || GG'
(2)
4.2 TRACKED ACQUISITION FUNCTION
Deriving the informativeness of unlabelled instances based solely on the value of the acquisition
function at single epoch can be erroneous. This is partially driven by limitations in the approximation of the version space, which known to hinder performance (Cohn et al.. 1994). To improve this approximation, we propose to track acquisition function over time (e.g.. epochs) before employing to acquire instances. The intuition that by incorporating temporal information, we accumulate hypotheses in the version space and thus obtain more reliable estimate of the relative informativeness of each instance. Acquiring such instances would help reduce the size of the version space at greater rate. For any tracked acquisition function, (tt), the corresponding area under the temporal acquisition function, AUTAF E IR. is calculated as follows:
AUTAF = [- ai()dl R
a(l + At) + ar(1) A
2
where the integral is approximated using the trapezoidal rule, A/ is the time-step (in epochs) between epochs at which ordinary acquisition values are calculated, and the epoch at which the AUTAF calculated and an acquisition of unlabelled instances is performed.
4.3 SELECTIVE ORACLE QUESTIONING
Alongside our consistency-bard based AL framework, we aim to minimize the burden of labelling on an oracle. To do so, learn network that dynamically determines whether to request label from
4

---

Under review as a conference paper at ICLR 2021
oracle generate pseudo-label for each acquired unlabelled instance. We refer to this strategy as selective oracle questioning in active learning (SoQal).
Oracle selection network. In addition the learners outlined in Sec. 3.1 fw and 9é. introduce an oracle selection network, he E IR E [0. parameterized by 0, that maps d-dimensional representations, U, to a scalar, t, as shown in Fig.
Prediction Network
Input ECG Segment
Ground truth class label
Feature Extractor far
A
B
Zero One Loss e Ground-truth oracle label
Oracle Selection
Network hp
DE [010
Figure 3: An instance, is provided as input to the feature extractor, sw- The representation, n. is passed through Pe to generate task predictions, and through hg help determine whether label is requested from oracle. The zero-one loss of 96 acts the ground-truth label for t where t Re 1 indicates a cola---yllssifyy instance that would benefit from an oracle label. Otherwise, the instance pseudo-labelled by taking the argmax of the outputs of the prediction network.
Proxy for misclassifications. Ideally, a network should only be reliant on an oracle when it misclas- sifies instance itself. Quantifying these misclassifications is trivial in the presence of ground-truth labels. However, in the AL framework, we are interested in making decisions on unlabelled instances. Therefore, we need reliable proxy for such misclassification. We propose that this proxy be the output of the oracle selection network, t. For example, low and high values of t can indicate correct and incorrect network predictions, respectively. To learn this behaviour, he needs an appropriate supervisory signal. For this. we choose the zero-one loss, of the prediction network, Iø- as the ground-truth label (see Fig. 3. For mini-batch of size, B, our objective function thus consists of two terms: 1) cross-entropy class prediction loss for the main task, and 2) a binary cross- cross-entropy oracle selection loss, with a weighting coefficient, 3, (described next).
B class prediction
oracle selection loss
log (p(y = cla,, w. o)) Beg log (he (f) [(]) (1 - e4) log (1 (t((xx;)
(4)
where is the target class. During the early stages of training on labelled data, network struggles to classify instances correctly. This means that the ratio zero to one losses will be skewed toward the latter. As training progresses and the network becomes more adept classifying instances, this ratio becomes skewed the opposite direction. Such an imbalance in the ground-truth labels, e, and their subsequent shift send strong supervisory signals to hg. For example, during the early stages of training, the outputs of hi will be high and pulled towards / = 1 (high error) even if the corresponding instance correctly classified. Such behaviour makes it difficult ascertain whether instances have been misclassified, hindering the reliability of t as proxy. To offset the aforementioned class imbalance, we introduce dynamic hyperparameter, function. As training progress, 3 - 8 1, as the ratio of correctly classified (e 0) to misclassified (e = 1) instances within mini-batch changes.
Semo
6a-1" 1 where ó is the Kronecker delta
Decision-making with proxy. We aim exploit for the binary decision of either requesting label from an oracle generating lseudo-label. One way to do so is via a simple threshold at 0.5. However, this threshold may not be appropriate. In designing robust selection strategy, we must account for the distribution of the / values that corresponds to each decision and the separability of such distributions. In Figs. 4a and [46] we illustrate these distributions during the early and late stages of training, colour-coded based on whether the values correspond to correctly-classifee (e or misclassified (e = 1) labelled training instances.
We now outline how the binary decision is made. After each training epoch, we fit the t values in Fig. 4b to two unimodal Gaussian distributions. This generates No ag) and (1 for e = and 1, respectively. We choose to quantify the separability of these two distributions using the Hellinger distance, Dnr E [0, 1], as it allows for straightforward threshold. Low separability expressed DH < S implies that hg has yet generate reliable proxy and thus oracle is
5

---

Under review as a conference paper at ICLR 2021
4.0
0250
25
30
0235
Hellage Dosaee
0200
2,5
0.175
Employ
0.150
1.5
Dy 1
Always Asi Oradle
10
0.02
0.060
0.2
860 022
36
0.8
1.0
25
25
100
125
Sclccion Pundice
(a) Early stage of training
(b) Late stage of training
(c) Hellinger distance
Figure 4: Density of the outputs, t, of go colour-coded based on the zero-one classification error (c) the Hellinger distance, Dz, between the two distributions of during training increases as they become more separable.
during the (a) early and (b) late stages of training.
requested for a label. We note that the value S can altered depending on the relative level of trust one has in the network and oracle. When DH 2 S, we evaluate No and N₁ at the t output for each acquired unlabelled instance and define p(A) as the probability of requesting a label from an oracle. We elucidate the entire active learning algorithm in Appendix E
1.
Du S
P(A)=
0)and 1. N (tlun, 1) > N (t/uo, og 0) and DH 2 s
(5)
0, otherwise
5 EXPERIMENTAL DESIGN
5.1 DATASETS
We conduct experiments in PyTorch (Paszke et al.) 2019) on four publically-available datasets. These datasets consist of cardiac time-series data such as the photoplethysmogran (PPG) and the electrocardiogram (ECG) alongside cardiac arrhythmia labels. We use Dj PhysioNet 2015 PPG. D: PhysioNet 2015 ECG (Clifford et al. 2015) (5 -aa)) D3 PhysioNet 2017 ECG (Clifford
et al. 2017) (4- way) D4 = Cardiology ECG (Hannun et al. 2019) (12-way), and D5 = CIFARTO Krizhevsky et al. 2009) (10-way).
To observe the impact of the availability of labelled training data on the active learning procedure, we take a fraction / (0.1, 0.3, 0.5, 0.7, 0.9) the training dataset and place it into the labelled set. Its complement is placed into the unlabelled set. Details about the data splits, preprocessing steps, and the network architecture can be found in Appendices and
5.2 BASELINES
Acquisition Functions. We compare our novel active learning framework to the state-of-the-art acquisition functions used in conjunction with MCD. These include Var Ratio, Entropy, and BALD, definitions of which can be found in Appendix leamnngg is not employed (No AL).
A
We also compare to the scenario in which active
Selective Oracle Questioning. We experiment with baselines that exhibit varying degrees of oracle dependence. No Oracle a scenario in which 0% of the labels that correspond to unlahelled instances are oracle-based and are instead pseudo-labelled by taking the argmax of the network predictions. Epsilon Greedy is a stochastic strategy [aatkins] (1989) that we adapt to exponentially decay the reliance of the network Response assumes that high entropy predictions generated by a network are indicative instances that the network is unsure of. Therefore, introduce threshold, Sentropy such that if it is exceeded, an oracle is requested to label the chosen instance (see Appendix G). The most dependent baseline is 100% Oracle, raditionally-employe strategy in AL where 100% of the labels oracle-based
oracle function of the number of acquisition epochs. Entropy

---

Under review as a conference paper at ICLR 2021
We do not compare methods to Softmax Response (Geifman & El Yaniv 2017) and SelectiveNet (Geifman El-Yaniy 2019). despite their strong performance for selective classification, as they do not trivially extend to the setting in which labels are unavailable.
5.3 HYPERPARAMETERS
For all experiments, we chose the number of MC samples T 20 to balance between computational complexity and accuracy of the approximation of the version space. We acquire unlabelled instances at pre-defined epochs during training which we refer to as acquisition epochs, T 5m, n N During each acquisition epoch, we acquire b = 2% of the remaining unlabelled instances. We also investigate the effect of such hyperparameters on performance (see Appendices R). When experimenting with tracked acquisition functions, we chose the temporal period, A/ 1. For the CIFARIO experiments, we chose T= T 2n, and 10%.
Selective Oracle Questioning, Recall that we delegate selective oracle questioning to the network only when DH 2 Given DH's increasing trend during training (see Fig, [c). we chose 0.15 to balance between the reliability of the proxy and the independence of the network from oracle. We also explore the sensitivity of SoQal to this choice of S.
EXPERIMENTAL RESULTS
6.1 ACTIVE LEARNING WITHOUT ORACLE
Active learning frameworks typically assume the presence of an oracle (expert annotator). However, oracles always available, particularly in the medical domain where physicians have limited time to provide annotations. To reflect this scenario, we evaluate the ability our AL framework to operate without an oracle. In Fig. [a we illustrate the validation AUC of various methods when exposed to a fraction, 8, of the labelled training data on D2 and D,, respectively.
-5.7
-1.8
-1.3
AUTAF
VerRatinun
Entropyess Entropywe
BALDwco
BALCgo
VarRatbuu
BALD
BALGao
0.4
-1.9
0.80
-0.5
1.3
0.70
-0.1
-1.6
-0.9
0.65
Rarto
Entropy
0.60
(b) Utility of tracked acquisition functions D2
0.55
0.4
0.50
-0.1
-0 00.2
0.2
0.4
-0.2
-0.1
0.45
0
40
AUTAF
0.2
Fpoch
0.4
-0.1
-0.2
3
0.3
0.1
0.2
0.0
(a) D2 at 8 0.3
-0.3
0.1
-0.2
69
-0.2
0.3
-0.1
Entroey
SALD
Var
BALD
-03
(c) Utility of tracked acquisition functions on D3
Figure 5: Validation AUC on (a) D, at = 0.3. Mean percent change in test AUC when comparing MCP with static and tracked acquisition functions to MCD with their static counterparts (b) D1 and (c) Ds. We show results for Var Ratio, Entropy, and BALD, at all fractions, [0.1, 0.3, 0.5, 0.7, 0.9 and across five seeds.
7

---

Under review as a conference paper at ICLR 2021
We find that BALCKLD achieves strong generalization performance and does so in a fewer number of epochs relative to the remaining methods. For example, in Fig. 5al BALCKLD achieves an AUC ss 0.69 after only 20 epochs whereas BALD MCD does at epoch 40. This implies that BALCKLD can result in a two-fold reduction in training time. It also achieves a higher final AUC R 0.72 relative to the remaining methods. We hypothesize that such behaviour is due to BAI.CKLD'S 's improved ability to estimate the region of uncertainty, and thus acquires more informative instances. Moreover, given the absence of an oracle, these informative instances are likely to have also been pseudo-labelled correctly. The acquisition of more informative instances which are labelled correctly by the network suggests that such instances are closer to the decision boundary than their non non-acquired counterparts yet are still on the correct side of the boundary. We arrive at similar conclusions for the remaining experiments (see Appendix and D.
Having illustrated the potential benefit of static acquisition functions, we now move on to quantify the effect of incorporating temporal information on generalization performance. In Figs. (5g..55jjs and [5c] we illustrate the percent change in the AUC when using MCP, with and without tracked acquisition functions, relative to MCD. We find that tracked acquisition functions are most useful when the initial size of the labelled dataset is small (L 3 values) (red rectangle). For example, incorporating temporal information into BALD at 3 = 0.1 improves the generalization performance by 11% We hypothesize that this improvement is due to the increased enumeration of hypotheses over time, which in turn, results in a more reliable approximation of the version space.
6.2 ACTIVE LEARNING WITH NOISE-FREE ORACLE
In this section, we relax the assumption that physicians are unavailable for annotation purposes. Instead, we focus on alleviating the labelling burden that is placed on physicians when they are available. More specifically, we assume that oracles can provide accurate labels, i.e., noise-free. In Table I
we illustrate the test-set AUC of the oracle questioning strategies on all datasets at ( = 0.1.
We find that SoQal consistently outperforms its counterparts across D2 Dz. For example, while using BALDMCD on D2, SoQal achieves an AUC = 0.707 whereas Epsilon Greedy and Entropy Response achieve AUC = 0.609 and 0.584, respectively. Such finding suggests that SoQal is better equipped to know when and for which instance a label should be requested from an oracle. One could argue that SoQal's 's superiority is due to its high dependence on the oracle. In fact, show that this is not the case in Appendix D On the other hand, we observe that SoQal performs on par with the other methods on D4 and D;- We hypothesize that this outcome is due to the cold-start problem (Konyushkova et al. 2017) where AL algorithms fail to learn due to the limited availability of labelled training data. We support this claim with experiments in Appendix M Moreover, remind readers that by increasing the value of S in the SoQal experiments, networks can cede more control the oracle and thus further improve performance, an effect we quantify in Appendix N
Table 1: Mean test AUC of oracle questioning strategies in the presence of a noise- free oracle. Results are shown for a subset of the acquisition functions on Dj- - D3 and are averaged across five seeds.
Oracle Questioning Method
Dataset
Ac. Function
No Oracle
Entropy Response
Epsilon Greedy
SoQal (ours)
100% Oracle
No AL
BALDMCD
0.465 0.017
0.496
0.039
0.491 1 0.028
0.621 1 0.021
0.653 0.013
BALDMCP
0.464 1 0.023
0.517
0.043
0.501 1 0.043
0.645
1 0.015
0.676
0.020
0.577 0.577=00011
BALCxLD
0.500 0.023
0.034
0.548
0.042
0.598
0.055
0.030
Temporal BALCKE
0.496 1 0.024
0.536
: 0.040
0.521
1 0.059
0.646
1 0.067
0.659 1: 0.033
BALDDAC
0.573 0083
0.041
0.609
0.107
0.033
D2
BALDMae
0,589 1 0.045
0.638 1 0.043
0.637 1: 0.044
0.677 1 0.042
0.735 +: 0.028
0.679
BALCKLD
0.602 2 0.044
0.582 0.017
0.643 0.033
0.677 0.024
0.722 0.018
Temporal BALCKLD
0.575 + 0.017
0.612 = 0.050
0.605 + 0.019
0.648 + 0.057
0.735 + 0.011
BALUUICD
0.581 + 0.014
0.583 0.013
0.673 + 0.015
0.721 + 0.023
0.812 0.008
D2
BALCKL.
BALDMeP
0.631 : 0.010
0.623 1 0.020
0.676 0.058
0.629 0.004
0.643 1 0.041
0.665 1. 0.028
0.731 1 0.033
0.720 0.044
0.787 0.008
0.798 0.007
0.716 0.012
Temporal BALCKLD
0.600 = 0.005
0.630 0.014
0.654
0.730 = 0.024
0.794 0.002
BALDscDD
0.486 : 0.011
0.489 0.030
0.474 1: 0.037
0.468 1: 0.021
0.585 0.011
1D4
BAILCKTD
BALDMCH
0.505 + 0.032
0.493 + 0.030
0.504 0.026
0.504 0.039
0.492 0492+00022
0.473 + 0.010
0.499 + 0.029
0.495 + 0.012
0.605 0.024
0.588 0.033
0.486 + 0.023
Temporal BALCKLD
0.030
0.023
0.496
0,503
0.027
BALD
0.891 1: 0.003
0.893 - 0.002
0.901
1: 0.001
0.899 1 0.003
0.902 1 0.003
D6
BALDDcc
BALCKLD
0,895 + 0.002
0.894 + 0.003
0.895 = 0.001
0.003
0,904 0.003
0.899 + 0.002
0,897
0.895 + 0.001
0.899 + 0.002
0.900 0.002
0.898 0.002
Temporal BALCCLD
0.895 0.001
0.944 0.002
0,899 1 0.002
0.896 0.002
0.899 0.004
8

---

Under review as a conference paper at ICLR 2021
6.3 ACTIVE LEARNING WITH NOISY ORACLE
So far, we have presented scenarios in which oracles are either absent or present with the ability to provide accurate labels. In healthcare, however, physicians may be ill-trained, fatigued, unable to diagnose a case due to its difficulty. We simulate this scenario by introducing two types of label noise. We stochastically flip each label 1) any other label randomly (Random), 2) its nearest neighbour from a different class in a compressed subspace (Nearest Neighbour). Whereas the first form of noise is extreme, the latter is more realistic as it may represent uncertain physician diagnoses. To simulate various magnitudes of noise, we chose the probability of introducing noise, = [0.05, 0.1, 0.2, 0.4, 0.8]. Fig. 6 we illustrate the effect label noise on the test AUC for the various oracle questioning strategies.
SoQai
Epsilon Greacy
Entropy Response
0.650
0.625
AUG
0.600
0.580
0.525
0.500
0.755
0.1
No Noise
Random Noise
Nearest Neighbour Noise
Figure 6: Average test AUC for the oracle questioning strategies in the absence and presence, of various magnitudes, of label noise on Dj using BALDMCP. We show that with up to 80% random or nearest neighbour label noise, SoQal still outperforms the remaining methods when trained without label noise. This illustrates that SoQal is better equipped to deal with oracle label noise.
We find that SoQal outperforms the remaining strategies regardless of noise type and magnitude (except at 40% random noise). For example, at 5% random noise, SoQal achieves an AUC so 0.66 whereas Epsilon Greedy and Entropy Response achieve AUC 0.56 and R 0.53, respectively. Surprisingly, we find that the introduction of label noise can sometimes improve performance. For example, SoQal's AUC 0.64 - 0.66 with no noise and 5% random noise, respectively. We hypothesize that this is due to inherent label noise in the datasets. By introducing further noise, we nudge these labels towards their ground-trute values. Moreover, SoQal is better able to deal with label noise than its counterparts. Specifically, SoQal at 80% random noise achieves AUC x 0.53 whereas Epsilon Greedy and Entropy Response trained without noise achieve AUC Rs 0.50 and 0.52, respectively. This effect, which is even more pronounced when dealing with nearest neighbour noise, indicates the utility of SoQal in the presence of a noisy oracle. We arrive at similar conclusions when experimenting with other datasets and acquisition functions (sce Appendix 0
7 DISCUSSION AND FUTURE WORK
In this paper, we proposed a novel consistency-bases active learning framework which perturbs both inputs and network parameters and acquires instances to which the network is least robust. We illustrate the utility of this approach in the absence of an oracle. Moreover, we propose strategy that dynamically determines whether an oracle should be requested for label. We empirically show that this approach is better able to deal with noisy oracles than the baseline methods. We now elucidate several future avenues worth exploring.
Incorporating Prior Information. The default mode for SoQal is deferral to oracle. However, relevant a priori information, such as the degree of noise inherent in the oracle's labels, can be incorporated to alter either the default mode or the Hellinger threshold, S.
Incorporating Multiple Oracles. this work, we focused on the presence of a single oracle. Scenarios in which multiple oracles exist may better reflect clinical environments which include multiple experts of various levels of competency. Therefore, dynamically querying these oracles might be of interest.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Zachi Attia, Suraj Kapa, Francisco Lopez-Jimenez, Paul M McKie, Dorothy J Ladewig, Gaurav Satam, Patricia A Pellikka, Maurice Enriquez- Sarano, Peter A Noseworthy, Thomas M Mun- ger, et al. Screening for cardiac contractile dysfunction using an artificial intelligence-enable electrocardiogram. Nature Medicine, 25(1):70-74, 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709 2020.
C Chow. On optimum recognition error and reject tradeoff. IEEE Transactions on Information Theory, 16(1):41-46, 1970.
Gari Clifford, Ikaro Silva, Benjamin Moody, Qiao Li, Danesh Kella, Abdullah Shahin, Tristan Kooistra, Diane Perry, and Roger G Mark. The physionet/computing in cardiology challenge 2015: reducing false arrhythmia alarms in the icu. In 2015 Computing in Cardiology Conference, pp. 273-276, 2015.
Gari D Clifford, Chengyu Liu, Benjamin Moody, H Lehman Li-wei, Ikaro Silva, Qiao Li, AE John- son, and Roger G Mark. Af classification from short single lead ECG recording: the physionet/computngg in cardiology challenge 2017. 2017 Computing in Cardiology, pp. 1-4, 2017.
David Cohn, Les Atlas, and Richard Ladner. Improving generalization with active learning. Machine Learning, 15(2):201-221, 1994.
Corinna Cortes, Giulia DeSalvo, and Mehryar Mohri. Learning with rejection. In International Conference on Algorithmic Learning Theory, pp. 67-82. Springer, 2016.
Ofer Dekel, Claudio Gentile, and Karthik Sridharan. Selective sampling and active learning from single and multiple teachers. Journal of Machine Learning Research, 13(Sep):2655-25) 2297,,2667--55 2012.
Ran El-vaniv and Yair Wiener. On the foundations of noise-free selective classification. Journal of Machine Learning Research, 16M5,))1600--641, 2010.
Yarin Gal and Zoubin Ghahramani. Dropout as Bayesian approximation: Representing model uncertainty in deep learning. In international Conference on Machine Learning, pp. 1050-1059, 2016.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep Bayesian active learning with image data. In Proceedings of the 34th International Conference on Machine Learning-Vir Volume 70, pp. 1183-1192. JMLR. org, 2017.
Mingfei Gao, Zizhao Zhang, Guo Yu, Sercan Arik, Larry Davis, and Tomas Pfister. Consistency- based semi-supervised active learning: towards minimizing labeling cost. arXiv preprint arXiv:1910.07153. 2019.
Yonatan Geifman and Ran El- Yaniv. Selective classification for deep neural networks. In Advances in neural information processing systems, pp. 4878-4887, 20888488811177 2017.
Yonatan Geifman and Ran El-Yaniv Selectivenet: A deep neural network with an integrated reject option. arXiv preprint arXiv:1901.09192 2019.
Wenbo Gong, Sebastian Tschiatschek. Richard Turner, Sebastian Nowozin, and José Miguel Hernández-Lobato. Icebreaker: clement-wise active information acquisition with bayesian deep latent gaussian model. arXiv preprint arXiv: 2019.
Awni Y Hannun, Pranav Rajpurkar, Masoumeh Haghpanahi, Geoffrey Н Tison, Codie Bourn, Mintu P Turakhia, and Andrew Y Ng. Cardiologist- arrhythmia detection and classification in ambulatory electrocardiograms using a deep neural network. Nature Medicine, 25(1):65, 2019.
Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning. arXiv preprint arXiv: / 1/2.5745 2011.
10

---

Under review as a conference paper at ICLR 2021
Ksenia Konyushkova, Raphael Sznitman, and Pascal Fua. Learning active learning from data. In Advances in Neural Information Processing Systems. pp. 4225 4235, 2017.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Ziyin Liu, Zhikang Wang, Paul Pu Liang, Russ Salakhutdinov, Louis-Philippe Morency, and Masahito Ueda. Deep gamblers: Learning to abstain with portfolio theory. In Advances in Neural Information Processing Systems, pp. 10622-10632, 2019.
Andrew Kachites McCallumzy and Kamal Nigamy. Employing and pool-based active learning for text classification. In Proc. International Conference on Machine Learning, pp. 359-367, 1998.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, higg-peerfrmmncc deep learning library. In Advances Neural Information Processing Systems, pp. 8024 8024-8035, 2019.
Ryan Poplin, Avinash V Varadarajan, Katy Blumer, Yun Liu, Michael V McConnell, Greg S Corrado, Lily Peng, and Dale R Webster. Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning. Nature Biomedical Engineering, 2(3):158, 2018.
Burr Settles. Active learning literature survey. Technical report, University of Wisconsin Madiioo,, Department of Computer Sciences, 2009.
Samarth Sinha, Sayna Ebrahimi, and Trevor Darrell. Variational adversarial active learning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 5972-5981, 2019.
Asim Smailagic, Pedro Costa, Hae Young Noh, Devesh Walawalkar, Kartik Khandelwal, Adrian Galdran, Mostafa Mirshekari, Jonathon Fagert, Susu Xu, Pei Zhang, al. Medal: Accurate and robust deep active learning for medical image analysis. In IEEE International Conference on Machine Learning and Applications, pp. 481-488 2018.
Asim Smailagic, Pedro Costa, Alex Gaudio, Kartik Khandelwal, Mostafa Mirshekari, Jonathon Fagert, Devesh Walawalkar, Susu Xu, Adrian Galdran, Zhang, al. O-medal: Online active deep learning for medical image analysis. arXiv preprint arXiv: 10508, 2019.
Nenad Tomašev, Xavier Glorot, Jack W Rae, Michal Zielinski, Harry Askham, Andre Saraiva, Anne Mottram, Clemens Meyer, Suman Ravuri, Ivan Protsyuk, et al. A clinically applicable approach to continuous prediction of future acute kidney injury. Nature, 572(7767): 116-119,, 2019.
Ruth Urner, Shai Ben David, and Ohad Shamir. Learning from weak teachers. In Artificial intelligence and statistics, pp. 1252-1260, 2012.
Vikas Verma, Alex Lamb, Juho Kannala, Yoshua Bengio, and David Lopez-Paz. Interpolation consistency training for semi-supervised learning. arXiv preprint arXiv:/9033 03885,, 2019.
Guijin Wang. Chenshuang Zhang. Yongpan Liu, Huazhong Yang, Dapeng Fu, Haiqing Wang, and Ping Zhang. A global and updatable ecg beat classification system based recurrent neural networks and active learning. Information Sciences, 501:523 542, 2019.
Christopher John Cornish Hellaby Watkins. Learning from delayed rewards. 1989.
Yair Wiener and Ran El-Yaniv. Agnostic selective classification. In Advances Neural Information Processing Systems, pp. 1665-1673, 2011.
Qizhe Xie, Zihang Dai, Eduard Hovy, Minh- Kinh-Thang Luong, and Quoc Lc. Unsupervised data augmentation for consisteney training. 2019.
Songbai Yan, Kamalika Chaudhuri, and Tara Javidi. Active learning from imperfect labelers. In Advances in Neural Information Processing Systems, pp. 2128-2136, 2016.
Chicheng Zhang and Kamalika Chaudhuri. Active learning from weak and strong labelers. In Advances in Neural Information Processing Systems, pp. 703-711, 2015,
Xiaojin Jeny Zhu. Semi-supervised learning literature survey. Technical report, University of Wisconsin-Maddisn, Department of Computer Sciences, 2005.
11


Reviews:
[{"review_id": "dfAG5aWOgOZ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an active learning framework which relies on a consistency-based acquisition function and a selective querying method. The idea is interesting and the paper is well written; however, I have the following comments and questions:\n\nhe motivation and intuition behind the proposed acquisition functions should be in the main text instead of appendix as this is one of the main contributions of the work (at least partially). Also Figure 6 in the Appendix does not really provide an intuition; it’s basically showing the description in the text by the figure. The terms NSR, AFib and LBBB are introduced in the figure without any definition in the text (I believe these are your e cardiac arrhythmia labels but the reader not familiar with these terms may have trouble figuring it out.). For a more intuitive figure, I suggest using a synthetic toy example. \n\nI believe the ‘selective query questioning’ module is quite similar to the approach proposed by Yoo and Kweon in ‘Learning Loss for Active Learning’ in CVPR 2019. The main difference is that the loss prediction module in the latter is more general and can support tasks other than classification. Can you elaborate on the advantages of your approach over theirs?\n\nCan you also provide the same analysis that you’ve done for D2 and D1 in Fig 4(a, b) for other datasets and \\beta values? I couldn’t find it in the appendix.  The results in the Appendix (I) do not provide a coherent story which makes it hard to judge the pros and cons of the method. Also it would be nice if you add some more standard datasets such as CIFAR-10 as it helps comparing the method with many other active learning baselines in the literature. \n\nIt’s a bit unintuitive to think about examples that the acquisition function is considering informative but the ‘selective oracle questioning’ is preferring pseudo labels for them over oracle labels. In other words, why should these two modules disagree on an instance? If we are highly uncertain about the label of an instance under what scenarios the pseudo label for that instance helps the model? What’s the advantage of your approach over using an approach which only has an acquisition function based on loss prediction (e.g. ‘Learning Loss for Active Learning’ )? \n\nExperiments in 6.2: Do you observe the same behavior for other beta thresholds? \n\nOverall, I think the paper is generally well written and the idea is promising but I believe the strengths and weaknesses of the method need to be discussed in more details. It’s totally fine that the model fails on some datasets and shows significant improvement over baselines in some other datasets; however, in the current version of the paper the reader is not able to understand what are the success and failure chances of the method. The experiments in the appendix fail to provide a coherent story or a discussion on why the method is not performing well in some experiments (e.g. ~40% of the experiments in test set performance without an oracle).\n\n-----------------------------------------------\nPost-rebuttal comments:\nI'd like to thank the authors for adding the experiments; the paper looks stronger now but unfortunately, the results on the new experiments are not that encouraging. Considering the results and also other reviews; I'd like to keep my score as marginally below acceptance threshold. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "_bOGNPR19_T", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors proposed a consistent-based active learning framework to annotate largely unlabeled physiological signals with the help of human annotators (oracles). The paper is well organized and easy to follow. It is somewhat novel to equipping active learning with consistency learning and selective classification. The experiments (along with the Appendix) give a comprehensive analysis of the method. \n\nHowever, I do have the following concerns: \n\nFirst, there might be a simple baseline to deal with unavailable labels: to augment all unavailable data with pseudo labels (such as predictions of the current model, or assign with nearest neighbor’s label), and train on [oracle labeled data] + [pseudo labeled data] together. How is this method compared with yours? \n\nSecond, both title and abstract emphasize physiological signals. But the proposed method is not related to physiological signals. And it seems no special design for physiological signals. Only all four experimental datasets are physiological signals - ECG and PPG. However, if it is targeting physiological signals, others like blood pressure (BP), heart rate (HR), and electroencephalogram (EEG) are also important physiological signals. It is necessary to see how this method performs on those data. \n\nThird, datasets have different sampling rates, but this paper used the same model architecture (in Appendix H.1). And such architecture is not a first-choice model for any of the four datasets. For D1, D2, D3, one can find many other delicate designed models from the challenge papers. For D4, Hannun et al., 2019 also used a much deeper neural network. Thus, in Table 1, we can see that even 100% Oracle is much lower than reported numbers in other papers. \n\nSome minors from Table 1: For AUC, do you mean area under ROC-curve, precision-recall curve, or the others? If it is ROC-AUC, it is confusing that table 1 AUCs of most methods on D4 are lower than 0.5 - even worse than a random classifier. Besides, BALC seems no obvious positive effect than BALD. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "KO32TrPRb9", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an active learning framework called SoCal that is consistency-based and can decide between whether to make use of the oracle to provide a label or to make use of a pseudo-label generated by the algorithm itself instead. The proposed method hopes to address resource-constrained active learning scenarios where the oracle is not always available or we wish to make use of the oracle as infrequently as possible. Experimental results demonstrate reasonable performance on four publically available physiological datasets.  Experimental results when the oracle is noisy is also reported.\n\nOverall, I  think the paper proposes a reasonable approach to tackle various challenges one might encounter in the resource-constrained active learning setting. I have the following concerns:\n\n1. Perhaps the major concern I have is the novelty of this paper. While the paper proposes a framework that harnesses several elements in design that are helpful to its success, most of these elements are not new to the literature. Perturbation in samples and network parameters are widely used practices in achieving consistency. Deciding to query the oracle or not is also a problem studied in active learning.\n\n2. Related to the first point, because of the lack of obvious novelty in the paper, providing insights and demonstrating why the proposed composition of design elements should work is all the more important. However, I find that the paper is not very insightful. There is a lack of justification and insights in explaining why the proposed method should work. As a remedy, theoretical justification of some sorts and ablation studies will be desirable.\n\nMiscellaneous:\n1. Why the Gaussian assumption made in the paper is reasonable?\n2. Any thoughts on why MCP or MCD alone \"can fail to identify instances that lie in the region of uncertainty\"?\n3. When there is no oracle, or when there is a noisy oracle, is it reasonable to compare the proposed method with methods that deal with noisy labels such as semi-supervised learning?\n4. While I understand that learning from physiological signals can be an important use case for the proposed method, the proposed method also seems to be general enough to be applied to other non-healthcare datasets. Is there any consideration on why or why not running the proposed methods on these datasets?\n\n\n=======Post Discussion==========\n\nAfter reading the author's response and other reviewers' reviews, I still find the novelty of this paper somewhat insufficient. Therefore, I would like to maintain my initial evaluation.\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ep2qhFB9C1q", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes the framework that dynamically determines whether to request a label from an oracle or to generate a\npseudo-label instead. It is claimed to remain strong performance with noisy oracle. In this work, the unlabelled instances in proximity to the decision boundary are considered more informative. After performing Monte Carlo Perturbations on unlabeled data and feeding into networks, the samples with significant different outputs are considered close to the boundary.  This paper proposes to perturb both instances and parameters as shown in Equation (1)(2).\n\nImportant technical details are missing. For example, based on the two Gaussian distributions of perturbed networks and the Hellinger distance of the distributions to determine the probability of requesting a label from an oracle as in Equation (5). It is not mentioned how to generate the pseudo-label. The connection of the probability in Equation (5) and previous sections are not close. \n\nMy main concern is that the novel contribution of this work is not strong. Though this work proposed to consider both the instance perturbation, parameters perturbation, temporal information and so on. They are not strongly motivated and no related references to support the motivations. There are also no related empirical experiments to support the motivations.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers still have several concerns about the paper after the author feedback stage: the novelty of the paper is not sufficient; the experimental results are not very encouraging. We encourage the authors fixing these issues in the next revision.
Decision: Reject