Learning in Confusion: Batch Active Learning with Noisy Oracle
Authors: Gaurav Gupta
Anit Kumar Sahu
Wan-Yi Lin
Venue:  (2020)
Keywords: Active Learning
Noisy Oracle
Model Uncertainty
Image classification

Abstract: We study the problem of training machine learning models incrementally using active learning with access to imperfect or noisy oracles. We specifically consider the setting of batch active learning, in which multiple samples are selected as opposed to a single sample as in classical settings so as to reduce the training overhead. Our approach bridges between uniform randomness and score based importance sampling of clusters when selecting a batch of new samples. Experiments on
benchmark image classification datasets (MNIST, SVHN, and CIFAR10) shows improvement over existing active learning strategies. We introduce an extra denoising layer to deep networks to make active learning robust to label noises and show significant improvements.


Under review as a conference paper at ICLR 2020
LEARNING IN CONFUSION: BATCH ACTIVE LEARNING WITH NOISY ORACLE
Anonymous authors Paper under double-blind review
ABSTRACT
We study the problem of training machine learning models incrementally with batches of samples annotated with noisy oracles. We select each batch of sam- ples that are important and also diverse via clustering and importance sampling. In particular, compensate poor estimation of the importance scores when the training data is too small to build meaningful model. Experiments tion datasets (MNIST, SVHN, and CIFAR1O) 10) shows improvement over existing active learning strategies. We introduce an extra denoising layer to deep networks to make active learning robust to label noises and show significant improvements.
incorporate model uncertainty into the sampling probability
benchmark image classifica-
INTRODUCTION
Supervised learning is the most widely used machine learning method, but it requires labelled data for training. It is time-consuming and labor-intensive to annotate large dataset for complex super- vised machine learning models. For example, ImageNet (Russakovsky et al., 2015) reported the time taken to annotate one object to be roughly 55 seconds. HHence an active learning approach which selects the most relevant samples for annotation to incrementally train machine learning models is a very attractive avenue, especially for training deep networks for newer problems that have littel annotated data.
Classical active learning appends the training dataset with single sample-label pair at a time. Given the increasing complexity of machine learning models, it is natural to expand active learning pro- cedures to append a batch of samples at each iteration instead of just one. Keeping such training overhead in mind, a few batch active learning procedures have been developed in the literature (Wei eet al. 2015 Sener & Savarese 2018 Sinha et al.) 2019).
When initializing the model with very small seed dataset, active learning suffers from the cold- start problem: at the very beginning of active learning procedures, the model is far from being accurate and hence the inferred output of the model is incorrect/uncertain.. Since active learning relies on output of the current model to select next samples, a poor initial model leads to uncertain estimation of selection criteria and selection of wrong samples. Prior art batch active learning suffers performance degradation due to this cold-start problem.
Most active learning procedures assume the oracle to be perfect, i.e., it can always annotate samples correctly. However. real-world scenarios and given the increasing usage crowd sourcing, for example Amazon Mechanical Turk (AMT), for labelling data, most oracles are noisy. The noise induced by the oracle in many scenarios is resolute. Having multiple annotations on the same sample cannot guarantee noise-free labels due to the presence of systematic bias in the setup and leads to consistent mistakes. To validate this point, we ran a crowd annotation experiment on ESC50 dataset (Piezak 2015): each sample is annotated by 5 crowdworkers on AMT and the majority vote of the 5 annotations is considered the label. It turned out for some classes, 10% of the samples are annotated wrong, even with 5 annotators. Details of the experiment can be found in Appendix A Under such noisy oracle scenarios, classical active learning algorithms such as (Chen et al.) 2013a) under-perform shown in [i Motivating from these observations, fashion batch active learning strategy to be robust to noisy oracles. The main contributions of this work are as follows: (1) propose batch sample selection method based on importance sampling and clustering which caters to drawing a batch which is simultaneously diverse and important to the model; (2) we incorporate model uncertainty into the sampling probability to compensate poor estimation of the

---

Under review as a conference paper at ICLR 2020
K 8 0.1
SCC, -033
Rander
0.35
0.50
0.35
BALD
0.85
0.50
Accrray
0.85
VIN
0.80
0.75
Racoo
0.75
Racor
0.75
0.70
BALD
0.70
BLLD
0,70
Good
Goad
Enereyy
Exerayy
0.61
MMI
0.90
1391
300
630
5D0
10031
0.90
1280
300
400
10001
0.00
1341
300
0.0
500
1001
Number acquired samples
Number acquired samplex
Number acquired samples
Figure 1: Prior active learning methods in MNIST degrades with oracle noise. Noise channel is assumed to be 10-symmetric channel, where is the probability of label error.
importance scores when the training data is too small to build meaningful model; (3) we introduce denoising layer deep networks to robustify active learning noisy oracles. Main results, as shown in Fig. 3 demonstrate that in noise-free scenario, our method performs as the best over the whole active learning procedure, and in noisy scenario, our method outperforms significantly over state-of-the-ar methods.
2 RELATED WORK
Active learning (Tong, 2001) is a well-studied problem and has gain interest in deep learning well. A survey summarizes various existing approaches in (Settles 2009). In nutshell, two key and diverse ways to tackle this problem in the literature are discrimination and representation. The representation line of work focuses on selecting samples that can represent the whole unlabelled training set while the discrimination line of work aims at selecting 'tough' examples from the pool set, for example, using information theoretic scores in (MacKay 1992).
Active Learning:
entropy as uncertainty in (Wang & Shang 2014). Along the lines of ensemble methods we have works, for example, (Beluch et al.) 2018 Freund et al., 1997 Lakshminarayanan al. 2016).
A recent work of discrimination- based active learning (Houlsby & Ghahramani] [2011) uses mutual information, Bayesian Active Learning by Disagreement (BALD), as discriminating criteria. In (Gal iet al. 2017) the authors used dropout approximation to compute the BALD scores for modern Con- volutional Neural Networks (CNNs). However, these approaches do not consider batch acquisition and hence lack of diversity in selected batch samples causing performance lag.
Batch Active Learning: Active learning in the batch acquisition manner has been studied from the perspective of set selection and using submodularity or its variants in a variety of works. The au- thors in (Wei al. 2015) utilize submodularity for naive Bayes and nearest neighbor. The concept of adaptive submodularity is related to active learning as well. The problem solves adaptive greedy optimization with sequential decision making(Golovin & Krause 2011). Using this concept, (Chen) & Krause, 2013) considers pool-based Bayesian active learning with a finite set of candidate hy- potheses. pool-based active learning also discussed in (Ganti & Gray 2011) which considered risk minimization under given hypothesis space. The work in (Wang & Ye 2013) uses both discrim- inative and representative samples to select batch. The authors in (Sener & Savarese 2018) use coreset approach to select representative points of the pool set. Recently, an adversarial learning of variational auto-encoders is used for batch active learning in (Sinha al. 2019). The work make representation of the training and pool, and adversarially select the pool representatives.
The uncertainty for deep learning models, especially CNNs, was first ad- dressed in (Gal& & Ghahramani] 2016 Gal 2016) using dropout as Bayesian approximation. Model uncertainty approximation using Batch Normalization (BN) has been shown in (Teye et al.) 2018). Both of these approaches in some sense exploit the stochastic layers (Dropout, BN) to extract model uncertainty. The importance of model uncertainty is also emphasized the work of (Kendall & Gal 2017]. The work witnesses model as well as label uncertainty which they termed as epistemic and aleatoric uncertainty, respectively. We also address both of these uncertainties in this work.
Model Uncertainty:
Noisy Oracle: The importance of noisy labels from oracle has been realized in the works like (Golovin et al. 2010 Chen al. 2015b Chen & Krause 2013) which utilized the concept of adap-
2

---

Under review as a conference paper at ICLR 2020
tive submodularity for providing theoretical guarantees. (Chen et al. 2017) studies the same prob- lem but with correlated noisy tests. Active learning with noisy oracles is also studied in (Naghshvar et al. 2012 Yan et al.,, 2016). However, these work do not consider deep learning setup. A binary classification task with the noisy oracle is considered in (Du & Ling 2010). The authors in (Khetan et al., 2018) used variation of Expectation Maximization algorithm to estimate the correct labels as well as annotating workers quality.
The closest work to in the noisy oracle setting for deep learning models are (Jindal et al. 2019; 2016). The authors also propose to augment the model with an extra full-connected dense layer. However, the denoising layer does not follow any probability simplex constraint, and they use mod- ified loss function for the noise accountability along with dropout regularization.
PROBLEM FORMULATION
In this section, we introduce the notations used throughout the paper. We then formally define the problem of batch active learning with noisy oracles.
The ith (jth) row (column) of matrix X denoted as X,,, (X. A A is the probability simplex of dimension K, where A = {(11, P2,,,,,........ E R Σ Pr = 1 A P: 2 Vi). For a probability vector E A AK-1 the Shannon entropy is defined as: HD()) >i p: log(p:), and for p.q E AK-1 the Kullback-Leibler (KL) divergence is de- fined as (L(p](q) = E P. log(p:/4:) The KL--divergence is always non-negative and and only if = q. The expectation operator is taken as E. We are concerned with K class clas- sification problem with sample space X' and label space y = (1.2...., K}. The classification model M is taken to be yo X y parameterized with O. The softmax output of the model is given by p = softmax (go (x)) E AK-1 The batch active learning setup starts with set of labeled
Notations:
samples D,, = {(x, }1) and unlabeled samples P {(x,)) }}. With query budget of b, we select a batch of unlabeled samples as, B ALG(D M, b, P), |B| S h, where ALG is the selection procedure conditioned on the current state of active learning (D b, P). ALG is designed with the aim of maximizing the prediction accuracy Epxxy [(h(()) = y)]- Henceforth, these samples which can potentially maximize the prediction accuracy termed as important samples. After each acquisition iteration, the training dataset is updated as Der = Der {(B ys)} where ye are the labels of B from oracle routine.
The oracle takes an input x E and outputs the ground truth label y E V. This is referred to as "deeal Oracle" and the mapping from x to y is deterministic. A "Noisy Oracle' flips the true output y to y' which is what we receive upon querying x. Similar to (Chen et al. 2015a). we assume that the label flipping is independent of the input x and thus can be characterized by the conditional probability ply' = ily = j). where j E y. We also refer this conditional distribution as the noisy-channel, and hence the ideal oracle has noisy channel value of 1 for i j and O otherwise.
For rest of the paper, we use the noise channel as a K -symmettric channel (SC), see Figure 2b which is a generalization of the binary symmetric channel. The K-SC is defined as follows
p(y' = ily = 5) = (1- - for i i=
e/(K- 1) for i {
(1)
where is the probability of label flip, i.e., p(y サ) = We resort the usage of K-SC because of its simplicity, and in addition, it abstracts the oracle noise strength with a single parameter e. Therefore, in noisy active learning, after the selection of required subset B, the training dataset (and then the model) is updated as Drr- Der U {(B v.)). Next, in Section we discuss the proposed solution to noisy batch active learning.
METHOD
4.1
BATCH ACTIVE LEARNING
An ideal batch selection procedure so as to be employed in an active learning setup, must address the following issues, (i) select important samples from the available pool for the current model, and (ii) select a diverse batch to avoid repetitive samples. We note that, at each step, when active learning acquires new samples, both of these issues are addressed by using the currently trained
3

---

Under review as a conference paper at ICLR 2020
model. However, in the event of uncertain model, the quantification of diversity and importance of a batch of samples will also be inaccurate resulting in loss of performance. This is often the case with active learning because we start with less data in hand and consequently an uncertain model. Therefore, we identify the next problem in the active learning as (ii) incorporation of the model uncertainty across active learning iterations.
The construction of batch active learning algorithm by solving the aforementioned first two problems begins with assignment of an importance score (p) to each sample in the pool. Several score functions exist which perform sample wise active learning. To list a few, max-entropy. variation ratios, BALD (Gal et al., (017), entropy of the predicted class probabilities (Wang & Shang 2014). We use BALD as an importance score which quantifies the amount of reduction of uncertainty by incorporating particular sample for the given model. In principle, wish to have high BALD score for a sample to be selected. For the sake of completeness, it is defined as follows.
Batch selection:
I(y; 0|x, Dir)= = (1//x Dar) D. HI x);
(2)
where O are the model parameters. We refer the reader to (Gal et al. 2017) for details regarding the computation of BALD score in (2). To address diversity, we first perform clustering of the pooled samples and then use importance sampling to select cluster centroids. For clustering, the distance metric used the square root of the Jensen-Shannon (JS) divergence between softmax out- put of the samples. Formally, for our case, it is defined as d A A A - [0, 1], where d(p. q) L + q)/2) + KL (p + q)/2))/2. With little abuse of notation, we in- terchangeably use d(p.. Pý) as di, where j.j are the sample indices and Pi are corresponding softmax outputs. The advantage of using JS-divergence is two folds; first captures similarity be- tween probability distributions well, second, unlike KL-divergence it is always bounded between O and 1. The boundedness helps in incorporating uncertainty which we will discuss shortly. Using the distance metric as d we perform Agglomerative hierarchical clustering (Rokach & Maimon| 2005) for a given number of clusters N. cluster centroid is taken as the median score sample of the cluster members. Finally, with all similar samples clustered together, we perform importance sam- pling of the cluster centroids using their importance score, and a random centroid is selected as p(c = k) x Pk- The clustering and importance sampling together not only take care of selecting important samples but also ensure diversity among the selected samples.
Uncertainty Incorporation: The discussion we have so far is crucially dependent on the output of the model in hand, i.c., importance score as well as the similarity distance. As noted in our third identified issue with active learning, of model uncertainty, these estimations suffers from inaccuracy in situations involving less training data or uncertain model. The uncertainty of model, in very general terms, represents the model's confidence of its output. The uncertainty for deep learning models has been approximated in Bayesian settings using dropout in (Gal & Ghahramani 2016). and batch normalization (BN) in (Teye et al. 2018). Both use stochastic layers (dropout, BN) to undergo multiple forward passes and compute the model's confidence in the outputs. For example, confidence could be measured in terms of statistical dispersion of the softmax outputs. In particular, variance of the softmax outputs, variation ratio of the model output decision, etc, are good candi- dates. We denote the model uncertainty as o E [0, 1], such that o is normalized between 0 and 1 with 0 being complete certainty and 1 for fully uncertain model. For rest of the work, we compute the uncertainty measure a as variation ratio of the output of model's multiple stochastic forward passes as mentioned in (Gal & Ghahramani, 2016).
In the event of an uncertain model (a + 1), we randomly select samples from the pool initially. However, as the model moves towards being more accurate (low σr) by acquiring more labeled sam- ples through active learning, the selection of samples should be biased towards importance sampling and clustering. To mathematically model this solution, we use the statistical mechanics approach of deterministic annealing using the Boltzmann- Gibbss distribution (Rose et al. 1990). In Gibbs distri- bution p() x e c i.e., probability of a system being in an äth state is high for low energy 6 states and influenced by the temperature T. For example, if T -> co, then state energy is irrelevant and all states are equally probable, while 7' + 0, then probability of the system being in the lowest energy state is almost surely 1.
We translate this into active learning as follows: For a given cluster centroid c, if the model uncer- tainty is very high (a + 1) then all points in the pool (including c) should be equally probable get selected (or uniform random sampling), and if the model is very certain (a - 0), then the centroid e itself should be selected. This is achieved by using the state energy analogue distance d between
4

---

Under review as a conference paper at ICLR 2020
Algorithm Batch Active Learning
Input:
Initial training data ( pool of unlabeled samples P. model architecture M(0), uncertainty inverse function f(),, batch size b, number of AL iterations T
Output: Selected batches B(t), final model MM(T)
1: for t 2....... do
2: Assign importance score to each x E P as P. [(: y|x, D(
Eq
3: Perform Agglomerative clustering of the pool samples with (b)) number of clusters using
square root of JS iS-divergence as distance metric to get
4: for = 1,2...2..... ... do
5:
Sample cluster centroid from the categorical distribution p(c= k) x Pk
6:
Compute uncertainty estimate ((t-1) of the model M(t-)), and ((0-1)) f(ol(-1)) 7: Sample from the Gibbs distribution p(ς s| , ()- D)
Eq.
8:
(3(*) BBt U {c}
9:
end for
Query oracle for the labels of B(() and update D t D U ((B(), .y)
11: Update model as M( using D(
12: Set P <- p (3())
end for
the cluster centroid and any sample in the pool, and temperature analogue uncertainty estimate a the model. The distance metric d used by us is always bounded between 0 and 1 and it provides nice interpretation for the state energy. Since, in the event of low uncertainty, wish to perform importance sampling of cluster centroids, and we have de.e = (lowest possible value), therefore by Gibbs distribution, cluster centroid is selected almost surely.
To construct a batch, the samples have to be drawn from the pool using Gibbs distribution without replacement. In the event of samples s1,.... Sn already drawn, the probability of drawing sam- ple Ç given the cluster centroid c, distance matrix D [day] and inverse temperature (or inverse uncertainty) 3 is written as
e -dd
Bdd
- -dd,,1mn
S| C, C, Categorical
(3)
Σ e B- s'EP!
dojal
Σ c s'EP!
d
Σ c- B do S'EP!
where p' P1 In theory, the inverse uncertainty B can be any such that f : -
and f(o) + xc as σ + 0 and f(o) = 0 for a = 1. For example, few possible choices for 3 (()) are - log(ar) e//o 1. Different inverse functions will have different growth rate, and the choice of functions is dependent on both the model and the data. Next, since we have drawn the cluster centroid e according to p(c = k) x Pk. the probability of drawing a sample s from the pool P is written as
e BB
p(c 8|8 D)
Σ
Pe
Ee Pe EvER. CB dndd
We can readily that upon setting + in p(c s|sinn, 3, D) reduces to 1/ which is nothing but the uniform random distribution in the leftover pool. On setting B - oc, we have ( = c with probability 0, i.e., selecting cluster centroids from the
with probability pe/ Ee Pet and pool with importance sampling. For all other < 3 < oo we have a soft bridge between these two asymptotic cases. The approach of uncertainty based batch active learning is summarized as Algorithm[[] Next, we discuss the solution to address noisy oracles in the context of active learning.
4.2
NOISY ORACLE
The noisy oracle, defined in Section has non-zero probability for outputting wrong label when queried with an input sample. To make the model aware of possible noise in the dataset originating from the noisy oracle, we append denoising layer the model. The inputs to this denoising layer are the softmax outputs p of the original model. Figure demonstrates the proposed solution for deep learning classification models. The denoising layer is fully-connected K x K dense layer with
5

---

Under review as a conference paper at ICLR 2020
M
ply
ply' =)
plu Aip j)
(x(x)
:
e 1)
x
validate
train
M
(a) denoising layer demonstration
(b) K-SC
Figure 2: Demonstration of appending the denoising layer to the model M for getting M in (a),
and K SCC channel with probability of error in (b).
Algorithm Noisy Oracle Active Learning
Input: Initial training data D( pool of unlabeled samples P, model architecture M batch size number of AL iterations T, active learning Algorithm ALG
Output: Selected batches g((), final model (
for = 1, ...,..., do
2: (z() < ALG(D((-1) (M((1),,
3: Query noisy oracle for the labels of g()) and update D(C D(C ((880,,)))
4: Get M (0) +- M! appended with noisy-channel layer at the end
5: Update noisy model as M (t) using (())
tr
6: Detach required model (t from M (f) by removing the final channel layer
7: Set +- (())
8: end for
weights W = [w.y] such that its output p' Wp. The weights way represent the noisy-channel transition probabilities such that wij ply ily j). Therefore, to be valid noisy-channel, W constrained W {W/ W.. E AAK1 V1 ك K). While training we use the model upto denoising layer and train using p', or label prediction y' while for validation/testing we use model output or label prediction y. The active learning algorithm in the presence of noisy oracle summarized Algorithm[]] We now proceed Section 5 for demonstrating the efficacy of our proposed methods across different datasets.
EXPERIMENTS
5.1
SETUP
We evaluate the algorithms for training CNNs on three datasets pertaining image classification; (i) MNIST (Lecun et al. 1998), (ii) CIFAR10 (Knzhevsky 2009), and (iii) SVHN (Netzer et al., al. 2017). For all the architectures we use Adam (Kingma & Ba 2014) with a learning rate of 1e 3. The implementations are done on PyTorch (Paszke al. (007). and we use the Scikit-learn (Pedregosa al.) 2011) package for Agglomerative clustering.
2011]. We use the CNN architectures from (fcholet 2015
For training the denoising layer, initialize it with the identity matrix Ix. i.c., assuming it to be noiseless. The number of clusters ((6) is taken to be as [5b]. The uncertainty measure o computed the variation ratio of the output prediction across 100 stochastic forward passes, as coined in (Gal & Ghahramani 2016), through the model using validation set which is fixed apriori. The inverse uncertainty function 3 = f(a) ) in Algorithm is chosen from / (et/o - 1). -7 where is scaling constant fixed using cross-validation. The coss-validattin is performed only for

---

Under review as a conference paper at ICLR 2020
noise free (MNIST)
SC, 8 0.1 (MNIST)
K SSC. 8 = 0.3 (MNIST)
1.00
1,00
1.00
Rondon
0.56
0.95
0.95
BALD
0.50
0.90
0.300
Acoraty
0.55
Acorray
Accoraay
0.85
VA.1
0.50
Randon BALD
Randor
0.75
BAID
Covnent
0.700
Comee
0.70
Errecpy
0.79
Emtrsyy
0.65
VAAL
0.65
Proposed
0.65
Proponed
Progosed
Prependt-devoie
01:41
0.00
0.60
1000
Number of acquired samples
Number of acquired samples
Number acquired samples
noise free. (CIFARIO)
(CIFARIO)
SC, E 0.3 (CIFARIO)
0.5
0.8
1..7
1.7
0.7
0.6
Aecuracy
Accuracy
0.5
0.5
0.4
Fandem
0.4
BALD
0.4
SALD
BALD
Corest
Corcast
0,3
Correc
0.3
Eneryy
0.3
Entraay
VMAL
0.2
VMAL
0.2
Proposed
Propoeal denobs
Proposed dancic
0.3
10000
20000
30000
0.1
10000
20000
30000
0.1
10000
20000
30000
Number of acquired samples
Number of acquired samples
Number acquired samples
naise free (SVHN)
K SC, & (SVHN)
K SC, & 0.3 (SVHN)
Accuracy
0.8
Accuracy
0.8
Accuracy
0.8
Randon
Rascim BALD
Randen BALD
Corrsct
Corseet
Coreet
Frenay
F Frtrep
Estragy
0.6
WAAL
0.6
VAAL
VMAL
Propised
Proposal
Proposed
Proposeddeecdee
Propored
0.5
0.5
5000
10000
15000
202000
10000
50.0
10000
15000
70000
25003
5000
10000
10000
20000
75,00
Number of acquired samples
Number of acquired samples
Number acquired samples
Figure 3: Active learning results for various algorithms under different levels of noise strength in the oracle decision (noise free, 0.1 and 0.3) for MNIST, CIFARIO and SVHN Image datasets.
the noise-free setting. and all other results with different noise magnitude & follow this choice. This is done so to verify the robustness of the choice of parameters against different noise magnitudes which might not be known apriori.
5.2 RESULTS
We compare our approach with: (i) Random: A batch is selected by drawing samples from the pool uniform random without replacement. (ii) BALD: Using model uncertainty and the BALD score, the authors in (Gal et al. 2017) do active learning with single sample acquisition. We use the highest b scoring samples to select batch. (iii) Coreset: The authors in (Sener & Savarese, 2018) proposed a coreset based approach to select the representative core centroids of the pool set. We use the 2 OPT approximation greedy algorithm of the paper with similarity measure l norm between the activations of the penultimate layer. (iv) Entropy: The approach of (Wang & Shang 2014) is implemented via selecting b samples with the highest Shannon entropy H(p) of the softmax outputs. (v) VAAL: The variational adversarial active learning of (Sinha et al. 2019).
In all our experiments, start with small number of images 40 50 and retrain the model from scratch after every batch acquisition. In order to make a fair comparison, we provide the same initial point for all active learning algorithms in experiment. We perform a total of 20 random initializations and plot the average performance along with the standard deviation vs number of acquired samples by the algorithms.
7

---

Under review as a conference paper at ICLR 2020
Figure
shows shows that our proposed algorithm outperform all the existing algorithms. As important observation, we note that random selection always works better in the imitial stages of all experi- ments. This observation is explained by the fact that all models suffer from inaccurate predictions at the initial stages. The proposed uncertainty based randomization makes a soft bridge between uni- form random sampling and score based importance sampling of the cluster centroids. The proposed approach uses randomness at the initial stages and then learns to switch to weigh the model based inference scores as the model becomes increasingly certain of its output. Therefore, the proposed algorithm always envelops the performance of all the other approaches across all three datasets of MNIST, CIFARIO, and SVHN.
Figure also shows the negative impact of noisy oracle on the active learning performance across all three datasets. The degradation in the performance worsens with increasing oracle noise strength E. We see that doing denoisification by appending noisy-channel layer helps combating the noisy oracle in Figure[3] The performance of the proposed noisy oracle active learning is significantly better in all the cases. The prediction accuracy gap between algorithm with/without denoising layer elevates with increase in the noise strength E.
The most recent baselines like (VAAL (Sinha et al. 2019)), (Coreset (Sener & Savarese 2018) which make representation of the Training + Pool may not always perform well. While coreset assigns distance between points based on the model output which suffers in the beginning, VAAL uses training data only to make representations together with the remaining pool in GAN like setting. The representative of pool points may not always help, especially if there are difficult points to label and the model can be used to identify them. In addition to the importance score, the model uncertainty is needed to assign a confidence to its judgement which is poor in the beginning and gets strengthened later. The proposed approach works along this direction. Lastly, while robustness against oracle noise is discussed in (Sinha et al., 2019). however, we see that incorporating the denoising later implicitly in the model helps better. The intuitive reason being, having noise in the training data changes the discriminative distribution from p(y|x) to p(y' x). Hence, learning ply' x) from the training data and then recovering p(y) x) makes more sense discussed in Section 4.2
The uncertainty measure a plays a key role for the proposed algorithm. We have ob- served that under strong noise influence from the oracle, the model's performance is com- promised due to spurious training data as we see in Figurc3] This affects the estimation of the uncertainty measure (variation ratio) as well. We see in Figure that the model uncertainty does not drop as expected due to the label noise. However, the aid provided by the denoising layer to combat the oracle noise solves this issue. We observe in Figure4that uncertainty drops at a faster rate as the model along with the denoising layer gets access to more training data. Hence, the proposed algorithm along with the denoising layer make better judgment of soft switch between uniform randomness and importance sampling using (4) The availability of
SC. 5 0.3 (MNIST)
K SC. - 0.5 (5VHNN
0.30
Pro and
0.4
Uncomiiity
0.10
01-
10000
10:00
soqu camppeee
Figure 4: Uncertainty a across active learning ex- periment for K-SC (5 0.3).
better uncertainty estimates for modern deep learning architectures is promising future research, and the current work will also benefit from it.
6 CONCLUSION
In this paper we have proposed a batch sample selection mechanism for active learning with access to noisy oracles. We use mutual information between model parameters and the predicted class probabilities as importance score for each sample, and cluster the pool sample space with Jenson- Shannon distance. We incorporate model uncertainty/coofiddnnee into Gibbs distribution over the clusters and select samples from each cluster with importance sampling. We introduce an additional layer at the output of deep networks to estimate label noise. Experiments on MNIST, SVHN and CIFAR show that the proposed method is more robust against noisy labels compared with the state of the art. Even in noise-free free scenarios, our method still performs the best for all three datasets. Our contributions open avenues for exploring applicability of batch active learning in setups involving imperfect data acquisition schemes either by construction or because of resource constraints.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
W. H. Beluch, Т. Genewein, A. Nurnberger. and J. M. Kohler. The power of ensembles for active learning in image classification. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9368-9377, jun 2018.
Daniel Busby. Hierarchical adaptive experimental design for gaussian process emulators. Reliability Engineering & System Safety, 94(7):1183 - 1193, 2009.
Yuxin Chen and Andreas Krause. Near-optimal batch mode active learning and adaptive submodular optimization. In ICMI,, volume 28, pp. 160-168, 2013.
Yuxin Chen, S. Hamed Hassani, Amin Karbasi, and Andreas Krause. Sequential information max- imization: When is greedy near-optimal? In Proceedings of The 28th Conference on Learning Theory, volume 40, pp. 338-363, 2015a.
Yuxin Chen, Shervin Javdani, Amin Karbasi, J. Andrew Bagnell, Siddhartha Srinivasa, and Andreas Krause. Submodular surrogates for value of information. In Proceedings of the Twenty-Nint AAAI Conference Artificial Intelligence, pp. 3511-3518, 2015b.
Yuxin Chen, Hamed Hassani, Andreas Krause, et Near-optimal bayesian active learning with correlated and noisy tests. Electronic Journal of Statistics, (1(2):4999-5011 2017.
J. Du and C. X. Ling. Active learning with human-like noisy oracle. In 2010 IEEE International Conference on Data Mining, pp. 797-802, Dec 2010.
fchollet. Keras, 2015, BRRLhhtss://gttieit /ttt://ppe//scthu.occcc//ooooeet
Yoav Freund, H. Sebastian Seung. Eli Shamir, and Naftali Tishby. Selective sampling using the query by committee algorithm. Machine Learning, 28(2):133-168, Aug 1997.
Yarin Gal. Uncertainty in Deep Learning. PhD thesis, University of Cambridge, 2016.
Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. In ICML,, volume 48, pp. 1050-1059, 2016.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In ICML, pp. 1183-1192 1192, 2017.
Ravi Ganti and Alexander G. Gray. Upal: Unbiased pool based active learning. In AISTA7S, 2011
Daniel Golovin and Andreas Krause. Adaptive submodularity: Theory and applications in active learning and stochastic optimization. J. Artif. Intell. Res., 42:427-486, 2011.
Daniel Golovin, Andreas Krause, and Debajyoti Ray. Near-optimal bayesian active learning with noisy observations. In NIPS, pp. 766-774, 2010.
Neil Houlsby and Zoubin Ghahramani. Bayesian active learning for classification and preference learning. arxiv preprint arxiv:1 112.5745, 2011.
Ishan Jindal, Matthew S. Nokleby, and Xuewen Chen. Learning deep networks from noisy labels with dropout regularization. 2016 IEEE 16th International Conference Data Mining (ICDM), pp. 967-972, 2016.
Ishan Jindal, Matthew S. Nokleby, and Daniel Pressel. A nonlinear, aoise- quasi-clustering approach learning deep cnns from noisy labels. In CVPR 2019, 2019.
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 5580-5590, 2017.
Ashish Khetan, Zachary C. Lipton, and Anima Anandkumar. Leaming from noisy singly- labeled data. In International Conference on Learning Representations, 2018. URL https:
//openrevi wet./foruse d=H1 sUHgb07)

---

Under review as a conference paper at ICLR 2020
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. abs/1412.6980, 2014.
CoRR,
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Balaji Lakshminarayanan. Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles, 2016.
Yann Lecun, Lon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. In Proceedings of the IEEE, pp.
1998.
David J. C. MacKay. information-basse objective functions for active data selection. Neural Com- putation, 4(4):590-66 604, 1992.
L. Martino, J. Vicent, and G. Camps- Valls. Automatic emulator and optimized look-up table gener- ation for radiative transfer models. In 2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS), pp. 1457-1460, July 2017.
M. Naghshvar, T. Javidi, and K. Chaudhuri. Noisy bayesian active learning. In 2012 50th Anmual Allerton Conference on Communication, Control, and Computing (Allerton), pp. 1626-1633, Oct 2012.
Yuval Netzer, Tiejie Wang. Adam Coates, Alessandro Bissacco, Baolin Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vito,, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS-W, 2017.
Pedregosa, G. Varoquaux, Gramfort, V. Michel, Thirion, O. Grisel, Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg. J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning Python. Journal of Machine Learning Research, 12:2825- 2830, 2011.
Karol J. Piczak. ESC: Dataset for Environmental Sound Classification. In Proceedings of the 23rd Anmual ACM Conference on Multimedia, pp. 1015-1018. ACM Press, 2015. ISBN 978-1-4503- 3459-4. doi: 10. 002114/7733333.2866653
URL http://dl acm. / efm?
foic-2733373 2806390
Lior Rokach and Oded Maimon. Clustering Methods, pp. 321-352. Springer US, Boston, MA, 2005.
Kenneth Rose, Eitan Gurewitz, and Geoffrey C. Fox. Statistical mechanics and phase transitions in clustering. Phys. Rev. Letr., 65:945- 948, Aug 1990.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Fei- Fei. Imagenet large scale visual recognition challenge. International Journal Computer Vi- sion, 115(3):211-253 Dec 2015. ISSN 1573-1405. doi: 1200077/111223-01-0086666 URL https://doi. 1tt.:/1111101100000 015--2666-11--11111
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In International Conference on Learning Representations, 2018. URL https://
openreview. net/ wow.fooduuniaaaaa.ma.
RM
Burr Settles. Active learning literature survey. Computer Sciences Technical Report 1648, Univer- sity of Miscoonin-Maadisn, 2009.
Samarth Sinha, Sayna Ebrahimi, and Trevor Darrell. Variational adversarial active learning, arxiv preprint arxiv:1904.003%7 2019.
Mattias Teye, Hossein Azizpour, and Kevin Smith. Bayesian uncertainty estimation for batch nor- malized deep networks. In ICML, 2018
10

---

Under review as a conference paper at ICLR 2020
Simon Tong. Active Learning: Theory and Applications. PhD thesis, Stanford University, 2001.
D. Wang and Y. Shang. A new active labeling method for deep learning. In 2014 International Joint Conference on Neural Networks (JJCN). pp. 112-119, July 2014.
Zheng Wang and Jieping Ye. Querying discriminative and representative samples for batch mode ac- tive learning. In Proceedings of the 19th ACM SIGKDD International Conference Knowledge Discovery and Data Mining, pp. 158-166, 2013.
Kai Wei, Rishabh Iyer, and Jeff Bilmes. Submodularity in data subset selection and active learning. In ICML, volume 37, pp. 1954-1963, 2015.
Songbai Yan, Kamalika Chaudhuri, and Tara Javidi. Active learning from imperfect labelers. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pp- 2136-21 2144, 2016

---

Under review as a conference paper at ICLR 2020
Cow
Frog
Cat
Crickets
Crying baby
Hellcopter
Chainsaw
Siren
Car horn
Train
Unsure
Cow
0.975
0.000
0.000
0.000
0.00
0.000
0.025
0.0
0.000
0.000
0.000
Frog
0.000
0.825
0.080
0.075
0.00
0.000
0.000
0.0
0.000
0.000
0.080
0.000
0.000
0.925
0.000
0.05
0.000
0.000
0.0
0.000
0.000
0.025
Crickets
Crying baby 0.000
0.000
0.000
0.000
1.00
0.000
0.000
0.0
0.000
0.000
0.000
Helicopter 0.000
0.000
0.000
0.000
0.00
0.875
0.000
0.0
0.000
0.100
0.025
Chainsaw
0.000 0.000
0.000
0,000
0.00
0.000
1,000
0.0
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.00
0.000
0.000
1.0
0.000
0.000
0.000
Car horn
0.000
0.000
Train
0.000
0.000
0.000
0.000
0.00
0.050
0.000
0.0
0.000
0.825
0.025
Figure 5: Annotation confusion matrix of 10 classes of ESC50
A ESC50 CROWD LABELING EXPERIMENT
We selected 10 categories of ESC50 and use Amazon Mechanical Turk for annotation. In each annotation task, the crowd worker is asked to listen to the sound track and pick the class that the sound belongs to, with confidence level. The crowd worker not think the sound track clearly belongs to one of the 10 categories. For quality control, we embed sound tracks that clearly belong to one class (these are called gold standards) into the set of tasks an annotator will do. If the annotator labels the gold standard sound tracks wrong, then labels from this annotator will be discarded.
also pick "Unsure" if he/she does
The confusion table of this crowd labeling experiment is shown in Figure each row corresponds to sound tracks with ground truth class, and the columns majority-vote crowd-sourced labels of the sound tracks. We can see that for some classes, such as frog and helicopter, even with 5 crowd workers, the majority vote of their annotation still cannot fully agree with the ground truth class.
B MORE EXPERIMENTS
We present rest of the experimental results supplementary to the ones presented in the main body of Section
B.1 MNIST
The active learning algorithm performance for oracle noise strength of = 0.2 and E = 0.4 are presented in Figurel6 Similarly what discussed in Section we observe that the performance of proposed algorithm dominates all other existing works for & 0.2. We witnessed that the proposed algorithm performance (without denoising layer) is not able match other algorithms (BALD and Entropy) when e = 0.4, even with more training data. The reason for this behavior can be explained using the uncertainty measure output in the Figure We that under strong noise influence from the oracle, the model uncertainty doesn't reduce along the active learning acquisition iterations. Because of this behavior, the proposed uncertainty based algorithm sticks to put more weightage on uniform random sampling, even with more training data. However, we see that using denoising layer, we have better model uncertainty estimates under the influence of noisy oracle. Since the uncertainty estimates improve, as we see in Figure for 0.4, the proposed algorithm along with the denoising layer performs very well and has significant improvement performance as compared to other approaches.
B.2 CIFARIO
The results for CIFARIO dataset with oracle noise strength of 0.2 and 0.4 are provided in the Figure We see that the proposed algorithm without/with using the denoising layer outperforms other benchmarks.
12

---

Under review as a conference paper at ICLR 2020
K SC, 0.2 (MNIST)
K SC, & 0.4 (MNIST)
1.00
Random
Random
0.95
BALD
0.95
BALD
Coreset
Coreset
0.90
0.90
Entropy
VAAL
Accuracy
0.85
Accuracy
0.85
Proposed
0.80
0.80
0.75
0.75
Entropy
0.70
0.70
VAAL
0.65
Proposed
Proposed-
Proposed denoiss
0.60
0.60
200
400
600
800
1000
200
400
600
800
1000
Number of acquired samples
Number of acquired samples
Figure 6: Active learning results for various algorithms under oracle noise strength & = 0.2, 0.4 for MNIST Image dataset.
K SC, 0.2 (MNIST)
K SC, 3 0.4 (MNIST)
0.30
0.35
Proposed
Proposed
Proposed
0.30
Proposed i noise
0.25
0.25
Uncertainty
0.20
Uncertainty
0.20
0.15
0.15
0.10
0.10
0.05
0.00
200
400
600
800
200
400
600
800
Number of acquired samples
Number of acquired samples
Figure 7: Uncertainty a across active learning experiment for K-SC (s = 0.2, 0.4) on MNIST dataset.
B.3 SVHN
We provide the active learning accuracy results for SVHN dataset with oracle noise strength of = 0.2 and 0.4 in the Figure Similar other results, that the proposed algorithm without/with using the denoising layer outperforms other benchmarks for & = 0.2. For oracle noise strength of = 0.4, uncertainty based batch selection. The reason again found in the uncertainty estimates plot in Fig for doesn improve with active learning samples acquisition. Hence, the proposed algorithm makes the judgment of staying close uniform random sampling. However, unlike MNIST in Figure[7] the uncertainty estimate is not that poor for SVHN, i.e., it still decays. Therefore, the performance loss in proposed algorithm is not that significant. While, upon using the denoising layer, the uncertainty estimates improve significantly, and therefore, the proposed algorithm along with the denoising layer outperforms other approaches by big margin.
similar trend MNIST regarding performance compromise the proposed
0.4. With more mislabeled training examples, the model uncertainty estimate
13

---

Under review as a conference paper at ICLR 2020
K SC, 8 0.2 (CIFAR10)
K SC, & 0.4 (CIFAR10)
0.8
0.7
0.7
0.6
0.6
Accurag
Accuracy
0.5
0.5
0.4
0.4
Random
0.3
Coreset
0.3
Coreset
Entropy
Entropy
0.2
Proposed
0.2
Proposed
Prenosed-denose
Proposed-dencee
0.1
0.1
10000
20000
30000
0
10000
20000
30000
Number of acquired samples
Number of acquired samples
Figure 8: Active learning results for various algorithms under oracle noise strength E 0.2, 0.4 for CIFAR10 Image dataset.
SC, & 0.2 (SVHN)
K SC, 3 0.4 (SVHN)
0.9
0.9
0.8
0.8
Accuracy
Accuracy
0.7
Random
Random
Coreset
Coreset
0.6
Entropy
0.6
Entropy
Proposed
Praposed
Proposed
Proposed + denoise
0.5
0.5
5000
10000
15000
20000
25000
5000
10000
15000
20000
25000
Number of acquired samples
Number of acquired samples
Figure 9: Active learning results for various algorithms under oracle noise strength e 0.2, 0.4 for SVHN Image dataset.
B.4 CIFARI00
Using the same setup explained in Section evaluate the performance CIFAR100 (Kizhevsky 2009) dataset for various active learning algorithms listed in Section We observe Figure II that the proposed uncertainty based algorithm perform similar or better than the base- lines. The incorporation of denoising layer helps in countering the affects of noisy oracle as we demonstrate by varying the noise strength 0.1, 0.3.
C ACTIVE LEARNING RESULTS
For quantitative look at the active learning results, mean and standard deviation of the performance vs. acquisition, in the F we present the results in the tabular format Table for MNIST, Tarle for CIFARIO, Table for SVHN, and Table for CIFAR 100, respectively.
14

---

Under review as a conference paper at ICLR 2020
K SC, 3 = 0.2 (SVHN)
K SC, 3 = 0.4 (SVHN)
0.5
0.5
Proposed
Proposed
Prapased noise
Progosed y nois
0.4
0.4
Uncertainty
Uncertainty
0.3
0.3
0.2
0.2
0.1
0.1
5000
10000
16000
20000
0
5000
10000
15000
20000
Number of acquired samples
Number of acquired samples
Figure 10: Uncertainty across active learning experiment for K-SC (E 0.2, on SVHN dataset.
noise free (CIFAR100)
K SC, E = 0.1 (CIFAR100)
Random
0.4
BALD
0.4
BALD
Coreset
Caresct
Entropy
Entropy
Accuracy
0.3
VAAL
Proposed
Accuracy
0.3
VAAL
0.2
0.2
0.1
0.1
Proposed
Proposed
10000
20000
30000
10000
20000
30000
Number of acquired samples
Number acquired samples
K SC, E = 0.3 (CIFAR100)
Random
0.4
BALD
Coreset
Entropy
Accuracy
0.3
VAAL
0.2
0.1
Proposed
Proposed i
0
10000
20000
30000
Number of acquired samples
Figure 11: Active learning results for various algorithms in noise free setting and under oracle noise strength 0.1, 0.3 for CIFAR Image dataset.
15

---

Under review as a conference paper at ICLR 2020
Table Active learning results for MNIST dataset.
Number of acquired samples
Algorithm
200
400
600
800
1000
noise free
Random
87.19 + 0.84
91.51 + 0.75
93.40 + 0.42
94.39 + 0.57
95.17 + 0.46
BALD
80.03 + 2.47
89.84 + 2.35
94.17 + 0.98
96.23 + 0.72
97.00 + 0.47
Coreset
87.87 + 1.53
92.40 + 1.33
94.53 + 1.00
95.57
96.02
Entropy
77.69 2.34
89.53 2.24
94.94 = 0.69
96.67
0.35
07.28
VAAL
82.96 2.52
89.48 = 1.09
92.03 + 0.73
03.84 0.46
94.96 + 0.34
0.34
Proposed
1.89
94.11
96.0010.31
96.9610.20
97.40-1..15
0.1
Random
81.87 +: 2.42
87.26 + 1.69
89.55 + 0.89
91.10 + 0.61
91.94 + 0.56
BALD
76.13 3.12
85.45 2.84
90.75 2.27
94.01 0.73
94.96 + 1.04
Coreset
83.46 2.49
88.42 2.28
90.80 + 1.85
91.37
93.09 + 1.13
Entropy
74.28 + 3.33
85.64 + 3.15
91.59 + 1.84
94.72
95.73 = 0.60
VAAL
2.447
85.70 + 1.95
88.24 + 1.06
90.56 0.83
91.67 + 0.72
Proposed
83.45+22333
90.74
93.6310.73
95.81 94.9410.68
Proposed
84.88-2.59
92.68:00.97
95.18:0.40
96.2910.34
96.8710.26
+denoise
0.3
Random
1..102..8
77.45 2.03
79.88 + 2.31
81.10 + 1.90
82.66 + 1.55
BALD
69.54 3.188
76.28
3.00
82.25 + 2.09
85.10 + 2.06
87.97 + 1.46
Coreset
74.58 3.07
2.51
79.89 2.77
82417
83.49 2.02
Entropy
68.1113.88
77.70
87.78
89.37 + 1.34
VAAL
3.446
74.17
78.50 2.18
80.24
82.12 + 1.85
Proposed
73.1813.48
81.7711.97
85.85
87.81$1.41
89.46
Proposed
77.81+2.59
85.96+1.46
90.95+1.07
93.3310.66
94.7810.41
+denoise
0.2
Random
77.57 2.74
82.44 + 2.29
85.03 + 2.07
86.41 + 1.50
87.73 1.25
BALD
73.20 3.92
$
+
90.06 + 1.64
91.63 + 1.66
Coreset
79.97 +: 2.96
84.72 + 2.25
86.88 + 2.07
88.84 + 1.89
88.93 + 1.52
Entropy
4.81
82.26 + 2.66
88.44 + 1.76
91.66 + 1.25
93.09 + 1.03
VAAL
15.02
80.81 2.32
83.18 + 1.86
85.67 1.34
87.26 = 1.08
Proposed
79.09+1.76
87.04+1.666
90.46+1.110
92.41+0.90
93.28-00.99
Proposed
80.8812.43
89.3911.27
93.5310.69
95.0310.46
95.98-0.37
+denoise
0.4
Random
66.14 +: 2.93
69.55 + 1.97
72.80 +: 2.30
74.44 +: 1.77
76.19 + 1.66
BALD
66.50 3.07
71.77 2.39
76.81
2.97
79.47 2.66
81.10#2.11
Coreset
68.41 + 4.51
72.75
3.35
75.03 3.17
77.15
77.25 3.08
Entropy
66.31 2.68
73.92 3.00
1.98
82.70
84.25 + 1.81
VAAL
64.09 3.71
08.50
2.338
74.11
75.65
Proposed
66.70+3.23
73.8042.41
76.2842.69
78.2221.64
79.6542.11
Proposed
71.3743.53
80.84+2.76
86.1911.665
90.00+1.24
92.19-10.84
+denoise
16

---

Under review as a conference paper at ICLR 2020
Table Active learning results for CIFARIO dataset.
Number of acquired samples 15000
Algorithm
5000
10000
20000
25000
30000
noise free
Random
58.90 1.77
67.44 1 0.72
71.79 1 0.49
74.28
0.53
75.99
1 0.17
77.58 1: 0.50
BALD
50.42 1.99
65.19 + 1.21
71.58
0.38
75.07 + 0.71
76.90
+ 0.82
78.35
1.01
Coreset
53.62 1.13
63.56 + 1.74
68.65 1.28
71.99
0.55
73.58
+ 0.49
76.10 0.48
Entropy
53.83
4.60
64.89 1 1.27
70.40 1 1.53
73.85
1.25
76.71 1 0.81
78.11 1 0.57
VAAL
57.20 1 0.80
67.66 1 1.37
71.86 1 0.57
74.06
1 0.47
75.87
0.40
77.39 - 0.38
Proposed
1.62 59.28
67.31
0.25
72.92 + 0.57
74.79
0.48
77.09
+ 0.73
78.28
E 0.71
0.1
Random
54.52 + 1.41
61.93 + 1.14
66.37 + 0.59
69.06
0.78
71.12
0.66 0.66
72.97 - 0.70
BALD
46.83 1.76
60.17 + 1.61
66.85 0.97
69.81
+ 0.71
72.42
0.79
74.27 = 0.64
Coreset
50.27 1.98
58,53 1
64.33 1.20
67.89 0.73
70.31
0.62
72.36 0.58
Entropy
48.63 + 2.78
60.05 + 1.70
65.89 = 1.33
70.31 1.28
72,43
1.99 0.99
74.69 = 0.89
VAAT
53.83 1.08
61.75 + 1.11
66.15
0.71
69.32
0.72
70.58
+ 0.62
72.63 0.29
Proposed
53.96 + 1.11
62.10
+ 1.21
66.81 + 0.80
70.30
0.87
72.28
0.76
73.68
+ 0.44
Proposed
52.95 1.54
62.20 + 0.97
67.54 1.15
71.44 0.96
73.94
0.85
76.03 + 0.66
+denoise
0.3
Random
44.82 2.02
52.44 + 0.91
56.58 + 1.52
60.33
62.63 + 0.87
64.82 1.07
BALD
38.70 +: 1.83
49.70 1.64
55.38 1.83
59.76 1.34
62.85
65.10 0.99 0.99
Coreset
41.27 = 1.75
49.46 + 1.32
-4.741115
58.98 + 1.19
11.4440844
64.03 = 0.82
Entropy
39.20 = 2.62
50.23 - 2.35
2.35
56.11 1.49
60.66
- 1.53
63.88
65.31
1.24
VAAL
45.00 1.36
50.41 + 1.52
57.02 0.81
60.03
0.58
61.20
1.79
62.56
1.10
Proposed
44.54 2.02
19.49994
+ 1.57
56.86
+ 1.34
60.52
- 1.38
63.15
+ 0.85
64.84
+ 0.83
Proposed
45.14 1.58
53.72
= 1.55
59.64
1.30
63.35
1.63
1.21 66.44
68.80 = 1.11
+denoise
0.2
Random
19.46 1 1.63
57.21 1 1.04
61.64 1 0.88
64.60 1.14
67.31
1 0.96
68.77 1 0.71
BALD
12.79 1 1.81
54.98 1 1.31
60.81 1 0.96
65.23 1: 0.93
67.97
0.72
69.78 1 0.69
Coreset
45.89 2.26
53.70 1.70
39.00
63.23
66.23
$ 0.62
68.44 =444000 0.88
Entropy
44.37 3.23
55.50 + 2.06
60.70 1.80
64.75
1.58
68.10 1.36
69.72 = 0.96
Proposed
49.38
1.40
57.45
+ 1.29
61.47 + 1.31
65.45
+ 0.92
68.08
+ 1.13
69.37
1 0.60
Proposed
49.03 1.01
58.85
+ 1.32
64.10 + 1.09
68.07
1.48
71.18
+ 0.96
72.65
+ 0.96
+denoise
0.4
Random
40.62 1.81
46.61 + 1.90
51.87 = 18
55.21 + 1.35
57.28 + 1.11
59.79 = 1.32
BALD
34.52 1: 1.59
43.55
49.15 2.09
1.66
53.85
1.82
57.19
1 0.87
59.53
0.98
Coreset
36.52 11.2114 1.83
44.52 1 1.50
49.97 1.58
53.32
1.02
56.86
59.46
0.98
Entropy
35.08 + 2.76
44.21 + 1.82
50.61 = 1.20
54.65 + 1.78
56.66 + 2.16
59.74 = 1.84
Proposed
39.69 = 1.40
46.67 + 1.43
51.47 + 1.38
54.63
= 1.66
58.16
1 0.80
59.87
+ 1.38
Proposed
39.68 1..64
1.56
52.35 + 1.89
56.56
2.89
59.28
+ 1.88
62.12
+ 1.19
+denoise
17

---

Under review as a conference paper at ICLR 2020
Table 3 Active learning results for SVHN dataset.
Number acquired samples
Algorithm
5000
10000
15000
20000
25000
noise free
Random
86.92 + 0.81
90.04 + 0.75
91.46 + 0.40
92.29 + 0.44
92.58 + 0.09
BALD
82.46 0.89
91.18 + 0.42
92.81 + 0.43
94.21 + 0.311 0.31
94.47 + 0.39
Coreset
85.64 = 1.62
90.05 = 0.52
90.76 = 0.30
91.92 0.23
92.43 = 0.31
Entropy
83.28 + 1.11
91.23 = 0.50
93.12 = 0.31
93.88 + 0.12
94.25 + 0.14
VAAL
86.43 + 0.53
89.99 + 0.37
91.35 + 0.28
92.04 + 0.56
92.78 + 0.43
Proposed
86.3010.90
91.8810..35
93.46.1..40
94.05
94.1810.28
0.1
Random
1.134
86.46 = 0.75
88.63 0.74
89.65 0.62
90.05 0.70
BALD
76.26 2.78
87.15 0.82
90.37 + 0.55
91.73 + 0.49
92.54 + 0.52
0.52
Coreset
80.99 + 2.27
86.08 + 0.86
88.08 + 0.69
89.45 0.63
90.19 + 0.76
Entropy
78.83 2.12
00.32
91.67 0.34
92.54 0.45
VAAL
81.99 0.49
68.76
88.35 0.89
89.74
Proposed
81.61+1.300
88.5440.91
90.3740.64
91.6010.59
92.24
Proposed
83.01 +denoise
80.96
91.9810.41
93.1510.37
93.7810.21
0.3
Random
73.75 + 2.33
79.07 + 1.79
82.53 + 1.31
83.74 + 1.49
85.49 + 1.14
BALD
65.72 4.43
79.72 + 1.52
83.38 + 1.72
86.15 + 1.33
1.14400.9
Coreset
71.842..55
13.45:1.0
81.87 + 1.55
83.99 + 1.38
85.78 + 1.28
Entropy
18.52
79.63 = 1.53
83.92 + 1.25
85.98 + 1.48
87.84 + 1.02
VAAL
72.69 1.11
78.17 + 1.95
81.51 + 1.84
84.18 1.03
85.77 + 1.22
Proposed
73.0442.60
81.14
83.14#1.62
85.87
87.71
Proposed
78.80
87.19
90.0440.76
91.56
92.26
+denoise
0.2
Random
78.46 + 1.73
83.87 1.11
85.57 + 1.24
87.60 0.82
88.27 + 0.92
BALD
70.82 + 4.09
83.97 + 1.59
87.08 + 1.24
89.42 0.82
90.53 - 0.76
Coreset
76.75 2.32
82.65 + 1.75
85.43 + 1.08
88.23 + 1.03
Entropy
74.00
84.29 = 1.33
87.46 = 0.99
0.89
90.12
Proposed
77.98+2.27
84.82+1.43
87.86+1.38
88.86+1.10
90.33
Proposed
81.0911.75
88.471 1.39
91.1910.55
92.4710.47
93.1610.31
+denoise
0.4
Random
68.23 +: 2.01
73.07 1 2.47
77.46 +: 2.27
79.79 + 2.37
81.52 + 1.93
BALD
58.76 1 4.05
73.83 2.25
78.98 2.12
81.78 1 1.65
83.59 1 1.68
Coreset
65.26:3.00
73.23 2.51
77.18 + 1.80
80.44 1.40
81.78 + 1.87
Entropy
01.36.4.331
1.76
78.61 2.21
81.40
83.88 + 0.59
Proposed
67.2313.37
74.0812.73
78.1542.115
80.4811.56
82.78
Proposed
76.0742.58
83.8941.00
88.4640..66
90.531:.551
91.58100.22
+denoise
18

---

Under review as a conference paper at ICLR 2020
Table Active learning results for CIFARIO0 100 dataset.
Number acquired samples
Algorithm
8000
16000
24000
32000
noise free
Random
21.40 + 0.79
29.02 + 1.04
34.14 + 1,00
38.46 + 1.00
BALD
20.08 + 0.74
27.25 1.01
33.30 + 1.42
38.10 0.90
Coreset
19.85 = 0.86
27.61 + 0.50
33.32 + 0.78
38.13 + 0.79
Entropy
20.65 + 0.80
28.62 + 1.14
33.95 + 1.17
38.54 + 1.56
VAAL
19.64
0.62
27.28 0.63
33.50 E 0.55
38.18 0.72
Proposed
21.68 + 0.86
28.82 + 0.87
34.16 + 0.88
38.27 + 0.69
0.1
Random
19.18 - 0.68
25.54 + 0.96
30.16 + 1.03
33.28 + 0.96
BALD
17.48 + 0.93
24.17 + 0.94
28.89 + 0.91
32.76 0.66
Coreset
17.04 + 1.02
24.14:0.90
28.96 + 0.83
33.30 0.58
Entropy
18.07 = 0.79
24.33 1.24
29.15 + 1.04
33.76 + 1.39
VAAL
19.11 = 0.82
25.55 0.82
28.82 + 0.48
33.80 1.06
Proposed
18.67 + 0.88
25.41 + 0.72
29.70 + 0.62
33.84 = 1.17
Proposed
19.30 + 0.79
26.17 1.02
30.84 1.03
34.51 1.11
+denoise
0.3
Random
14.50 + 0.52
18.99 0.77
22.84 0.92
25.94 0.99
BALD
12.67 + 1.15
17.62 + 0.44
22.02 + 0.73
25.34 0.83
Coreset
13.34 + 1.01
18.17 0.75
22.26 + 1.03
25.32:0.77
Entropy
14.02 + 0.82
17.63 + 1.00
21.71 + 1.14
25.25 - 1.36
VAAL
14.76 = 0.72
18.79 0.42
23.23 0.69
24.82 = 0.50
Proposed
14.81 + 0.48
19.19 0.90
22.78 + 1.09
26.01 0.85
Proposed
15.35 + 0.51
21.60 0.80
25.56 + 0.76
28.20 + 1.00
+denoise
19


Reviews:
[{"review_id": "rJlOwYVRtB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The topic handled in this paper is very important (hot topic), in my opinion. The authors tackled is the problem of training machine learning models incrementally using active learning with the oracle is noisy. Multiple samples are selected instead of a unique  sample as in the classical framework. The paper seems technically sound.\n I have some suggestions for improving the quality of the paper. See below.\n\n- Improve the captions of Figures 1 and 2 (more explanation, more clarity).\n\n- Use bigger parentheses in Eq. (3).\n\n- In other to increase the impact of your work, consider in your introduction (or in the \"related works\" Section) this kind of approaches that are also active learning algorithms:\n\nD. Busby, “Hierarchical adaptive experimental design for Gaussian process emulators,” Reliability Engineering and System Safety, vol. 94, pp. 1183–1193, 2009.\n\nL. Martino, J. Vicent, G. Camps-Valls, \"Automatic Emulator and Optimized Look-up Table Generation for Radiative Transfer Models\", IEEE International Geoscience and Remote Sensing Symposium (IGARSS), 2017\n\nThis discussion can increase the number of interested readers.\n\n- Upload the final version of your work in Research Gate and ArXiv (to increase the impact of your work).", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "Skgso79atr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper provides a solution for batch active learning with noisy oracles in deep neural networks. Their algorithm suffers less from the well-known cold-start issue in active learning. They also improve the robustness by adding an extra denoising layer to the network. \n\nThe main concern is that the two contributions are rather orthogonal to each other and each of them is not that significant. \nThe first contribution, which alleviates the cold-start problem, is not very surprising, since it is a soft version of previous method BALD. \nThe second contribution, a de-noising layer, is relatively orthogonal to batch active learning. \n\nIn the experiments, the authors compared Proposed +noise with Proposed, Random, BALD, Coreset, and Entropy, but I think the only fair comparison here is between Proposed+noise and Proposed. \n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1lbEu1e5r", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The paper proposes an uncertainty-based method for batch-mode active learning with/without noisy oracles which uses importance sampling scores of clusters as the querying strategy. Authors evaluate their method on MNIST, CIFAR10, and SVHN against approaches such as Core-set, BALD, entropy, and random sampling and show superior performance.\n\nPros:\n(+): The paper is well-written and well-motivated.\n(+): The problem is timely and has direct real world applications.\n(+): Applying the denoising layer is an interesting and viable idea to overcome noise effects.\n \nCons that significantly affected my score and resulted in rejecting the paper are as follows:\n \n1 - Experimental setting and evaluations:\nThe biggest drawback in this paper is the experimental setting which is not rigorous enough for the following reasons:\n\n(a) Weak datasets: Authors have chosen some standard benchmarks but they do not seem to be convincing as the datasets are too easy. Based on my experience, the behavior of an active learning agent trained on small number of classes does not necessarily generalize to cases where the number of classes is large. So I’d like to ask authors to try to evaluate on datasets with more number of classes as well as more realistic images (as opposed to thumbnail images). \n(b) Comparison to state of the art: More importantly, authors are missing out on an important baseline which is a recent ICCV paper [1] on task-agnostic pool-based batch-mode active learning that has explored both noisy and perfect oracles and to the best of my knowledge is the current state of the art. Authors can extend their experimental setting to the datasets used in [1] including CIFAR100 and ImageNet and provide comparison. The reason that it is important to compare is that the method in [1] is task-agnostic and does not explicitly use uncertainty hence it is interesting to see how this method performs against it. \n(c) More on baselines and related work: In addition to [1], different variation of ensemble methods have been serving as active baselines in this field and I recommend adding one as a baseline. For a recent work in this line you can see this paper from CVPR 2018 [2]. Moreover, the authors seem to be missing on a long-standing line of active learning research known as Query-by-Committee (QBC) began in 1997 [3] in the related work section which should be cited as well. \n(d) Hyper parameter tuning: Last but not least about the experiments is the hyper parameter tuning which is not addressed. It is important to not use the well-known hyper parameters for these benchmarks that have been obtained using validation set from the entire dataset. Authors should explain how they have performed this. \n \n2 - Report sampling time:\nAnother important factor missing in the evaluations is reporting time complexity or wall-clock time that it takes to query samples. Authors should measure this precisely and make sure it is being reported similarly across all the methods. I am asking this because random selection is still an effective baseline in the field and it only takes a few milliseconds. Therefore, the sampling time of a new algorithm should be gauged based on that while performing better than random. Given the multiple steps in this algorithm I am skeptical that the sampling time would be proportional to the gain obtained in accuracy versus labeling ratio over random selection baseline. \n \n3: Section 5.2 is not informative:\n(a) My last major concern is section 5.2 where the discussion on results is given along with supporting figures. \nLack of quantitative results: First of all, no quantitative results are given for the values plotted in figure 3 and 4 (neither in the main text nor in the supplement) and different methods happen to be too close to each other, making it hard to see the right color for standard deviations. Also, in the discussion corresponding to those figures no information is provided in this regard. It is important to report how much labeling effort this algorithm is saving by comparing number of samples needed by each method to achieve the same accuracy because that is the main goal in AL. Lack of numbers also makes it hard for this work to be used by others.\n(b) Figure legends: The way authors have labeled their method in Figure 3 is confusing as the “Proposed+noise” happens to achieve better performance over “Proposed”. I think by “noise” authors meant denoising layer was being used (please correct me if I am wrong) but this is not what the legends imply. \n(c) X axis label: It is common to report accuracy versus percentage of labeled data making it more understandable of how far each experiment has been through each dataset. Additionally, I recommend reporting the maximum achievable accuracy for each dataset assuming that all the data was labeled. This serves as an upper bound.\n(d) Font sizes in figures: It will be helpful to make them larger.\n  \n4. I also have a more general concern about uncertainty-based methods. I know that they have been around for a long time but given the fact that predictive uncertainty is still an open problem and there is still no concrete method to measure calibrated confidence scores for outputs of a deep network (Dropout and BN given in this paper have been already outperformed by ensembles (see [4])), hence relying on uncertainty is not the best direction to go. It is literally chicken and egg problem to try to rely on confidence scores of the main-stream task while it is being trained itself. This issue has been raised in this paper but I am still not convinced that the paper has fully addressed it. I think the community needs to explore task-agnostic methods more deeply. [1] is a good start on this path but there is always more to do. This concern is not necessarily a major part of my decision assessment and I only want the authors to state their opinion on this and explain how accurately they think this issue is being addressed.\n \nThe following issues are less major and are given only to help, and not part of my decision assessment:\n\n1- In Figure 3(c), it appears that the accuracy for “Proposed + noise” when \\epsilon=0.1 is higher than when it is noise-free. It might be a miss-reading as the figure is coarse and it is hard to compare but if that is the case, can authors explain it?\n\n2- The Abstract does not read well and does not state the main contribution. It has put too emphasize on batch-mode active learning which has become an intuitive approach since deep networks have become popular. Also the wording “Our approach bridges between uniform randomness and score based importance sampling of clusters” should be changed as all other active learning algorithms are trying to do that. \n\n3 - In section 5.1 please state that you used VGG 16 (I assume so since it is what was used in the cited reference (Gal et al. 2017) but authors need to verify that. Also, the other citation given for this (Fchollet, 2015) is confusing as it is Keras package documentation while in the next sentence authors state that they have implemented their algorithm in PyTorch. So please shed some light on this.\n\n*******************************************************************\nAs a final note, I would be willing to raise my score if authors make the experimental setting stronger (see suggestions above).\n\n[1] Sinha, Samarth, Sayna Ebrahimi, and Trevor Darrell. \"Variational Adversarial Active Learning.\" arXiv preprint arXiv:1904.00370 (2019). \n[2] Beluch, William H., et al. \"The power of ensembles for active learning in image classification.\" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.\n[3] Freund, Yoav, et al. \"Selective sampling using the query by committee algorithm.\" Machine learning 28.2-3 (1997): 133-168.\n[4] Lakshminarayanan, Balaji, Alexander Pritzel, and Charles Blundell. \"Simple and scalable predictive uncertainty estimation using deep ensembles.\" Advances in Neural Information Processing Systems. 2017.\n\n*******************************************************************\n*******************************************************************\n*******************************************************************\nPOST-REBUTTAL:\n\nIn the revised version, there are new tables (Table 1-4) provided in the appendix which I found too different than results reported for previous baselines by more than 6%. For example, according to Core-set paper (Sener, 2018), Figure 4, they achieve near 80% using 40% of the data (20000 samples), and according to VAAL paper (Sinha et al. 2019 github page: https://github.com/sinhasam/vaal/blob/master/plots/plots.ipynb), they achieve 80.90+-0.2. However, the current paper reports 71.99 ± 0.55 for Core-set, and 74.06 ± 0.47 for VAAL which is a large mismatch.\nMore importantly, looking at the results provided in VAAL paper (Sinha et al. 2019 or Core-set paper (Sener, 2018) they show their performance as well as most of their baselines is superior to random selection by a large gap, but in this paper results shown in Table 1 to 4 in almost all of them random is superior (or on-par) to all baselines and the proposed method is the only method that outperforms baseline which is clearly a wrong claim. Therefore, I decrease my score from weak reject to reject.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new active learning algorithm based on clustering and then sampling based on an uncertainty-based metric. This active learning method is not particular to deep learning. The authors also propose a new de-noising layer specific to deep learning to remove noise from possibly noisy labels that are provided. These two proposals are orthogonal to one another and its not clear why they appear in the same paper.

Reviewers were underwhelmed by the novelty of either contribution. With respect to active learning, there is years of work on first performing unsupervised learning (e.g., clustering) and then different forms of active sampling. 

This work lacks sufficient novelty for acceptance at a top tier venue. Reject
Decision: Reject