LA-BALD: An Information-Theoretic Image Labeling Task Sampler
Authors: Yuan-Hong Liao
Sanja Fidler
Venue: Submitted to ICLR 2023 (2023)
Keywords: active learning
data annotation

Abstract: Large-scale visual recognition datasets with high-quality labels enable many computer vision applications, but also come with enormous annotation costs, especially since multiple annotators are typically queried per image to obtain a more reliable label. Recent work in label aggregation consolidates human annotations by combining them with the predictions of an online-learned predictive model. In this work, we devise an image labeling task sampler that actively selects image-worker pairs to efficiently reduce the noise in the human annotations and improve the predictive model at the same time. We propose an information-theoretic task sampler, Label Aggregation BALD (LA-BALD), to maximize the information contributing to the labeled dataset via human annotations and the model. The simulated experiments on ImageNet100-sandbox show that LA-BALD reduces the number of annotations by 19% and 12% on average compared to the two types of baselines. Our analysis shows that LA-BALD provides both more accurate annotations and a better online-learned predictive model, leading to better labeling efficiency over the baselines.


Under review as a conference paper at ICLR 2023
LA- LA-BALD: AN INFORMATION-THOORETI IMAGE LABELING TASK SAMPLER
Anonymous authors Paper under double-blind review
ABSTRACT
Large-scale visual recognition datasets with high-quality labels enable many com- puter vision applications, but also come with enormous annotation costs, especially since multiple annotators are typically queried per image to obtain more reliable label. Recent work in label aggregation consolidates human annotations by combin- ing them with the predictions of an learne-learned predictive model. In this work, we devise an image labeling task sampler that actively selects image-worker pairs to efficiently reduce the noise the human annotations and improve the predictive model at the same time. We propose an cetormation-thhoreee Aggregation BALD (LA-BALD), to maximize the information contributing to the labeled dataset via human annotations and the model. The simulated experiments on ImageNet 100-sandbox show that LA-BALD reduces the number of annotations by 19% and 12% average compared to the two types of baselines. Our anal- ysis shows that LA-BALD provides both more accurate annotations and better online-learned predictive model, leading to better labeling efficiency over the baselines.
task sampler, Label
INTRODUCTION
Machine learning has led to large advances in wide range of applications such as machine translation, early cancer detection, virtual reality, and autonomous driving. Large-scale labeled datasets play vital role in the success of modern ML. The CheXNet dataset Rajpurkar et al. (017)) benefits automatic chest radiograph interpretation by providing clinical decision support. The Waymo Open dataset (Sun al. (2020) advances machine perception for self-driving vehicles by collecting data from diverse geographic locations. Diverse animal datasets Beery et al. (2018); Swanson et al. 2015 facilitate automatic animal monitoring.
Human annotators play essential role in large-scale and high-quality dataset creation Vaughan (2188). The level of inter-human variabilities, such as workers' interest and familiarity with the topic. and perceived task difficulty Kazai et al. (2012), are major factors that govern dataset quality Giuftrida et al. (2018); Jungo et al.] (018). However, the inter-human variability is a double-edged sword for data labeling. With an unlimited monetary budget, we approximate true label distributions by sampling multiple diverse workers per each example Peterson et al. (211)). On the other hand, we might obtain high-variance noisy labels when we operate with limited budget and thus need to trust a single or very few annotators per example.
Given the collected human annotations, label aggregation a common way infer the latent true labels Zheng et al. (2017). Usually, the aggregator captures each annotation's 's quality by estimating the data difficulty and the workers' competencies. Recent success in label aggregation for image labeling Branson et al. (2017); Liao et al. (20TT) leverages data similarity via an online-learned predictive model to infer the true labels, increasing the labeling efficiency by a large margin. Under this framework, an aggregator's 's label quality is blocked by the individual annotation quality and the
predictive model's performance.
In this work, adopting the state-of-the-art label aggregator Liao et al. 2221 we propose an information-theoree image labeling task sampler, Label Aggregation BALD (LA-BALD), tar- geting both blockers: which image-worker pair provides the best expected quality, and ii) which image labels benefit the predicted model the most. We formulate two different goals into an information maximization problem. As shown in Fig. each annotation provides information the

---

Under review as a conference paper at ICLR 2023
Annotation:
Label Aggregator
Komondor
Train model, Predict
Task Sampler (This Work)
Update Individual Annotation Likelihood
High-Quality Dataset with Minimum Cost
Global Influence
Workers
Images
Labeled Dataset
Figure 1: Goal of an Image Labeling Task Sampler. State-of-the label aggregator Liao et al. (202T) leverages an online-learned predictive model increase the labeling efficiency. In this trame- work, each annotation influences the labeled dataset both locally (via individual human annotation) and globally (via the online-learned predictive model). The image labeling task sampler needs to maximize the information contributing the labeled dataset via the local and global influence by choosing image-worker pairs in an iterative fashion.
labeled dataset in two ways: local and global influences. The local influence is associated with the individual label, and the global influence relates the learne- predictive model. Our proposed task sampler maximizes the expected information contributing to the labeled dataset via both the local and global influence by choosing relevant image- worker pairs.
We provide rigorous empirical comparisons between LA-BALD and the baselines on ImageNet 00- sandbox Liao et al. (02T);; Tian al. (2020) and show that LA-BALD reduces the number of annotations by 19% and 12% average compared to two different types of baselines. Our analysis shows that LA-BALD provides both more accurate annotations
better online-learned predictive
model, resulting in the improvements over the baselines.
2 RELATED WORK
2.1 LABEL AGGREGATION
Crowdsourcing provides low-cost noisy annotations. To infer true labels from noisy annotations, must infer each annotation's 's importance. The seminal work this problem, dating back 40 years, is the Dawid-Skene model Dawid & Skene (1979) (DS model), optimized with EM Many variants Demartini (2018) adapt the DS model to different settings. GLAD Whitehill et al. 2009) additionally models heterogeneous image difficulty. Welinder al. 2010 uses multidimensional entities to represent workers with different competencies and biases. EBCC al. (2019) scales to larger worker cohorts by modelling worker correlations. However, as studied in the survey by Zheng et al. (217)) there no dominant label aggregation algorithm across all datasets. Recent trend Branson et al. (2017) in label aggregation combines manual labelling with automatic labelling, opening up a new research paradigm. In this work, adopt the label aggregator proposed by Liao et al. (202l) because of its thorough study of the different factors and improved labelling efficiency.
al. (0122); Kim & Ghahramani (2012); Ma
(2015); Van Horn al.
2.2
ACTIVE LEARNING
Active learning aims to increase the ML model performance and reduce the labelling cost by actively sampling labels from the oracle SSttles (209)). Recent work on active learning for neural networks approaches the problem by core-set construction (Snner Saavaesee (2018), information- theoretic scores Houlsby et al. (2011) with dropout mask Gal et al. (2017) Kirsch et al. 2019). or
'The code is released in the anonymized repository at http:///manyynouu..ppposcccccett 807D/READMF.md
2

---

Under review as a conference paper at ICLR 2023
reinforcement learning Fang et al. (2017). Unlike active learning, in image labelling, need to consider heterogencous workers and data re-labelling.
Heterogeneous workers yield noisy annotations. Prior work mitigates this problem by training different classifier Zhang & Chaudhuri (2013) to predict where weak workers differ from strong workers, leveraging abstention responses Yan et al. (2016) avoid noisy observations, filtering them out Younesian et al. (222)). CEAL Huang et al. (2017) and ABL Gao & Saar- Sechansky (220 further incorporate worker costs into consideration. Re-Labeling has been common practice in large-scale dataset annotation Lin et al. (2014). Prior work [pperotts et al. 2012) recommends re-labelling on selected subset when annotations are imperfect. Re-active learning Lin al. (2016)
computes the expected error reduction to the contribution of possibly repeated annotations.
Most related work to ours are Yan al. (201T 20126) which perform image-worker sampling to increase the model accuracy via an wwtoommataootttrretttee acquisition function. Unlike the learning algorithm used in prior work Yan et al. (2012b), our label aggregator learns from the worker annotations and combines them with worker annotations in probabilistic manner. In addition, image labelling, the target goal to increase the label accuracy, not the accuracy novel test set. To summarize, prior work Yan et al. (2012b) only considers the global influence described in Fig.
3 BACKGROUND
In this work, propose information we formulate the problem and introduce the notation and the essential background knowledge of the label aggregator in the prior work Liao et al. (202T). For more details the label aggregator, please refer to Liao et al. (202T)
-iformationn-hhooefiic theoretic task sampler for image labelling. In this section,
3.1 PROBLEM FORMULATION
We target the image labelling of a K-class image classification problem. Given N images X {z; (=i)==::N and M workers f with unknown skills W /= we infer the latent ground truth Y (U:li=1.N.34 KK. We represent each worker's skill wy as a K-class confusion matrix.
We formulate the task sampling problem as an active learning problem over images and workers. An acquisition function S(.) defined over images and workers. At each step the task sampler samples batch B of image worker pairs according to the acquisition function S. For each image- morkew pair (i,j), we obtain an annotation Zty. In Sec. 3.2 we explain how to infer the latent ground truth Y from the annotations Z.
3.2 LABEL AGGREGATION
Label aggregation consolidates noisy annotations. The state-of-the-art label aggregator Liao al. 20011: Branson et al. (017) follows the classic Dawid-Skene (DS) model [Dawid & Skene 1979) and performs sopectatioo-mmaxiizziieo generation of noisy annotations Z is conditioned on both the latent ground truth Y and the latent worker skills W, visualized Fig. 2 We can express the joint probability as p(Z, Y, W) Y (W)) Let the annotations before step as Z, and γ = ( W (wū be the maximum likelihood solution. At step t, Bayes rule in Eq. !
(EM) infer latent ground truths. In the DDS model, the
write down the E-step and M-step via the
We adopt the label aggregator in Liao al. (2021) that incorporates the predictions of an online- learned predictive model with DS model Branson et al. (2017); Liao et al. (221) results drastic improvements. The predictions pe (Y|X) are treated the prior for latent ground truths Y in Eq. 2 The online-learned predictive model trained on the estimated labels at the previous step p(Y 7 X). We defer the model training details to the appendix.
3

---

Under review as a conference paper at ICLR 2023
Algorithm 1 Image Labeling Process
1: function TASKSAMPLER
Zij
2: Input: Unconfident set U
3: Compute S j): V(i, f), i U,j [M]
4: ((11,11), -m (BB.B)) <- Top (i,j) pairs in S
5: return (1). (iB.jB)}
6: end function
1:M
7: Input: Images threshold 7
8: Zo = -
9: Y, W <- Label Aggregator( X, Zo) // Eq.
Figure 2: Graphical Model in the DS Model Dawid Skene 1979). We use to index images and to index workers. the DS model, the annotation žgj is conditioned on the ground truth y4 and the worker skills wy-
10: repeat
U U {il arg max di S 7, Vù: E Y
11:
REK {(11,11, ... (iB. - TaskSampler(U)
12:
Ze (Z Workers (( (1),,)),.... (iB. H)}
14:
Y,W +- Label Aggregator X, Z,) // Eq.
15:
t +
16:
until [U] =
E-step: p(Y Zz, X) x (
E-step: p(Y]Z) x p(Y))(Z()) W) M-step: p(W]Z,) x pW)))r
M-step: p(W Z1) (((W)p(Z,1 W)
(1)
+ arg min Σ C(x,, p(y6 x),,
(2)
For K-class image classification problem, at each step first determine the image set that requires annotations, the unconfident set U, constructed by 7-thresholding the max probability of the estimated labels Io over K classes. We sample B image-worker pairs according to the acquisition function S overall workers and the unconfident set U. Finally, we perform label aggregation in Eq with the annotations Z,. A the pseudocode of the online image labelling process.
3.3 BAYESIAN ACTIVE LEARNING DISAGREEMENT (BALD)
In this work, we formulate the problem as active learning problem that maximizes the information gained from the labelled dataset. Here, we give a brief introduction to applying coormatioon-thoreee acquisition functions in active learning. BALD Houlsby et al. 2 is ifformation-hhoorttie sampler for active learning, measuring the mutual information between ground truth and model parameters 0:
SBALD = I(y: 0) H(y:) HH]
(3)
where TH denote mutual information and entropy. Applying BALD to deep neural networks is challenging since it requires computing the distribution of millions of parameters. Prior works Gal & Ghahrrmann] (2066); Gal et al. (2017) use MC dropout to perform variational approximation, Recent works Kirsch et al. (2021) Minddermmmmmmmm Mindermann al. (202T) extend BALD to training examples prioritization and distribution shifts between unlabeled sets and test sets, respectively. We defer the details of MC dropout the appendix. In this work, we extend BALD image labelling.
METHOD
Our goal is to improve labelling efficiency by actively sampling image-worker pairs. Since the label aggregator generates labels, we need to consider how the label aggregator processes the annotations when designing our image labelling task sampler. In particular, we target the label aggregator discussed in Sec. 3.2 The main challenge is quantifying the improvements of the online-learned model and the improvements through EM under the same framework.
4

---

Under review as a conference paper at ICLR 2023
We devise novel information -hheoorttctthhttcccccco acquisition function, LA-BALD, to sample image- worker pairs. LA -A--AALD is defined over image-worde worker pairs and explicitly considers the information gains to the labelled dataset in two ways: local and global influences. In this section, we start by inspecting how the labels change after getting a single annotation in Sec. 4.1 and propose LA-BALD in Sec. 4.2 Finally, we analyze the bias of LA-BALD. We provide additional analysis in the appendix.
4.1 LABELS CHANGE FROM A SINGE ANNOTATION
We adopt the label aggregator described in Sec. 3.2 By adding an annotation zzj. the annotation likelihood is expanded to p(zzj. Y W. Tracing the exact change adding an arbitrary annotation in Y, W is computationally expensive. Therefore, we make the following assumption: Assumption 1. By adding an annotation 2ij. the change of the estimated worker skills are negligible.
p(W|ZA s p(W Zaj Zt)
We only use the Assumption when computing the acquisition function. We still perform standard EM during the label aggregation to compute W as in Eq 2
With Assumption M we express the annotation likelihood by using the estimated worker skills from the last step p(zy= YY W). In the DS model, every annotation is independent of each other, and the annotation Zij only depends
its latent ground truth y; and worker skills viy- Therefore, we have p(z;j, Ze Y, W) = p(zig Mi:, úū )p( Z, W). We refer to the additional term p(z;g lys, (iş) as local influence since only affects the zth label.
The added annotation 2ij can lead to a larger or more accurate dataset and improve the online-learned ML model accordingly. Since we use the predictions of the ML model as the prior for the ground truths, the improved ML model affects all the labels Y. We refer to the change of the ML model as global influence.
We ground the local and global influence of adding single annotation zij in the label aggregation step in the following:
p(Y Fij, Fij Zz X, W)
po xo p(zij Vi. p(z,),
glohul influence
local influence
(4)
0 arg min [(r p(yslzig Ze, xx)), 8') + Elr 2;),,
From Eq. local influence only affects individual label yi. while global influence affects all labels Y. In the next section, we propose LA-BALD, which balances local and global influences brought by adding the annotation Zij.
4.2 LA-BALD: LABEL AGGREGATION BALD
We propose an information-theortii acquisition function to explicitly consider the local and global influence in the label aggregator. The prior art Gal et al. (2017) measures the mutual information between ground truth and model parameters 1(y; 0) as the acquisition function active learning. In image labelling, we obtain noisy annotations from workers. Therefore, mutual information between the annotation and the model parameters T(21y; X, Zz, W). We will ignore the conditional variables X, Z,, W in the following for clarity.
interested in the
We interpret the mutual information as the entropy reduction after conditioning the other variable, so T(zyy: 0) tells us the entropy reduction of the model parameters after conditioning on the annotation Zij, corresponding to the global influence in Sec. 4.1
As described in Sec. 4.1 adding an annotation affects both locally and globally. Therefore, we consider additional term measuring the mutual information between the annotation and the individual label, 2((2 y:). However, directly summing up I(zzy; 8) and I[zy: y4) overestimates the influence gain from the annotation ži, as shown in Fig. Bla. Instead, modify the local influence to be I(2izi ysld) which measures conditional entropy reduction of yild after conditioning on the annotation zij- The acquisition function of LA-BALD becomes:
5

---

Under review as a conference paper at ICLR 2023
Zij
Actesses
Ainagrs
Worker Ace.
Dog
18330
Insect+Fungus
13000
Veriebraee
18000
0.72
Commodity
16000
0.76
(a) May. (9) Maza y)
Key. 0) Magini/
Place
15000
DogeVerietrate
49339
0.59
Figure 3: Venn Diagram between the annotation Σ:zs the in- dividual label y/ and the model parameters 0. (a) Directly summing up I22y; 0) and I[(zy;; y/) overestimates the informa-
Figure 4:
Statistics of
ImageNet100 (ongeeNett0-sanddox provides sub-tasks with various difficulties affects to the labels. T[(zy; () is in charge of the global influence, and realistic worker simulation. while Zzzz yı (0) measures the local influence.
sub-tasks.
tion gain from the annotation 2ij. (b) LA-BALD explicitly considers the local and global influence that the annotation
SLA ((I (2zy + (12es;
(5)
Later in this section, discuss the biases introduced by I[(zy): 0) and ((
4.3 BIAS INTRODUCED BY ((zyj) 0)
Dissecting the bias introduced in ZZi;;; 8) and [Zzy); wi(0) helps understand the interactions between local and global influence. To investigate which image-worker pairs are preferred by I(zyj 0), we compare it with BALD scores in Eq. I[y;; 0) measures the entropy reduction of the parameters after conditioning on the ground truth y/.
Tyye 0) is the upper bound Tfzsz if zij w/ yıs where wy the confusion matrix representing the jth worker skills.
Proposition
Proof. With slight overloaded notations, y; is an one-hot vector over classes and žyy is a probability distribution over K class. With Zij = wo yi. from the data process inequality Beaudry & Renner 20TT) have the following inequality:
[zzg 0) = [[w] y1;0) S [1(y;; 0)
The equality holds if wy an invertible function.
Proposition tells us that I(zy;; 0) is maximized when I( 8) is maximized and Wi is invertible. I(y;)) b) tells us how much information gain we can have when obtaining the ground truth of the 7th image, and an invertible worker wy provides noiseless annotations. These elements are aligned with the intuition to have more valuable and reliable annotations in image labelling.
4.4 BIAS INTRODUCED BY (21)18110)
To investigate which image-worker pairs are preferred by ((21))3100) we first bring back the ignored conditional variables Z,, W and we have:
Proposition 2. If the label prior p(y.) is uniform distribution. I[(2yy) v0 Ze, W) can be expressed as follow:
I( v:10, Ze, W)
1
= Ex, Σ pluale 0, Z,, W) log p(a Ur w.))
log p( (ogp(ziylurs úz)
K
g1
Proof. We provide the full derivation in the appendix.

---

Under review as a conference paper at ICLR 2023
From proposition 2 express [zyy ys|0, Ze, W) into two terms i) minus entropy between
the annotation likelihood p(z M6. ú;) and the label posteiror p(y: 22g, 0, Z,, W) if) cross entropy between the annotation likelihood p(2y) (s. (iz) and the uniform prior. The image- image-worker pairs (z. j) provides higher local influences when 1) the annotation 2ij is different from the current posterior belief and 2) the jth worker has a higher probability of providing the annotation with higher certainty.
5 EXPERIMENTS
In this section, we show the efficacy of LA-BALD compared the baselines and analyze the behavior of LA-BALD. We first introduce the image annotation benchmark, ImageNet AmageNet100-sandeee -0ndddoo in Sec. 5.1 and detail the baselines in Sec. [.2 Then, in Sec. 5.3 we show that LA-Bi LA-BALD outperforms the baselines on most datasets, showing the importance of the additional term == ti (0). In Sec. 5.4
we dissect if the improvements of LA-BALL come from having more accurate annotations or better model. We provide additional analysis visualizing the interactions between the local and global influence through the annotation time in the appendix.
5.1 IMAGENET -0-SAADBOX
Evaluating and ablating multi-class image annotation efficiency scale requires large-scale datasets with diverse and relatively clean labels. We adopt the image annotation benchmark, ImageNet sandbox Liao et al. 2021). Some image classification datasets, e.g.. Snapshot Serengeti Swanson ct al. (2015), provide raw annotations of each worker, potentially allowing us to simulate the labelling process. However, most of the workers in Snapshot Serengeti only provide less than 10 annotations, making it hard to fit into our experimental setting.
1mageNet100-sandhox uses 100 classes sampled from the ImageNet label set TTan al. and constructs smaller sub-tasks of varying difficulty. In addition, (rnggeeet00--snnddiie provides realistic worker simulations initialized by the offline crowdsourced worker information. We simulate 50 crowd workers and 10 domain experts. Each worker is represented as a confusion matrix over the label space. The annotation 24y is drawn from the / row of the j worker's confusion matrix. We summarize ImageNet 1aett100-samm -0-sddd in Table 4.2 and provide more simulation detail in the appendix.
5.2 BASELINES
We categorize our baselines into sequential samplers (SS) and joint samplers (IS). SS compute the scores of images and workers separately, i.e., Si) and (j), while IS treat them as a whole ((,,)). Depending on which dimension gets sampled first, we categorize SS into image-centric and worker-centric SS.
For SS, we adopt image-centric SS (image -> worker) since worker-centric SS (worker + image) might overlook specific hard images. We use the following as our JS baselines: Rand-I+Rand- W. Randomly select the image and worker. Rand-I+Greedy-W Randomly select an image and choose greedily the worker based on the estimated worker skills. Ent-L+Rand-Wy W. Select the image with the highest entropy in label distribution and randomly select the worker." Ent-I+Greedy-WW -. Select the image with the highest entropy in label distribution and greedily select the worker based on the estimated worker skills.
For JS, adopt the BALD Gal et al. 2017); Houlsby et al. (211) in Eq. active learning, to image labelling. The acquisition function of J-BALD is BALDD(( which only considers the global influence. A similar idea resonates around a decade ago in Yan et al.) (2122)) but they do not consider the label aggregator with both local and global influence.
originally proposed in
T[zejj 0).
5.3 LABELING EFFICIENCY IN COOTEENET1O--AADDEEE
For all experiments, we refer "label accuracy" as the final labels accuracy, "machine accuracy" as the accuracy of the online learning model in Eq. and "annotation accuracy" as 2zj accuracy. We
7

---

Under review as a conference paper at ICLR 2023
Dog
Vertebate
Insect Fungus
Commodity
Place
0.46
0.96
0.50
0.52
Label Azc.
0.0
0.92
0.00
0.80
0.0
0.04
0.00
0.0
MAnistatioons
Figure 5: Image Labeling Efficiency on ImageNet100-sandhox We compare LA-BALD with J-BALD, inspired by Gal et al. (011)) Houlsby et al. (011)) LA-BALD reduces the number of annotatiosn by 5% to 37% compared to J-BALD.
run 3 different random seeds in all experiments. We use high-opacity lines to represent the average of different runs and low-opacity lines to represent each run.
We first compare LA-BALD with the JS baseline in Fig. 5 and compare LA-BALD with the SS baselines in Fig- 6 We run over three random seeds for every run and report the average. We show each run with lower opacity at the back. If the different runs end at different annotations, we only report the average when all runs have sample points.
LA-BALD vs. J-BALD. As demonstrated in Sec. J-BALD considers the change of the ML model but ignores the change of the individual annotation likelihood. In Fig. 5 show that LA-BALD outperforms J-BALD in Dog. To be more specific, LA-BALD reduces the number of annotations by 26% compared to J-BALD when reaching 85.2% label accuracy. LA-BALD also gives consistent improvements in other sub-tasks, reducing the number of annotations by 5% to 37% compared J-BALD, Vertebrate (-37%), Fungus (-21%), Commodity (-5%), and Place (-6%).
To further understand when we can expect more significant improvements, we compute the average of the top 10 pairs of worker confusion in each task and find that LA-BALD provides more significant improvements in the sub-tasks with higher average confusions, e.g., Dog (0.31), Vertebrate (0.28) and Insect+ Fungus (0.24). On the other hand, the improvements are smaller in the sub-tasks with lower average confusions, c.g., Commodity (0.16) and Place (0.12).
Another reason for the superiority of LA-BALD is that LA-BALD tends to sample the images with higher worker confusion later in the labelling process. Image classes with higher worker confusions, e.g., fiddler crab and rock crab, tend to confuse the online-learndd ML model since the ML model is trained on the worker annotations. When the number of annotations is small, the noises from the worker confusion lead overconfident model predictions and therefore decrease the labelling efficiency. To validate if J-BALD tends to sample the images with high confusion in the early stage of the labelling process, we compute the probability that the images from the most confusing class in each sub-task are sampled in the first half of the labelling process. We find that J-BALD is on average 6% more probable to sample images from the most confusing class than LA-BLDD in Dog (+2%), Vertebrate (+11%) () and Insect+ Fungus (+6%).
LA-BALD vs. SS. We show that LA-BALD brings consistent improvements compared the SS baselines across different sub-tasks. Fig. shows that LA-BAL.D reduces the number of annotations by 12% on average compared to the second-best SS baselines in each sub-task, ranging from 3% to
20%, Dog (-5%), Vertebrate (-18%), Insect+Fungus (-20%), Commodity (-2%), and Place (-13%). Note that prior work Liao et al. (202I) adopts
(black curves in Fig. 3 LA-BALD
almost halves the number of annotations required in Liao et al. (021
Larger Datasets. To see if LA-BALD can scale to larger datasets, we conduct experiments on a larger dataset containing 37 classes, Dog+ Vertebrate. Fig. 7 shows that LA-BALD outperforms all the baselines in terms of labelling efficiency, reducing the number of annotations by 30% compared to J-BALD and 14% compared Ent-I+Greedy-W.
8

---

Under review as a conference paper at ICLR 2023
Dog Veriebate
Draintande
Graddy
Dog
Vertebate
Insact Fungus
Commodity
0197
0.20
1.79
07
0.9
0.22
0.02
300
AAremdadionss perimngge
#Armitations per imagee
Figure 7: Comparison on larger sub-task. We compare the labeling ef- ficiency of LA-BALD and the baselines.
Figure 6: Comparison of LA-BALD and Sequential Samplers. LA- BALD outperforms the sequential samplers (SS) in every sub-task, reducing 12% number of annotations compared the second-best SS basclines in each sub-task.
Dog
Vertebate
Place
JAADD
LAWAAD
6:00
Arrodaaion Adc
0.00
Machine
0u10
LGAALD
0:00
D13 WestES 198S PURDO DURFOOS
Rond Hocrrity
155
0.
0.4
66:
59
64
06
Figure 8: Annotations Accuracy of J- BALD and LA-BALD. With similar costs, the accuracy of the annotations sampled by LA-BALD is 6% on average better than that sampled by J-BALD.
#Armotations image
Figure Comparison of the MI. Model Accuracy. The model trained on the annotations collected from LA-BALD learns quicker than the models trained on the annotations sampled from SS.
5.4 MORE ACCURATE ANNOTATIONS BETTER MODEL?
We analyze if the improvement of LA-BALD comes from having more accurate annotations better model.
LA-BALD J-BALD. Fig. g shows that the annotations sampled by LA-BALD are 6% better than the annotations sampled by J-BALD on average, showing that LA-BALD better at choosing relevant image-worker pairs compared to J-BALD. On the other hand, both of them achieve similar ML model performances. To summarize, LA-BALD obtains similar performant ML models compared J-BALD but better choosing relevant Image-worker pairs, leading better label accuracy.
LA-BALD vs. SS. Since most sequential samplers tend to stop at early steps and, therefore, only annotate small set of images, comparing the average annotation accuracy does not provide fair comparison. Instead, we compare the MI. model performance along the time axis Fig. y We ignore the SS baselines with random worker selection for clarity. The model learns quickly in the early stage when trained on the annotations collected from LA-BALD. Please refer to the appendix for complete comparison of the model accuracy. To summarize, LA-BALD quickly obtains better ML models, leading to better label accuracy.
6 CONCLUSION
This work presents LA-BALD, which explicitly considers the local and global influences identified in the sfae--of-th-arr label aggregator. Our studies find that L.-BALD reduces the number of annotations by 19% compared to the common sampler baseline, J-BALD, and 12% compared to the sequential sampler baselines. Our analysis shows that LA-BALD provides more accurate annotations and better leamed predictive model, leading to better labelling efficiency over the baselines.

---

Under review as a conference paper at ICLR 2023
7 REPRODUCIBILITY STATEMENT
To implement LA-BALD, heavily rely MC dropout (explained in Sec. refback and the appendix). For the additional term l[(z)y yı (0), we follows the BatchBALD Kirsch et al. (2119) implementation. We first separate the second term into 2(2 y: 22, H 24g 3/. B). The challenge is to sample p(zi (3. 8) = p(zty ((2))8:))p p (theta). Following the prior work Kirsch al. (2019), we sample the second term via MC dropout. We release the code in the anonymized
repository at
ETHICS STATEMENT
Our work could be applied to many computer vision applications, especially image classification problems requiring high expertise. For example, it could reduce the number of annotations required to train a fine- fine-graing grained bird classification model used in mobile apps. Our method actively allocates different images to workers, favouring workers with higher expertise. Many reasons lead the lack of expertise, such as lack of background knowledge, motivations, and familiarity with the tool. However, expertise change over time. Since our proposed method favours workers with higher expertise, could fail to assign the tasks to new incoming workers, who show low expertise due to the lack of familiarity with the tool. This leads to an imbalanced workload between workers. Expert workers overwork, while workers with low expertise lose their jobs.
Encouraging a balanced workload is essential in practice since it implies fair job recruitment and better parallelism. We can adopt e-greedy sampling strategy to encourage a balanced workload with LA-BALD. For each step, the task sampler performs random sampling with probability e; otherwise, it performs greedy sampling.
REFERENCES
Normand Beaudry and Renato Renner, An intuitive proof of the data processing inequality. Quantum Information and Computation, 12, 07 2011.
Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
Javad Zolfaghari Bengar, Joost van de Weijer, Bartlomiej Twardowski, and Bogdan Raducanu. Reducing label effort: Self-supervised meets active learning, 2021.
Steve Branson, Grant Van Horn, and Pietro Perona. Lean crowdsourcing: Combining humans and machines in online system. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), July 2017.
Yao-Chun Chan, Mingchen Li, and Samet Oymak. On the marginal benefit of active learning: Does self-supervision eat its cake? In ICASSP 202/ -002 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 3455-3459, 2021. 9414665.
doi: 10. (0.1109/CAASSP9972222000
Alexander Philip Dawid and Allan Skene. Maximum likelihood estimation of observer error-rates using the em algorithm. Journal of the Royal Statistical Society: Series C' (Applied Statistics), 28 (1):20-28, 1979.
Gianluca Demartini, Djellel Eddine Difallah, and Philippe probabilistic reasoning and crowdsourcing techniques for large-scale entity linking. In Proceedings of the 21st International Conference on World Wide Web, www 12, pp. 469-478, New York, NY. USA, 2012. Association for Computing Machinery. ISBN 1781450312295. doi: 10.1145/ 2187836.2187909 URL /Utp://ttLt/////oaa...oo///oo/.. 100.//..14422220 2832221888332. 2187900
Zencrowd: Leveraging
Meng Fang, Yuan Li, and Trevor Cohn. Learning how to active learn: A deep reinforcement learning approach. In EMNLP, 2017.
10

---

Under review as a conference paper at ICLR 2023
Yarin Gal and Zoubin Ghahramani. Dropout Bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the 33rd International Conference on Machine Learning (ICML-16, 2016.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In Proceedings of the 34th International Conference Machine Learning- Volume 70, ICML' 17, pp. 1183-1192 Jog,. 2017.
Mingfei Gao, Zizhao Zhang, Guo Yu, Sercan O. Arik, Larry S. Davis, and Tomas Pfister. Consistency- based semi-supervised active learning: Towards minimizing labeling cost, 2020.
Ruijiang Gao and Maytal Saar- -sechascca., Cost-accuracy aware adaptive labeling for active learn- ing. Proceedings of the AAAI Conference Artificial Intelligence, 34(03):259-22576 Apr. 2020. doi: 10.1609/ htt.://///aaa..44........33 v34103.5640. BRLhttps:: / /ttps//jss.o... aaa.iis.aaaa.................... wndex.n pbAAAA/AA
article/view/54D
M. Valerio Giuffrida, Feng Chen, Hanno Scharr, and Sotirios A. Tsaftaris, Citizen crowds and experts: observer variability in image-based plant phenotyping. Plant Methods, 14, 2018. URL
ht.pss://do http:://doo.orgg////
/0. C. 1186/ s 3000000111-007777 0278-7
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap your latent a new approach to self-supervised learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 21271-21284. Curran Associates, Inc., 2020. URL
cc/
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1321-1330. PMLR, 06-11 Aug 2017. URL https://proceddings www.nngsssm.....mm http://eedd.ag........rssss////e guol hutml
Neil Houlsby, Ferene Huszar, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning. CoRR, abs/ 1its//112 2011. URL http: dbll
uni-trier de/ journa 1s htt//////oorrllrecc 1
Sheng-Jun Huang, Jia-Lve Chen, Xin Mu, and Zhi-Hua Zhou. Cost-effective active learning from diverse labelers. In Proceedings of the Twenty-Sive Sixth International Joint Conference Artificial
Intelligence, IJCAI-17, pp. 1879-18888 1885, 2017. doi: 10.24966/iccai22117761.. URL https
V/doi. org. 10. 24963/ caai 20170017222
Siyu Huang, Tianyang Wang, Haoyi Xiong, Jun Huan, and Dejing Dou. Semi-supervised active learning with temporal output discrepancy, 2021.
Panos Ipeirotis, Foster Provost, Victor Sheng, and Jing Wang. Repeated labeling using multiple noisy labelers. Data Mining and Knowledge Discovery, 28, 10 2012. doi: 10.1007/s10618, 01661880---006-66
Alain Jungo, Raphael Meier, Ekin Ermis, Marcela Blatti-Moreno, Evelyn Herrmann. Roland Wiest, and Mauricio Reyes. On the Effect of Inter-observer Variability for Reliable Estimation of Uncertainty of Medical Image Segmentation, pp. 682-690. 09 2018. ISBN 978-3-030-00997-4. doi: 10.1007/978-3-0000 1030000928-1222
Gabriella Kazai, J. Kamps, and Natasa Milic-Frayling. An analysis of human factors and label accuracy in crowdsourcing relevance judgments. Information Retrieval, 16:138-178, 2012.
Hyun-Chul Kim and Zoubin Ghahramani. Bayesian classifier combination. In Neil D. Lawrence and Mark Girolami (eds.), Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics, volume 22 of Proceedings of Machine Learning Research, pp. 619 627. La Palma, Canary Islands, 21-23 Apr 2012. PMLR. URL https://prooeedingg.. mir
press/ Preas/v22/K5 v22. 1immm..1 html]
11

---

Under review as a conference paper at ICLR 2023
Andreas Kirsch, Joost Amersfoort, and Yarin Gal. Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. htt.tttps:/pppooeee
neerips
cC/
Andreas Kirsch, Tom Rainforth, and Yarin Gal. Test distribution-aware active learning: principled approach against distribution shift and outliers, 2021.
Yuan Li, Benjamin Rubinstein, and Trevor Cohn. Exploiting worker correlation for label aggregation in crowdsourcing. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 3886 3895. PMLR, 09-15 Jun 2019. URL httpsss//ppoccccllgggg
html
Yuan- Yuan-Hong Liao, Amlan Kar, and Sanja Fidler. Towards good practices for efficiently annotating large-scale image classification datasets. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4350-4359, June 2021.
C. H. Lin, Mausam, and Daniel S. Weld. To re(label), or not to re(label). In HCOMP, 2014.
C. H. Lin, Mausam, and Daniel S. Weld. Re-active learning: Active learning with relabeling. In AAAI, 2016.
Fenglong Ma, Yaliang Li, Qi Li, Minghui Qiu, Jing Gao, Shi Zhi, Lu Su, Bo Zhao, Heng Ji, and Jiawei Han. Faitcrowd: Fine grained truth discovery for crowdsourced data aggregation. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD '15, pp. 745-754, New York, NY, USA, 2015. Association for Computing doi: 10.1145/2783258278831311 URL httphtt////o..orr
Machinery. ISBN 450363636
10. 11 14452763258 2783314
Sören Mindermann, Muhammed Razzak, Winnie Xu, Andreas Kirsch, Mrinank Sharma, Adrien Morisot, Aidan N. Gomez, Sebastian Farquhar, Jan Brauner, and Yarin Gal. Prioritized training on points that are learnable, worth learning, and not yet learned, 2021.
Takumi Nakayama, Masaki Matsubara, and Atsuyuki Morishima. Crowd-Worker Skill Improvement with Al Co-Learners, pp. 316-322. Association for Computing Machinery, New York, NY. USA, 2021. ISBN 9781450386203. http:tpp.//.....o//111111///////
3484684
Hieu T. Nguyen and Arnold Smeulders. Active learning using pre-clustering. In 1 Proceedings of the Twenty-First International Conference on Machine Learning, ICML '04, pp. 79, New York, NY, USA. 2004. Association for Computing Machinery. ISBN 1581 138388. doi: 10.1145/1015330.
Ttt.1111333.11133... 1015349. URL httttt:://ddo /tt.::////....coo/....////0000..
Joshua C. Peterson, Ruairidh M Battleday, Thomas L. Griffiths, and Olga Russakovsky. Human Uncertainty makes Classification more Robust. In International Conference on Computer Vision (ICCV), 2019.
Pranav Rajpurkar, Jeremy Irvin, Kaylie Zhu, Brandon Yang. Hershel Mehta, Tony Duan, Daisy Ding, Aarti Bagul, Curtis Langlotz, Katie Shpanskaya, Matthew P. Lungren, and Andrew Y. Ng. Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning, 2017.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In International Conference on Learning Representations, 2018. URL https:/7
hwtp://eee/ooourmmmmmmmmmm openreview.co net. forum? dd=lla Iuk-RV
Burr Settles. Active learning literature survey. Computer Sciences Technical Report 1648, University of Muscoosii-MMadioon 2009. URL attp://axon. CS byu. wdw. wtt.:///wmaatteeeeeeeeeeeee 778/Papers/sett. les.etteee.... act ve learning.ppl
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Sheng Zhao, Shuyang Cheng. Yu Zhang, Jonathon Shlens, Zhifeng Chen, and Dragomir Anguelov. Scalability in perception for autonomous driving: Waymo open dataset, 2020.
12

---

http://eeeeeeeeeeeeppssssss///ee Under review as a conference paper at ICLR 2023
AB Swanson, M Kosmala, CJ Lintott, RJ Simpson, A Smith, and C Packer. Data from: Snapshot serengeti, free-freuenncy annotated camera trap images of 40 mammalian species in an african savanna, 2015. httpss://ooi.oroo// 5061/ /or0////ryad..m 5pt
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In ECCV, 2020.
Grant Van Horn, Steve Branson, Scott Loarie, Serge Belongie, and Pietro Perona. Lean multi- class crowdsourcing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Jennifer Wortman Vaughan. Making better use of the crowd: How crowdsourcing can advance machine learning research. Journal of Machine Learning Research, 18(193): 2018. URL
Peter Welinder, Steve Branson, Pietro Perona, and Serge Belongie. The multidimensional wisdom of crowds. In Lafferty. C. Williams, J. Taylor, R. Zemel, and A. Cu- lotta (cds.), Advances in Neural Information Processing Systems, volume 23, Curran Asso- ciates, Inc., 2010. URL UIScafa014dbb. 0ataaaaa11aaa0077770000022--2-20
www.:eeedddnn....nrr.ss...... neurips http:///e..aaaaoc///////////////
pdf
Jacob Whitehill, Ting-fan Wu, Jacob Bergsma, Javier Movellan, and Paul Ruvolo. Whose vote should count more: Optimal integration labels from labelers unknown ex- pertise. In Y. Bengio, D. Schuurmans, J. Lafferty, C. Williams, and A. Culotta (eds.), Advances in Neural Information Processing Systems, volume 22. Curran Asso- ciates, Inc., 2009. URL Itttps://proceeednnne neurips. pctpcc//ppp////3//11eel Papeer..ddpppp..ppp..
Songbai Yan, Kamalika Chaudhuri, and Tara Javidi. Active learning from imper- feet labelers. In Lee, M. Sugiyama, U. Luxburg, Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29, Curran Asso- ciates, Inc., 2016. URL httpt:
ptcp///ss..cc/ppar////00//////// pdff
Yan Yan, Rómer Rosales, Glenn Fung, and Jennifer G. Dy. Active learning from crowds. In ICML, 2011.
Yan Yan, Romer Rosales, Glenn Fung, Faisal Farooq, Bharat Rao, and Jennifer Dy. Active learning from multiple knowledge sources. In Neil D. Lawrence and Mark Girolami (eds.), Proceedings of the Fifieenth International Conference on Artificial Intelligence and Statistics, volume 22 of Proceedings of Machine Learning Research, pp. 1350- 1357, La Palma, Canary Islands, 21-23 Apr htw.gggssss.psps.ss.pppp.m
2012a. PMLR. URL
mlm press//s///........ wwt..pppessssseeeeeeeeeeee w222//22222aa222 html
Yan Yan, Rómer Rosales, Glenn Fung, Faisal Farooq, R. Bharat Rao, and Jennifer G. Dy. Active learning from multiple knowledge sources. In AISTATS, 2012b.
Taraneh Younesian, Dick Epema, and Lydia Y. Chen. Active learning for noisy data streams using weak and strong labelers, 2020.
Chicheng Zhang and Kamalika Chaudhuri. Active learning from weak and strong labelers. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Gamett (eds.), Advances in Neural Information Processing Systems, volume 28. Curran Asso- ciates, Inc., 2015. URL
Yudian Zheng, Guoliang Li, Yuanbing Li, Caihua Shan, and Reynold Cheng. Truth inference in crowdsourcing: Is the problem solved? Proc. VLDB Endow., 10(5):441-552, jan 2017. ISSN 2150- 8097. doi: 0117778830555440005554,, 3055547
org Lt
13


Reviews:
[{"review_id": "wHltwBQ8Vq0", "reviewer": "Reviewer_B5qt", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Briefly, to efficiently reduce the noise in the human annotations and improve the predictive model, this paper introduces an image labeling task sampler by actively selecting image-worker pairs according to the formulated information maximization problem, i.e., improving the quality and benefit of image-work pairs for the predicted model. The experimental results are strong.", "summary_of_the_review": "This paper requires significant improvements. Please address my concerns. I will give a higher rating if most of them are well-addressed. ", "strength_and_weaknesses": "Strengths:\n\nExtensive experimental results demonstrate the effectiveness of the proposed method.\n\nWeaknesses:\n\n1) The novelty in the proposed contributions (employing the label aggregator) to solve the issue is somewhat limited and incremental. More discussions and analyses are required to highlight the contribution of the proposed method.\n\n2) It seems that the proposed method heavily relies on its assumptions (e.g., Assumption 1). This makes the paper of low interest, especially to those expert readers who may not be necessarily working on image classification but other vision problems.\n\n3) The authors do not provide a detailed complexity analysis and efficiency comparison.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "WeblL1Cb9AI", "reviewer": "Reviewer_vATu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces information-theoretic task sampler, Label Aggregation BALD (LA-BALD) that actively selects image-worker pairs to maximize the information contributing to the labeled dataset via human annotations and the model, and efficiently reduce the noise in the human annotations and improve the predictive model.  ", "summary_of_the_review": "Although there are some questions, the paper proposes an interesting and effective sampler for improving labeling efficiency, and also provides some theoretical analysis and detailed formula derivation. ", "strength_and_weaknesses": "The major contribution of this work is to extend  bayesian active learning by disagreement to image labeling and incorporates the label aggregator. The combination on improving labeling efficiency and solving local and global influences of individual annotation is interesting. However, there are still some questions that need to be explained.\nQuestions and detailed comments:\n1. What is the motivation to consider BALD to imgae labeling and the difference with the framework motioned by this paper [A].\n2. As mentioned in figure 3, the summing up $\\mathbb{I}\\left(z_{i j} ; \\theta\\right)$ and $\\mathbb{I}\\left(z_{i j} ; y_{i}\\right)$overestimates the information gain from the annotation $z_{i j}$. If the local influence is not considered, what changes will this overestimates？Please describe in more detail the importance of local influence.\n3. The details of the dataset is missing like the statistics and descriptions of the noisy label it contains. In addition to ImageNet100-sanbox, are there additional datasets for model evaluation? It will be hard for readers to catch up how challenging the dataset is and how effective the model is. \n4. The new-built baselines are all based on BALD and lack other public baselines on the ImageNet100-sanbox dataset. is there a lack of comparative experiments for method [A]? This comparison result does not clearly show and needs to be described in detail.\n[A] Yuan-Hong Liao, Amlan Kar, and Sanja Fidler. Towards good practices for efficiently annotating large-scale image classification datasets. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4350–4359, June 2021.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "BFNvMLoTk3S", "reviewer": "Reviewer_f1dh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed LA-BALD for crowdsource labeling of classification datasets. Experiments demonstrated the superiority of LA-BALD over two previous baselines by reducing 19% and 12% of annotation cost.", "summary_of_the_review": "This paper targeted at a practically important problem of efficient active crowdsource annotation, and proposed an effective approach based on local aggregation framework and Bayesian active learning by disagreement.\n\nHowever, I expect 1) more novel ideas 2) better experimental designs 3) clearer writing for recommendation of acceptance.", "strength_and_weaknesses": "S1. LA-BALD reduces the number of annotations by 19% and 1% on average compared to two baselines.\n\nW1. LA-BALD = LA + BALD. Neither LA (Liao 2021) nor BALD (Gal 2017) was novel.\n\nW2. Motivations of using mutual information or BALD were NOT clear.\n\nW3. Experimental design was relatively simple and did NOT bring more information except direction comparisons.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "A_rxDZXJYxH", "reviewer": "Reviewer_Dx5T", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "- The authors proposed an image labeling task sampler that actively selects image-worker pairs to reduce the noise in human label annotations and improve the predictive model.\n- The image labeling task sampler is based on an information-theoretic (Label Aggregation BALD, LA_BALD) to maximize the dependencies of the label datasets and the model parameters.\n- The experiments on ImageNet100-sandbox showed that the proposed LA-BALD reduces the number of annotations by 19% and 12% on average compared to the two baselines.", "summary_of_the_review": "- I summarized above.\n- There are unclear points to follow Propositions 1 and 2. It would be clear if empirical observations or ablation studies on those points were prepared.", "strength_and_weaknesses": "- (+) The LA-BALD maximizes the expected information contributing to the labeled dataset via local and global influence by choosing relevant image-worker pairs.\n    - The local influence is associated with the individual label.\n    - The global influence relates to the online-learned predictive model.\n    - The author showed several relevant application results - Bayesian active learning, Imbalanced Learning, and Neural Network Learning. However, there seem to be not enough comparison results.\n- (-) As the authors stated, computing the scores of every image-worker pair is time-consuming.\n- (-) There are no evaluations on semi-supervised works.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject