SIMILAR: Submodular Information Measures Based Active Learning In Realistic Scenarios
Authors: Suraj Nandkishor Kothawade
Nathan Alexander Beck
Krishnateja Killamsetty
Rishabh K Iyer
Venue: NeurIPS 2021 Poster (2021)
Keywords: Submodularity
Active Learning
Information Measures
Realistic
Redundancy
Rare Classes
Out of distribution
Robust

Abstract: Active  learning  has  proven  to  be  useful  for  minimizing  labeling  costs  by selecting  the  most  informative  samples. However,  existing  active  learning methods do not work well in realistic scenarios such as imbalance or rare classes,out-of-distribution data in the unlabeled set, and redundancy.  In this work, we propose SIMILAR (Submodular Information Measures based actIve LeARning), a unified active learning framework using recently proposed submodular information measures (SIM) as acquisition functions. We argue that SIMILAR not only works in standard active learning but also easily extends to the realistic settings considered above and acts as a one-stop solution for active learning that is scalable to large real-world datasets. Empirically, we show that SIMILAR significantly outperforms existing active learning algorithms by as much as ~5%−18%in the case of rare classes and ~5%−10%in the case of out-of-distribution data on several image classification tasks like CIFAR-10, MNIST, and ImageNet.

SIMILAR: Submodular Information Measures Based Active Learning In Realistic Scenarios
Suraj Kothawade University of Texas at Dallas suraj kothawade@utdallas edu
Nathan Beck University of Texas at Dallas nathan. beckkudaalaa edu
Krishnateja Killamsetty University Texas at Dallas krishnate killamsttty@uttallllla edu
Rishabh Iyer
University Texas at Dallas rishabh. Ayeerutdallas edu
Abstract
Active learning has proven to useful for minimizing labeling costs by selecting the most informative samples. However, existing active learning methods do not work well in realistic scenarios such as imbalance or rare classes, out-of-distribution data in the unlabeled set, and redundancy. In this work, we propose SIMILAR (Submodular Information Measures based actIve LeA Rning, a unified active learning framework using recently proposed submodular information measures (SIM) as acquisition functions. We argue that SIMILAR not only works in standard active learning but also easily extends the realistic settings considered above and acts real-worid datasets. Empirically, we show that SIMILAR significantly outperforms existing active learning algorithms by as much as R 5% 18% in the case of rare classes and x 5% 10% in the case of out- diitt--diittitttion data several image classification tasks like CIFAR-10. MNIST, and ImageNet. SIMILAR is available part of the DISTIL toolkit: https: // //gi/ttae gighub....... com wom/decile-eeeeooeeoeeeeeeeeeeee team/ diittid
one-stop solution for active learning that is scalable to large
Introduction
Deep neural networks (DNNs) have had a lot of success in a wide variety of domains. However, they require large labeled datasets which are
9
6
7
0
0 0
0
4
4
5
9
often taxing, time-consuming, and expensive to
9
6
7
0
0
9
3
4
3
2
obtain. Active learning (AL) [12, 13, 39, 3, 9] is promising approach to solve this problem. It aims to select the most informative data points from an unlabeled dataset to be labeled in an adaptive manner with human in the loop. The
/
6
4
4
8
3 6
6
7
7
5
4 6
O
8
4
4
7
4
6
/
2
3
3
R
B
F
goal of AL to achieve maximum accuracy of the model while minimizing the number of data points required to labeled.
2
3
2
O
& 7 4
Classes
Redundancy
[c] Out-of-distribution
Figure 1: Motivating scenarios for realistic active
Current AL methods have been tested rel- atively simple, clean, and balanced datasets. However, real-world datasets are not clean and have number of characteristics that makes learning from them challenging [10, 46, 47, 38, 1, 8J. Firstly, these real-world datasets are im- balanced, and some classes very (e.g., Fig (a)). Examples of this imbalance medical
learning: (a) rare classes: digits 5 and 8 are rare: (b) redundancy: digits 0 and are redundant; (c) -iitt---istriutiinn (OOD): letters A, R, B. digit classification.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).

---

imaging domains where the cancerous images are rare. Secondly, real-world data has a lot of redun- dancy (e.g., Fig I(b)). This redundancy more prominent in datasets that are created by sampling frames from videos (e.g., footage from a car driving on a freeway or surveillance camera footage). Thirdly, it is common to have out-of- of-dissribbution (OOD) (e.g., Fig I()) data, where some part of the unlabeled data is not of concern to the task at hand. Given the amount of unlabeled data, it is not realistic to assume that these datasets can be cleaned manually; hence, it is the need of the hour to have active learning methods that are robust to such scenarios. We show that current AL. approaches (including the statc-of-thed tatte-of-the-art approach BADGE [3]) do not work well in the presence of the dataset biases described above. In this work, we address the following question: Can machine learning model be trained using a single unified active learning framework that works for a broad spectrum of realistic scenarios? As solution, propose SIMILAR', unified active learning framework which enables active learning for many realistic scenarios like rare classes, out-of -iut---diitriuttion (OOD) data, and redundancy.
1.1 Related Work
Active learning has enabled efficient training of complex deep neural networks by decreasing labeling costs. The most commonly used approach is to select the most uncertain items. Examples of uncertainty strategies include ENTROPY [41], LEAST CONFIDENCE [44], and MARGIN [37]. One challenge of this approach is that all the samples within a batch can be potentially similar even though they are uncertain. To overcome this problem in batch active learning, many recent works have attempted to select diverse yet informative data points. [45, 22] propose simple approach: Filter set of points using uncertainty sampling and then select a diverse subset from the filtered set. [40] propose CORESET, which forms core-sets using greedy k-center clustering while maintaining the geometric arrangement. BADGE [3], another recent approach, proposes to select data points corresponding to nigh-magnitude, diverse hypothesized gradients by using K-MEANS++ [2] initialization to distance from previously selected data points in the batch. Most existing AL approaches fail to ensure diversity across AL selection rounds and do not perform as well when there is a lot of redundancy. Sinha al. [42] used a variational autoencoder (VAE) [25] learn a feature space and an adversarial network [32] to distinguish between labeled and unlabeled data points. However, their approach is computationally expensive and requires extensive hyperparameter tuning. Similarly, BATCUB ALD [26] does not scale to larger batch sizes since their method would need large number of Monte Carlo dropout samples to obtain a significant mutual information. Such limitations reduce the scope of applying these methods realistic settings.
Closely related to our work are two recently proposed works. The first is GLISTER- BAISTER-ACTOO ACTIVE [24], which formulates the AL acquisition function by maximizing the log-likelihood on held-out validation set. This validation set could consist of examples from the rare classes or in in-distribution examples. The second approach is the work of Gudovskiy et al. [15], who study AL for biased datasets using a self-supervised FISHER kernel and pseudo-label estimators. They address this problem by explicitly minimizing the KI. divergence between training and validation sets via maximizing the FISHER kernel. Although their method shows promising results, they make multiple unrealistic assumptions: a) They use large labeled validation set, and b) they use feature representations from model pretrained using unsupervised learning on a balanced unlabeled dataset. In this work, we compare against both MASSEE--ACTVIE [24] and FISHER [15] approaches in the more realistic setting of a small held-out validation set (smaller than the seed labeled set) and an imbalanced unlabeled set. Another work proposed discrete optimization method for k-NN-type algorithms in the domain shift setting [6]. However, their approach is limited to A--NS.
This work utilizes submodular information measures (SIM) by [19] and their extensions by [23]. SIMs encompass submodular conditional mutual information (SCMI), which can then be used to derive submodular mutual information (SMI); submodular conditional gain (SCG); and submodular functions (SF). We discuss these functions in detail in Sec. 2. [23] also studies these functions on the closely related problem of targeted data selection.
1.2 Our Contributions
The following are our main contributions: 1) Given the limitations of existing approaches in handling active learning in the real world, we propose SIMILAR (Sec. 3), unified active learning framework that can serve as a comprehensive solution to multiple realistic scenarios. 2) We treat SIM as a
SSbmodular Information Measures based active LeARning

---

common umbrella for realistic active learning and study the effect of different function instantiations offered under SIM for various realistic scenarios. 3) SIMILAR not only handles standard active learning but also extends to a wide range of settings which appear in the real world such as rare classes, out-of distribbitittiinnn (OOD) data, and datasets with lot of redundancy. Finally, 4) we empirically demonstrate the effectiveness of SMI-based measures for image classification (Sec. 4) in number of realistic data settings including imbalanced, out-of ditribbbbii,i,nnn, and redundant data. Specifically, in the case of imhalanced and OOD data, we show that SIMILARR achieves improvements of more than 5 to 10% on several image classification datasets.
2 Background
In this section, we enumerate the different submodular functions that are covered under SIM and the relationships between them.
We let denote the unlabeled set of n data points L/ = (1,2,11,33.00 n2 and a set function f 2 + TR. Formally, a function f is submodular [14] if for x E U. -( 2 ((BU)) - f(B), A C B C U and x & B. For a set A C U, [(A) provides a real-valued score for A. In the context of batch active learning, this is the score of an acquisition function f on batch A. Submodularity is particularly appealing because it naturally occurs in real world applications [43, 4, 5, 20] and also admits constant factor 1 [34] for cardinality constraint maximization. Additionally, variants of the greedy algorithm maximize a submodular function in near-linear time [33].
Submodular Functions.
Submodular Mutual Information (SMI. Given sets A, Q C U, the SMI [16, 19] is defined as (A; Q) = f(A) + f(2) f(A U Q). Intuitively, SMI models the similarity between Q and A, and maximizing SMI will select points similar to Q while being diverse. Q here is the query set.
Submodular Conditional Gain (SCG). Given sets A, C U, the SCG [(A(P) is the gain in function value by adding A to P. Thus, [(AAP)) ((UUP) - ((P) [19]. Intuitively, SCG models how different A is from P. and maximizing SCG functions will select data points not similar to the points in P while being diverse. We refer to P the conditioning set.
Submodular Conditional Mutual Information (SCMI) Given sets 2, S U, the SCMI is defined as (AA QIP) = ((AUP) S(QUP) - - ((AUQUU) - f(P). Intuitively, SCMI jointly models the similarity between A and Q and their dissimilarity with P.
Relationship between SIM The rela- tionship between the above measures is the key component that unifies our AL framework [19, 23]. The unification comes from the rich modeling capacity of SCMI: (A; Q)P) where Q. P C U. This facili- tates a single acquisition function that can be applied to multiple scenarios. Concretely, the submodular function ff can be obtained by setting Q + Ul and p <- O. Next, the SMI can be obtained by setting Q + Q and P < 0, while we obtain SCG by setting Q + 0, P + P. We summarize the relationships between SIM in Tab. 1.
Function Submodular
Setting
Realistic Scenario
Q x- Q-u, u,P+D00 4-0 0
Standard AL
SMI
Q - -. + 0
Imbalance, OOD Redundancy
SCG
Q Q- @-0.P. 0.P P
SCMI
P e Q,P
OOD
Table I: Relationship between SIM and their applica- tions to realistic scenarios by choices of Q and P.
Instantiations of SIM. The formulations for Facility Location (FL), Graph Cut (GC) and Log Determinant (LOGDET) are in [19, 23] and we adapt them acquisition functions for batch active learning. We use two variants for FL. FLQMI, which models pairwise similarities of only the query set Q to the unlabeled dataset. and FLYMI, which additionally considers the pairwise similarities within the unlabeled dataset U. The SCG and SCMI expressions corresponding to FL are referred as FLCG and FI.MMI respectively (see row 1 in Tab. 2a and 2b). For LOGDET, we refer to the SMI, SCG and SCMI expressions as LOGDETMI. LOGDETCG and LOGDETCMI, respectively (see row 5 in Tab. 2a and row 2 in Tab. 2b). Similarly, the SMI and SCG expressions are respectively referred to as GCMI and GCCG for GC (see row 3 in Tab. 2a and 2b). For notation in Tab. 2, the pairwise similarity matrix between items in sets A and B denoted as SA.B. Also, we denote Siy as the (i,j) entry of S.
3 SIMILAR: Our Unified Active Learning Framework
In this section, we propose unified active learning framework SIMILAR, which uses SIMs to address the limitations of the current work (see Sec. 1.1). We show that SIMILAR can be effectively applied to broad range of realistic scenarios and thus acts as one-stop solution for AL.
3

---

Table 2: Instantiations of SIM. Note how the relationships in Tab. can be applied to SCMI
instantiations to obtain SMI and SCG instantiations.
(b) Instantiations of SCG and SCMI functions.
(a) Instantiations of SMI functions.
SCG
[(A(P)
SMI
1, 2)
maxfmna S1z
max Siz, 0)
FLVMI
min(max Sizi Siz) JEA
LogDeICG
log degie SA 1.8pp5. sA,))
FLQMI
Sis+ Σ JEA
Σ
Sij
GCOG
((A) -2 Σ Sis
JER
ieddeP
GCMI
2 Σ Σ Stz
SCMI
KEASEO
LOGDETMI
det(( A) log SSt $4.0 5 5
FLCMI
(mix( max Sis max 8iz) max S)
JEG
dettl 8p o $7.00
LogDeCMI
log
2at(1- AUD A 2 Au
The basic idea behind our framework is to exploit the relationship between the SIMs (Tab. 1) such that can be applied to any real-world dataset. Particularly, we use the formulation of SCMI and appropriately choose query set Q and/or conditioning P depending on the scenario hand. Towards this end, we use the inspiration from [3] where they select data points based on diverse gradients. The SIM functions (see Tab. 2) instantiated using similarity kernels computed using pairwise similarities S.j between the gradients of the current model. Specifically, we define Siy = (0), (O), where (0) = HH(z; 0) is the loss on the ith data point. Similar to [45, 31, we use hypothesized labels for computing the gradients, and the corresponding similarity kernels. The hypothesized label for each data point is assigned as the class with the maximum probability. We then optimize SCMI function:
(, QIP)
AÇU,AAIBB
with appropriate choices of query set Q and conditioning set P. In the context of batch active learning, A is the batch and is the budget (batch size in AL). We present our unified AL framework in Algorithm and illustrate the choices of query and conditioning set for realistic scenarios in Fig. 2.
Algorithm 1 SIMILAR: Unified AL Framework
Require: Initial Labeled set of data points: L, large unlabeled dataset: U, Loss function H. for learning model M. batch size: B, number of selection rounds: N
1: for selection round i 1 N do
2: Train model M with loss H on the current labeled set and obtain parameters
3: Using model parameters 0,, compute gradients using hypothesized labels (VoH(azj ti 0), Vj E4) and obtain a similarity matrix X.
4:
Instantiate a submodular function If based on X.
5: A; + argmax 5 P)P) (Optimize SCMI with an appropriate choice of and
P. see Tab. 1)
6: Get labels L(A,) for batch A; and < M Ar
7: end for
8: Return trained model M and parameters 0.
In the scenarios below, we will discuss how this paradigm can provide unified view of active learning, handle aspects like standard active learning (Sec. 3.1,, rare classes and imbalance (Sec. 3.2), redundancy (Sec. 3.3) and, OOD/outlers in the unlabeled data (Sec. 3.4).
3.1 Standard Active Learning
We refer to standard active learning for ideal scenarios when there is no imbalance, redundancy or OOD data the unlabeled dataset. such cases, there is no requirement for having a query set and conditioning set. Hence, given a SCMI function (AA; P)P). we get T (A: 2)P) = f(A) by setting Q - U (the unlabeled dataset) and P 0. In nutshell, the standard diversified active learning ting can be seen as a special case of our proposed unified AL framework (Equ. (1)) by choosing Q, P above. Note that this approach is very similar and closely related to BADGE [3], where the authors

---

also choose points based on diverse gradients. Furthermore, the authors discuss the use of Determinan- tal Point Processes (DPP) [28] for sampling, and this very similar maximizing dog-determmnants. In the supplementary paper, we compare the choice of different submodular functions for AL..
3.2 Rare Classes
A very common and naturally occurring scenario is that of imbalanced data. This imbalance is because some classes or attributes are naturally more frequently occurring than others in the real- world. For example, in a self-driving car application, there may be very few images of pedestrians at night on highways, cyclists at night. Another example is medical imaging, where there are many rare yet important diseases (e.g., various forms of cancers), and it often the case that non-cancerous images are much more than compared to the cancerous ones. While such classes are rare, it is also critical to be able to perform well in these classes. The problem with running standard active learning algorithms in such a case is that they may not sample too many data points from these rare classes, and as a result, the model continues to perform poorly on these classes. In such cases, we can create a (small) held-out set R which contains data points from these rare classes, and try to encourage the AL by sampling more of these rare classes by maximizing the SMI function I R):
max AC
(2)
This setting is shown in Fig. 2(a). R contains small number of held-out examples of classes 5, 8.8 which are rare, and the AL acquisition function is Equ. (2). Note that this is exactly equivalent to maximizing the SCMI function with Q < R and P < 0 (i.e. Equ. (1) in Line 5 of A Algorithm 1). Fur- thermore, since the SMI functions naturally model query relevance and diversity, they will also try to pick a diverse set of data points which are relevant to R. Finally, we also point out that this setting was considered in [15] where they use a FISHER kernel based approach to sample data points. Note that for this setting to be realistic, it is critical that the size of this validation set very small - [15] uses a much larger validation set which is not very realistic (c.g., 200x our set, see Appendix B for more details).
3.3 Redundancy in Unlabeled Data
[a] Rare Classes
(b) Redundancy
[c) Outaf--isttribtion
Another commonplace scenario is where we are dealing with a lot of redundancy - e.g. frames sampled from a video, where subsequent frames visually similar. In such cases, existing AL algorithms tend to pick data points that are se- mantically similar to the ones selected in some earlier batch. This true even for the ofa the-art AL algorithm BADGE [3] that attempts to enforce diversity, but only in the current batch of data points and not the already selected labeled set. To illustrate this, consider the scenario in Fig. 2(b). The digits 0, are redundant in the unlabeled set, and they are already present in the labeled set C. Algorithms which just focus on diversity in the current batch could fail at ensuring diversity across batches. To mitigate inter-batch redundancy, we use SCG acquisition function and condition upon the already labeled set L:
R
O
S
O 0
OI
3
5
8
2 3
U
U
9
9
7
0 0
O
4
4 5
9 3
9
6
6
7
0
0
O 6
9
3
2
/
6
4 % 2 2
4
8
3
7
7
5
4
5 5 + 1)(A; A/ A:
4 3 3
4 6
7
4
6
6
/
3 2
A
R
0
2
J
O
7
4
f(ALL)
4IATIO)
AGPL
A:C A
Figure 2: An illustration of realistic scenarios where SIMILAR is applied with appropriate choices of query and conditioning sets: a) SIMILAR finds rare digits 5, 8 E U, by optimizing the SMI func- tion (A; R) with R containing 5, 8 as queries, b) select samples from l/ which are diverse among themselves and also diverse w.r.t those in L by optimizing ((AA(C (here, we want to avoid dig- its O. E 24 altogether because they are present in C), c) select digits (in-distribution) and avoid in 2t by optimizing alphabets where I are IDD labeled points and O (A;;
max AC
f(AIC)
(3)
Notice that this a special case of our proposed unified AL framework (Equ. (1)) since the SCG function f(A is basically SCMI function with Q < and P + L.
3.4 Out of Distribution Data
are OOD points selected so far.
In real world scenarios, we often have out-of- distribution (OOD) data or irrelevant classes in the unlabeled set. Such OOD data is not useful for
5

---

the given classification task at hand. Using an acquisition function that selects lot of OOD data points will lead a waste of labeling effort and time. This because annotators have spend time in filtering out OOD data points and discard them from the training dataset. To account for OOD data, we add additional class called "OOD" in our model. Since the goal is to improve on in-distribution classes, we ignore the prediction for the OOD class at test time. For our AL acquisition function, we use the currently labeled OOD points O as the conditioning set P. and the currently labeled in-distribution (ID) points as the query set Q. In other words, our acquisition function is to optimize:
max AC
({(A
(4)
This is illustrated in Fig. 2(c), where the labeled set consists of six examples, four of them being ID data points (set Z) and two being OOD data points (set O). In Fig. 2(c), the ID data are digits
(digit classification) and the OOD examples are alphabets. This SCMI based approach will naturally pick points "close" to the ID data while avoiding the OOD points.
Another approach for designing the acquisition function is to not explicitly condition on the OOD data points. In other words, we can just optimize the SMI function:
max
(A;
(5)
ACU.
We contrast the choices of SCMI (Equ. (4)) and SMI (Equ. (5)) functions in our experiments.
3.5 Multiple Co-occurring Realistic Scenarios
We also apply SIMILAR to datasets where more than realistic scenarios are co-occurring. As illustrated in Tab. 3, we can use the formulation of SCMI and make appropriate choices of Q and P to tackle multiple realistic scenarios.
Function ((()
Setting
Realistic Scenario
o - R. O
Rare classes + OOD)
[r R) ZOuZ'
@ + R, C
Rare classes Redundancy
@ I.P++tOu
Redundancy OOD
Table 3: Choices for Q and P for multiple co-occuring realistic scenarios
We set Q <- R and P <- O and maximize 1, RJO). Intuitively, this function would pick points close to R while avoiding the OOD points. this scenario, we can also optimize an SMI function 1 (A; R) if the data points belonging to the rare classes are not similar to the OOD data points, meaning that only searching for classes may suffice. Regardless, the SCMI approach above will further reinforce the avoidance of the OOD points.
Rare classes and OOD:
Rare classes and Redundancy: We set Q < R and + L R. Here, R is the subset of data points from the labeled set L that belong to the rare classes. Intuitively, this function would pick points close to R while avoiding points already in L - R. thereby avoiding redundant data. Just focusing R by optimizing (A; R) is also feasible option because rare classes generally not redundant. As before, the SCMI approach will only reinforce the avoidance of redundant samples in any non-rare class instances selected.
This is more challenging scenario than the ones above. We start with using the SCMI formulation for the OOD scenario, i.e., T (A; OO), where I is the set of ID samples and O is the set of OOD samples. Optimizing this function will pick diverse in-distribution samples within a batch. For selecting diverse samples across different batches, we can tackle this by using an appropriate kernel for the conditioning set. For instance, consider the FL.CMI function in Tab. 2(b). On setting P +- UI, we can rewrite the FLCMI function by splitting the penalty term as follows: Σ max (min( (max Sij, max S:₁) max(max Sij, max Sty) ()). While S is computed using cosine jEO
Redundancy and OOD:
kEWl
jEA
JEZ
jEZ
similarity, we can compute S using an exponential kernel to magnify the value of S using the exponent when i and j are very similar. This exponent is hyperparameter which can be tuned to penalize selecting redundant samples from / (denoted as 7') in Tab. 3.
3.6 Realizing Realistic Scenarios in Applications
In this section, we discuss a few insights on how these realistic scenarios can be realized. To begin with, the initial labeled set used in AL usually follows the distribution of the unlabeled set. The

---

statistics of this set can be used to identify rare classes. If the initial seed set is small, the rare classes/OOD data points can be realized after few rounds of standard AL. Until such scenarios are discovered, standard AL can be done using a diversity- acquisition function like the log determinant (LOGDET). For production-level models, they go through test deployment phase. During this phase. systematically recurring errors are often found. An example is of undetected bicycles at night in an object detector (false negatives). Such recurring failure cases can be due to rare classes in the labeled set. Moreover, we as users often know whether there are rare classes or if there is redundancy from domain knowledge. For instance, in the biomedical domain, images of cancer cells are typically rarer than ones of non-cancer cells because cancer inherently is rare disease.
3.7 Scalability and Computational Aspects of SIMILAR
The computational complexity of the different SMI functions determined by (1) the kernel computation time, and (2) the time complexity of the greedy algorithm. All functions considered here are graph based functions and require computing a kernel matrix. The LOGDET functions (LOGDET, LOGDETMI, LOGDETCG, LOGDETCMI), some Fr. functions (FI., FLVMI, FLCMI), and GC, GCMI all require the n X n similarity matrix (n [44 is the number of unla- beled points) which entails a complexity of (n²) to construct the similarity kernel. Once constructed, the complexity of the greedy algorithm for LOGDET class of functions is roughly O(Bn)) [11], while the complexity of the greedy algorithm with FL. FLVMI, and FLCMI is O(Bn²) [18, 20](B is the batch size). Different from others, FLQMI does not require computing a n X n kernel, but only a x q kernel (where q = [ is the number of query points). Correspondinglyy the complexity of the greedy algorithm with FLQMI is B3) and is linear in n. In Appendix. A, we provide a detailed summary of the complexity of different SF, SMI, SCG, and SCMI functions.
Computational Complexity:
Partition Trick: The deal with the high )n²) of the LOGDET, GC, and some of the Ft. variants (except FI.QMı), we also propose the following partitioning algorithm: We randomly split the unlabeled set U into p partitions U1 Up. and we then define the corresponding function (SF SMI, SCMI, SCG) on each of the partitions and independently optimize them. In each partition, we select B/p points. The complexity of this reduces from O to O(n² /p) and with an appropriate choice of p. we can significantly reduce the computational complexity. We use this in our ImageNet experiments (see Sec. 4.1), and observe that our approaches continue performing well while being more scalable. We provide more details on partitioning in Appendix. A.
Deep models have numerous parameters leading to very high dimensional gradients. Since our kernel matrix is computed using the cosine similarity of gradients, this becomes intractable for most models. To solve this problem, we use last-layer gradient approximation by representing data points using last layer gradients. BADGE [3], CORESET [40] and GLISTER [24] are other baselines that also use this approximation. Using this representation, we compute a pairwise cosine similarity matrix instantiate acquisition functions in SIMILAR (see lines 3,4 in Algorithm 1).
Last Layer Gradients:
4 Experimental Results
In this section, we empirically evaluate the effectiveness of SIMILAR on a wide range of scenarios like rare classes (Sec. 4.1). redundancy (Sec. 4.2) and out-of out-if-distribution (Sec. 4.3). We do so by comparing the accuracy and selections of various SCMI based acquisition functions with existing AL approaches. Using these experiments, we cover the issues with the current AL methods and show that these issues can be mitigated by using a unified implementation using SCMI with appropriate choices of query and/or conditioning sets. Although this section focuses on realistic scenarios, we also study SIMILAR in standard active learning setting and show that it performs at par with current AL methods (see Appendix. C). Furthermore, we present some experiments on a real-world medical dataset in Ap- pendix. H and some experiments on multiple co-occurrngg realistic scenarios (Sec. 3.5) in Appendix. I.
Baselines in all scenarios: We compare SCMI based functions against several methods. Particularly, we compare against: (1) three uncertainty based AL algorithms: i)ENTROPY: Selects the top B data points with the highest entropy [41], ii) MARGIN: Select the bottom B data points that have the least difference in the confidence of first and the second most probable labels [37], LiASEE CONF: Select B samples with the smallest predicted class probability [44], (2) state-of-the-ar diversity based algorithms: iv) BADGE [3] v) GL.ITERR [24] vi) CORESET [40] which are all discussed in section Sec. 1.1, and, 3) RANDOM: Select B samples randomly. Additionally. in the rare classes scenario, we compare against FISHER [15] which is also discussed in Sec. 1.1.
7

---

LOGDETMI PLVMI
DIV-GCMI - PISHIER-LV BADGE
ENTROPY CORESET LEAST-CONT RANDOM
MARGIN
FLQMI GCMI
PISHTER
GLISTER
Auteewy un Bare Clesse
(1) Rare Classes Accuracy
(b)IIFAR-10: Overall Accuracy
Crmmiileivv wittar Sumplos
(c)CIFAR-10: # Raro Class Samples
Ovegnill Arcuney
125 1230
100
1105
1250
135 250 573 D0D
625
750
573
1800 1125 1250
135
250
575
500
615
7M0
573
1800
1125
1230
135
250
575
540
0.5
750
875
1000
1125
(d)MNIST: Rare Classes Accuracy
(e)MNIST: Overall Accuracy
Cuunlaaive Stumpee
(F)NNIST: Rarn Class Sumples
Accuray
300
Areurayy
60
100
125 250 375 500 eds
750 als 1000 riss 1260
125
250
355
500
025
750
als
1000 1:55 1290
125
230
375
500
625
730
675
1000
1125
(g]amageNet: Rare Clusses Accuracy
[b)anageNet:: Overall Accuracy
(IImageNet: # Rare Class Samples
10
18
30
300000
30000
102000 127000
152000
1971900 2020000
2271030
102500
1220000
152000
177950 202000 2271000
Lelocked Sei Steee
Enboked Sei Stee
Lebecaa Sct Soe
Figure 3: Active Learning with rare classes on CIFAR-10 (top row), MNIST (middle row), and ImageNet (bottom row). Left side plots (a.d.g) are rare class accuracies, center plots (b.e.h) are overall test accuracies, right plots (c,f.i) are number of rare class samples selected. The SMI functions (specifically LOGDETMI, FLQMI) outperform other baselines by more than 10% on the rare classes.
We apply our framework to CIFAR-10 [27] and MNIST [30] classification tasks. Additionally, we also evaluate our method on down sampled 32x 32 ImageNet-2011 [38] for the rare classes setting (Sec. 4.1). Due to the lack of test split on ImageNet, we used the validation split for evaluation. In the sections below, we discuss the individual splits for f. L4 R. I, and O in each realistic scenario. To ensure that all the selection algorithms that we are studying are given fair and equal treatment across all realistic scenarios, we use a common training pro- cedure and hyperparameters. We use standard augmentation techniques like random crop, horizontal flip followed by data normalization except for MNIST which does not use horizontal flip to preserve labels. For training, we use an SGD optimizer with an initial learning rate of 0.01, the momentum of 0.9, and a weight decay of Se-4. We decay the learning rate using cosine annealing [31] for each epoch. On all datasets except MNIST, we train a ResNet18 [17] model, while on MNIST we train a LeNet [29] model. For all the experiments in particular scenario (rare classes, redundancy and OOD), we start with identical initial model M and initial labeled set D. We reinitialize the model parameters at the beginning of every selection round using Xavier initialization and train the model until either the training accuracy reaches 99% the epoch count reaches 150. We run each experiment 3 CIFAR- 10 and MNIST and xx on ImageNet and provide error bars (std deviation). All experiments were run on a V100 GPU, For more details on the experimental setup, baselines, and datasets see Appendix. B.
Datasets, model architecture and experimental setup:
4.1 Rare Classes
Following [15, 24], we simulate these rare classes by creating a class imbalance. We initialize the batch active learning experiments by creating a custom dataset which is subset of the full dataset with the same marginal distribution. Given that C consists of data points from the imbalanced classes and D consists of data points from the balanced classes, we create initial labeled set L such that |Dxl = p|Cz\ and an unlabeled set |Dxy| = plCexl, I. where P is the imbalance factor. We use small and clean validation/query set R containing data points from the imbalanced classes (== 3 data points per imbalanced class). We create an imbalance CIFAR- 10 using 5 random classes, p= = 10 and for MNIST we create an imbalance using the same classes as in [15] (5 (5...9) and use p= = 20. For both datasets: ICe |Dzl 125, |Cyl + |Dul 16.5K, B 125 (AL batch size) and, [R] = 25 (size of the held out rare instances). For MNIST, we also present the results for B = 25 and p 100 in the supplementary. On ImageNet, we randomly select 500 classes out
Custom dataset:
8

---

LOCDETCG FLCC- CLISTER BADGE CORESET ENTROPY MARGIN RANDOM
(a) Overall Accuracy
(b)MNIST: Overall Accuracy
40
Samplos
(e)CTPAR-
Unique Samples Selected
2000
17.00
42
2000
20
13.17
67.08 AM
Number
4500
5000
410
50p
300 1010
1300 2003 2000 2014 53AN)
4002 E5o0 5000
30
100
150
250
204
300
350
400
454
300
1000 520
1500 2000 2000 2000 3000 4000 4500
Figure 4: Active Learning under 10x redundancy for CIFAR-10 and MNIST. The CG functions (LOGDETCG, FI.CG) pick more unique points and outperform existing algorithms including BADGE.
of 1000 classes for imbalance and p = 5 such that |C2] + |Del = 102K, Cal + |Dul = 664K' B 25K and, R 2.5K. These data splits are chosen to simulate a low initial accuracy on the rare classes and at the same time maintain the imbalance factor in the labeled and unlabeled datasets.
The results are shown in Fig. 3. We observe that SMI based functions not only consistently outperform uncertainty based methods (ENTROPY, LEAST-CONE and MARGIN) but also all the state-of-the-art diversity based methods (BADGE, GLISTER, CORESET) by ~ 5 - 10% in terms of overall accuracy and R 10 18% in terms of average accuracy on rare classes (see Fig. 3a, 3d, 3g). The reason for the same can be seen in Fig. 3c, 3f, 3i which illustrates that they fail to pick an adequate number of examples from the rare classes. Evidently, FLQMI and LOGDETMI which balance between diversity and relevance perform better than GCMI which only models relevance. Furthermore, DIv-GCMI which is a linear combination of GCMI and a diversity term performs consistently worse, which suggest that naive combination of the two may not be as effective. This suggests the need of SMI based acquisitions functions (Equ. (2)) with richer modeling capabilities like FI.QMI and LOGDETMI within SIMILAR. Furthermore, all SMI based functions also outperform the FISHER kernel based method when the validation set is small and realistic, i.e., R = 25. Since, [15) use very large validation set in their experiments, we try their method FISHER-LY -LV with a 40x larger validation set of size 1000 (which is not practical) and observe a comparable performance with the SMI functions which use a small validation set. Furthermore, we see that FISHER-LV actually picks significantly larger number of rare class instances in MNIST, but yet is comparable in performance of FLQMI and LOGDETMI. This suggests that both these methods select higher quality and diverse rare class instances. We observe that the Gc SMI variants( GCMI and DIV-GCMI) do not perform well on MNIST classification. Finally, we point out in the case of ImageNet, FLQMI performs the best and outperforms FLVMI and LOGDETMI this is because we do not need to do the partition trick for FLQMI since is already linear in time complexity. For FLVMI and LOGDETMI, we set the number of partitions = 50 for ImageNet. Finally, we do a pairwise t-test to compare the performance of the algorithms (Appendix. D) and observe that the SMI functions (and particularly FLVMI and LOGDETMI) statistically significantly outperform all AL baselines.
Results:
4.2 Redundancy
To simulate a realistic redundancy scenario we create a custom dataset by duplicat- ing 20% of the unlabeled dataset 10x. For CIFAR-10, the number of unique points in the unlabeled set U 5K, the initial labeled set |C|| 500, B 500, whereas for MNIST Z 500, | 50
Custom dataset:
and B- = 50. For MNIST, we also present the results for 5x and 20x in the Appendix. E.
SCG vs Baselines: As expected, the diversity and uncertainty based methods outperform random. Importantly, we observe that the SCG functions (FL.CG and LOGDETCG) significantly outperform all baselines by x 3 - 5% towards the end as the conditioning gets stronger with increase in L (see Fig. 4a. 4b). This implies that simply relying on model parameters for diversity and/or uncertainty is not sufficient and that conditioning on the updated labeled set L (Equ. (3)) is required in batch active learning. In Fig. 4c we show that SCG based acquisition functions select significantly more unique data points than other baselines. We also perform a pairwise t-test (Appendix. E), to prove that the SCG functions consistently and statistically significantly outperform BADGE and other baselines.
4.3 DutttrrDisti
Custom dataset: We simulated a scenario where we convert the classification problem in CIFAR-10 and MNIST to 8-class classification, where the first 8 classes represent the set IF of in-distribution

---

LOGDETCMI LOCDETMI
FLVMI
DIV-GCMI
BADGE CORESET
ENTROPY MARGIN
RANDOM
FLCMI
FLQMI
GCMI GLISTER
(a)CIFAR-10: SCMI vs Baselines
(1)CIFAR-10: SCMI
teJOOD: Variance Comparison
78
76
2,09
Acorsey
Sverge Veriowe
7451
72.17
00
1000
1000 1050 2300
2330 2030
2634 2500 3300 3000 3630
1000 1850 2300
2030 2000
2650
3100
3500 3000 Sobu
Labeled
(dJMNIST: SCMI Baselines
(e)MNIST: SCMI SMI
(f)MNIST: #ID
Nouber ID Sumples
Accunccy
Accuracy
110
12
70
00
100 120 140 160
100
200
220
100 120 140
100
180
200
320
40
60
500 120
3.00
100
380
200
Figure 5: Active Learning with OOD data in unlabeled set. Top row: CIFAR-10 results for (a) SCMI vs Baselines, (b) SCMI vs SMI, and (c) variance comparison of different baselines, bottom row: MNIST results for (d) SCMI vs Baselines, (c) SCMI vs SMI, and (f) Number of ID points selected. We see that, i) the SCMI functions consistently outperform the baselines by 5% - 10%, ii) SCMI functions outperform the corresponding SMI functions for later rounds, and (ii) SCMI functions have the least variance compared to the rest, showing that they are more robust in performance.
(II) data points and the last 2 represent the set Op of cittoffddisrrbbutionnoOO initial labeled set L consists only of ID points, i.e. OF L 0. The unlabeled set is simulated to reflect a realistic and somewhat extreme setting where the unlabeled ID data points [Zpl is much smaller than the unlabeled OOD data points Orl. Additionally, we also assume we have very small validation set of ID points Zy. For CIFAR- 10: Ic = 1.6K, [Z,| = 4K, | = 10K, [Zy = 40, B 250 whereas for MNIST which is relatively simpler task, we use a smaller initial labeled sets and keep the unlabeled sets of the same size: [ = 40, [Ip] = 400, |0,l = 10K, [Tv| = 16, B- = 20. Recall that our algorithm uses ID set I (initialized to Iv) and OOD set O which we build as follows, Every time our selection approach selects a set A, we update I = I U (An IP) and i.e. we augment the ID and OOD points in A to the sets I and O respectively.
data points. The
O = O
Since we care about the predictive performance of the ID classes, we report the ID classes accuracy. We see that SCMI based acquisition functions significantly outperform existing AL approaches by ~ 5 - 10% (see Fig. 5a, 5d). We also observe that existing acquisition functions have a high variance, which is undesirable in real-world deployment scenarios where deep models are being continuously developed. Our SCMI based acquisition functions (LOGdETCMI and FLCMI) show the lowest variance in training (see Fig. 5c). This reinforces the need of having framework like SIMILAR that facilitates query and conditioning sets.
SCMI vs Baselines:
We compare SCMI functions against SMI functions to study the effect of conditioning and observe that the SCMI functions are comparable to the SMI functions initially but in the later selection rounds of active learning, the SCMI functions consistently outperform SMI functions. In particular, we see an improvement of 2 - 3% as the conditioning becomes stronger (see Fig. 5b, 5e). We also observe the SCMI tends to select more IDD points than SMI and other baselines (see Fig. 5f), and SCMI functions have a lower variance overall compared to even the SMI functions (Fig. 5c).
SCMI vs SMI:
5 Conclusion
In this paper, we proposed a unified active learning framework SIMILAR using the submodular information functions. We showed the applicability of the framework in three realistic scenarios for active learning, namely rare classes, redundancy, and out of distribution data. In each case, observed that the functions in SIMILAR significantly outperform existing baselines in each of these tasks. Our real-world experiments on MNIST, CIFAR-10, and ImageNet show that many of the SIM functions (specifically the LOGDET and FL variants) yield 5% 18% gain compared to existing baselines, particularly in the rare class scenario and RJ 5% 10% OOD scenarios. The main limitations of our work is the dependence on good representations compute similarity. A potential negative socictal impact of this work is the use of SIMILAR to perpetuate certain biases through a malicious use of the query and conditioning set. We discuss this in more detail in Appendix. G.
10

---

Acknowledgments and Disclosure of Funding
This work is supported by the National Science Foundation under Grant No. IIS-2106937, a startup grant from UT Dallas, and by Google and Adobe research award. Any opinions, findings, and conclusions recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation, Google or Adobe.
References
[1] Sami Abu-EI-Haija, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakrishnan Varadarajan, and Sudheendra Vijayanarasimhan. Youtube-8m A large-scale video classification benchmark. arXiv preprint arXiv: urXiv:1609.000 (609.0867755 2016.
[2] David Arthur and Sergei Vassilvitskii. k-means++: the advantages of careful seeding. In SODA '07: Proceedings of the eighteenth annual ACM-SIAM symposium on Discrete algorithms, pages 1027-1035, Philadelphia, PA, USA, 2007. Society for Industrial and Applied Mathematics. ISBN 978-0-8987 167776-2224 24-5..5.
[3] Jordan T Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agar- wal. Deep batch active learning by diverse, uncertain gradient lower bounds. arXiv preprint arXiv:1906.080 03671, 2019.
[4] Francis Bach. Learning with submodular functions: A convex optimization perspective. arXiv preprint 1eXiv:///1.64533: 2011.
[5] Francis Bach. Submodular functions: from discrete to continuous domains. Matherattiaal Programming, 175(1):419-459, 2019.
[6] Christopher Berlind and Ruth Urner. Active nearest neighbors in changing environments. In International Conference on Machine Learning, pages 1870-1879. PMLR, 2015.
[7] Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency, pages 77-91. PMLR, 2018.
[8] Holger Caesar, Varun Bankiti, Alex Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 11621-11631, 2020.
[9] Colin Campbell, Nello Cristianini, Alex Smola, et al. Query learning with large margin classifiers. In ICML, volume 20, page 0, 2000.
[10] Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Kochn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint ar 131.330050 2013.
[11] Laming Chen, Guoxin Zhang, and Hanning Zhou. Fast greedy map inference for determinantal point process improve recommendation diversity. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 5627-5638, 2018.
[12] Shai Fine, Ran Bacahdah and Eli Shamir. Query by committee, linear separation and random walks. Theoretical Computer Science, 284(1):25-4) 2002.
[13] Yoav Freund, Sebastian Seung, Eli Shamir, and Naftali Tishby. Selective sampling using the query by committee algorithm. Machine learning, 1332)1133-168, 1997.
[14] Satoru Fujishige. Submodular functions and optimization. Elsevier, 2005.
[15] Denis Gudovskiy, Alec Hodgkinson, Takuya Yamaguchi, and Sotaro Tsukizawa. Deep active learning for biased datasets via fisher kernel self-supervision. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 9041-9049, 2020.
[16] Anupam Gupta and Roie Levin. The online submodular cover problem. In ACM-SIAM Symposium Discrete Algorithms, 2020.
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 770-778, 2016.
11

---

[18] Rishabh lyer and Jeffrey Bilmes. A memoization framework for scaling submodular optimiza- tion large scale problems. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 2340-2349. PMLR, 2019.
[19] Rishabh Iyer, Ninad Khargoankar, Jeff Bilmes, and Himanshu Asanani. Submodular combina- torial information measures with applications in machine learning. In Algorithmic Learning Theory, pages 722-754. PMLR, 2021.
[20] Rishabh Krishnan Iyer. Submodular optimization and machine learning: Theoretical results, unifying and scalable algorithms, and applicationn. PhD thesis, 2015.
[21] Kimmo Karkkainen and Jungseock Joo. Fairface: Face attribute dataset for balanced race, gender, and age for bias measurement and mitigation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1548-1558, 2021. 2021.
[22] Vishal Kaushal, Rishabh Iyer, Suraj Kothawade, Rohan Mahadev, Khoshrav Doctor, and Ganesh Ramakrishnan. Learning from less data: A unified data subset selection and active learning framework for computer vision. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1289-1299. IEEE, 2019.
[23] Vishal Kaushal, Suraj Kothawade, Ganesh Ramakrishnan, Jeff Bilmes, and Rishabh lyer. Prism: unified framework of parameterized submodular information measures for targeted data subset selection and summarization. arXiv preprint arXiv:2103.00/22 2021.
[24] Krishnateja Killamsetty, Durga Sivasubramanian, Ganesh Ramakrishnan, and Rishabh Iyer. Glister: Generalization based data subset selection for efficient and robust learning. arXiv
preprint arXiv:2012.106.00 2020.
[25] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv: 14, 2013.
[26] Andreas Kirsch, Joost Van Amersfoort, and Yarin Gal. Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning, arXiv preprint arXiv:1906.081 2019.
[27] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[28] Alex Kulesza and Ben Taskar. Determinantal point processes for machine learning. arXiv preprint arXiv: /207. 6083, 2012.
[29] Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4):541-551, 1989.
[30] Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. at&t labs, 2010.
[31] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv: 160.,,1660.0000 033983, 2016.
[32] Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adver- sarial autoencoders. arXiv preprint 1.1.5:151..05644, 2015.
[33] Baharan Mirzasoleiman. Ashwinkumar Badanidiyuru, Amin Karbasi, Jan Vondrák, and An- dreas Krause. Lazier than lazy greedy. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
[34] George L Nemhauser, Laurence A Wolsey, and Marshall L Fisher. An analysis of approximations for maximizing submodular set functions-i. Mathematical programming, 14(1):265-294, 1978.
[35] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
[36] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury. Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, F. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019. URL www.roorrranc-deeeeeeeeeaannngn. 1iii-rrbrrrr.. 2015-pytoree ch an www-iiipaaieeeeeeeeeee style high www.orewnceeeeeee pdf.
12

---

[37] Dan Roth and Kevin Small. Margin based active learning for structured output spaces. In European Conference on Machine Learning, pages 413-424. Springer, 2006.
[38] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 115(3)2111-553 2015.
[39] Greg Schohn and David Cohn. Less more: Active learning with support vector machines. In
ICML, volume 2, page 6. Citeseer, 2000.
[40] Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv: 1en:::00800040 00489, 2017.
[41] Burr Settles, Active learning literature survey. 2009.
[42] Samarth Sinha, Sayna Ebrahimi, and Trevor Darrell. Variational adversarial active learning. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 5972-5981, 2019.
[43] Ehsan Tohidi, Rouhollah Amiri, Mario Coutino, David Gesbert, Geert Leus, and Amin Karbasi. Submodularity in action: From machine learning to signal processing applications. IEFE Signal Processing Magazine, (7(5):120-133) 2020.
[44] Dan Wang and Yi Shang. A new active labeling method for deep learning. In 2014 International joint conference neural networks (IJCNN), pages 1921199.. IEEE, 2014.
[45] Kai Wei, Rishabh Iyer, and Jeff Bilmes. Submodularity in data subset selection and active learning. In International Conference Machine Learning, pages 1954-1963. PMLR, 2015,
[46] Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. arXiv preprint arXiv: 1509001600 2015.
[47] Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pages 19-27, 2015.
13


Reviews:
[{"review_id": "qWfktrqzTJB", "reviewer": "Reviewer_hRHU", "summary": "The paper introduces a novel active learning framework (called SIMILAR) that leverages Submodular Information Measures to deal with both \"mainstream\" active learning scenarios and situations in which there are rare classes, redundancy, or out-of-domain classes.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors propose a novel approach to an important problem: active learning in the real world. The paper is well written & reasonably easy to follow, and it makes a strong case for the proposed approach. The contribution appears to be original, and the authors make a compelling case for the use of Submodular Information Measures to deal with rare classes, data redundancy, and out-of-domain classes. \n\nHowever, the paper suffers from two main weaknesses:\n- while it claims to be motivated by applying active learning in the real world, the datasets used to illustrate its ability to handle in real world scenarios (i.e., rare classes, data redundancy, and out-of-domain classes) are synthetically derived from existing domains that do NOT exhibit these properties. Even though the experimental results are encouraging, the paper would greatly benefit by showcasing at least one organic, real-world domain that has these properties\n-  part of the challenge in dealing with rare classes & out of domain (OOD) classes (and, in part, data redundancies) is to have an algorithm that can automatically detect the rare or OOD classes, such as [Muslea et al, 2006] below. Requiring the user to discover and provide examples from rare or OOD classes represents a challenge in itself. While this is easily done in synthetic domains, it if far from obvious that this assumption  will hold in the real-world, especially if it is to be done a reasonable cost/effort. This is another reason while the paper would benefit from a real-world domain. \n\nMuslea, I., Minton, S., & Knoblock, C. A. (2006). Active learning with multiple views. Journal of Artificial Intelligence Research, 27, 203-233.\n\nOTHER COMMENTS:\n- line 247: please elaborate & quantify \"more scalable.\" In this reviewer's opinion, the scalability issue deserves a section of its own, rather than being hidden under a paragraph called \"Partition Trick,\" especially given the scalability claim in the abstract (line 9)\n- line 343: does it matter which classes are chose as out of distribution? it probably does depend on how similar an OOD class is wrt an In-Distribution one ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "none", "review_text": "", "extra_scores": {"ethical_concerns": "none", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "npxfKpD6ak6", "reviewer": "Reviewer_Pxy3", "summary": "The paper introduces a general framework for batch-based deep active learning\nthat (1) leverages submodularity of the acquisition function to select a\nbatch in almost linear time, (2) defines a modular acquisition function\nthat can support cases like selecting instances different from known OOD\nsamples and favoring instances similar to known rare samples, (3) supports\na variety of submodular functions defined over a gradient-based kernel.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper flows very well, all concepts are presented clearly.  The\ncontribution is clearly on-topic, simple and elegant.  I appreciate having a single formulation\nthat covers several challenging use-cases.\n\n\nThe related work does a good job at positioning the paper method against the SotA.  The only missing\npaper that I was expecting to see cited is:\n\n  Shui et al. \"Deep Active Learning: Unified and Principled Method for Query and Training\".\n\nas they give the (to the best of my knowledge) best theoretical justification / foundation for batch-based deep AL to\ndate.  Not a huge deal.\n\nThe design of the experiments is reasonable.  The results show clear improvements over the competitors in most\nsettings.\n\nGenerally speaking, I enjoyed reading this paper.  I only have a few remarks / questions:\n\n1. The proposed method is not really justified from a learning theoretical standpoint, in the sense that the specific choice of submodular function is not derived from first principles / clear desiderata.  I realize that SIMILAR is meant as a framework, and therefore it can fit a variety of different submodular functions without requiring modifications, but it is also true that different submodular acquisition functions may have different properties.  This is exactly what the empirical evaluation shows.  Not a huge deal.\n\n2. One downside of the experiments is that the data sets are manipulated to introduce the desired conditions, e.g., rare classes.  It would have been more convincing to use data sets in which OOD data and rare classes occur naturally.  It is also not ideal that all experiments are done on image data only.\n\n3. My only real complaint is that the use-cases are not entirely realistic.  For AL under OOD, OOD data is often not available - this is precisely the problem - and cannot be used to guide the batch selection strategy.  A more thorough discussion of the practical applicability of the proposed method would have been appreciated.\n\n(As a side note: I see that in many papers BADGE is considered SotA - however in practice it tends to underperform compared to simpler methods.  The experiments in the BADGE paper are not particularly convincing either.  I wonder why people keep comparing to it.  Of course, I don't want to single this paper out;  plenty of papers out there do the same.)", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "A better description of the practical applicability of SIMILAR would have been appreciated.\n\nThe broader impact does a good job at covering potential negative effects.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "iQwUhVxDIOs", "reviewer": "Reviewer_gPM8", "summary": "The paper proposes a new active learning framework SIMILAR based on submodular information measures (SIM). The framework aims to solve the problem of active learning (AL) raised in realistic scenarios including rare classes, redundancy and out of distribution. ", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths:\nThe proposed AL framework, SIMILAR can solve multiple realistic scenarios of active learning problem with a unified acquisition function form.     \nComputation of the acquisition function is reasonable.    \nThe experiments on real-world high dimensional datasets demonstrate the effectiveness of the proposed framework.     \n\n### Weaknesses:\nThe paper needs a more detailed description of the active learning problem setting for the readers, for example this paper focuses on the \"pool-based AL with batch queries...\". \n\nAlthough SIMILAR can applied on multiple scenarios, it seems that the framework still lacks flexibility for more general and realistic scenarios. \n* For example, in the rare classes scenario, the acquisition function is defined as the SMI between $A$ and $R$, focusing only on the rare classes, so the SIMILAR is only applicable to the case that the rare classes are much more important than the regular classes, there is no parameter to balance the tradeoff between them. \n* Another example is that in the redundancy scenario, the labeled set $L$ is set as the conditioning set, while in the OOD scenario, the labeled in-distribution set $I$ is set as the query set, so SIMILAR cannot be applied on scenarios with both Redundancy and OOD. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the limitations and societal impact.\n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "EC9tL8m4sn6", "reviewer": "Reviewer_YdBA", "summary": "This paper is in the field of Active learning, which has proven to be useful for minimizing labeling costs by selecting the most informative samples from unlabeled datasets, which are then to be labeled by a human in the loop. The goal is to achieve high accuracy of the trained model, while minimising the number of points to be labeled by the human in the loop. However, as the authors say, many existing state of the art active learning methods are trained on datasets which are not very realistic, and they do not perform well in many common real-life situations such as imbalance or rare classes, out-of-distribution data in the unlabeled set, and redundancy. Given the size of these datasets, cleaning them manually is not realistic. The authors propose an active learning framework which they call SIMILAR, which uses submodular information measures as acquisition functions. They argue that SIMILAR performs well on the realistic scenarios mentioned previously, and empirically demonstrate that SIMILAR outperforms previous state of the art methods in these scenarios. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think this paper is well written, addresses important practical problems and has interesting results. Some comments:\n\n1) It seems like out of the three cases of imbalance/rare classes, out-of-distribution data in the unlabeled set, and redundancy, the framework can only deal with solving one of these issues per round/batch. Because it can handle only one situation at a time, the framework may not perform well for datasets facing multiple of these issues.\n\n2) The performance of the algorithm depends crucially on and is very sensitive to the choice of the similarity function. The choice of similarity function needs to be better justified, and I suspect finding an ‘optimal’ similarity function will be a very hard problem to solve. A bad choice of similarity function could lead to bad performance, and maybe even lead to discrimination or unfairness. \n\n3) It seems that one would often need to know the unlabeled dataset pretty well before using this framework. For example, in the task of ‘creating  a (small) held-out set R which contains data points from rare classes’ before running the algorithm, which may not be an easy task. \n\n4) It seems like a user of this framework would need to know beforehand what issues the unlabeled dataset faces (for eg., rare classes, OOD, redundancy), which may not always be a reasonable assumption. \n\nIn addition, a more detailed explanation of choices made in designing the framework and why they lead to good performance would be nice.\n\nEdit (after author response): Thanks for the detailed comments. After going through them, I have changed my score accordingly.  \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The work proposes a novel active learning framework based on submodular information gain maeasures. The framework is well motivated and explained, and then empirically evaluated.

This submission was well appreciated by the reviewers. The authors are encouraged to take the reviewer's comments and their responses into account when preparing their final version for publication.
Decision: Accept (Poster)