SIMILAR: Submodular Information Measures Based Active Learning In Realistic Scenarios
Authors: Suraj Nandkishor Kothawade
Nathan Alexander Beck
Krishnateja Killamsetty
Rishabh K Iyer
Venue: NeurIPS 2021 Poster (2021)
Keywords: Submodularity
Active Learning
Information Measures
Realistic
Redundancy
Rare Classes
Out of distribution
Robust

Abstract: Active  learning  has  proven  to  be  useful  for  minimizing  labeling  costs  by selecting  the  most  informative  samples. However,  existing  active  learning methods do not work well in realistic scenarios such as imbalance or rare classes,out-of-distribution data in the unlabeled set, and redundancy.  In this work, we propose SIMILAR (Submodular Information Measures based actIve LeARning), a unified active learning framework using recently proposed submodular information measures (SIM) as acquisition functions. We argue that SIMILAR not only works in standard active learning but also easily extends to the realistic settings considered above and acts as a one-stop solution for active learning that is scalable to large real-world datasets. Empirically, we show that SIMILAR significantly outperforms existing active learning algorithms by as much as ~5%−18%in the case of rare classes and ~5%−10%in the case of out-of-distribution data on several image classification tasks like CIFAR-10, MNIST, and ImageNet.

SIMILAR: Submodular Information Measures Based Active Learning In Realistic Scenarios
Suraj Kothawade University of Texas at Dallas suraj kothawade@utdallas edu
Nathan Beck University of Texas at Dallas nathan. beckkudaalaa edu
Krishnateja Killamsetty University Texas at Dallas krishnate killamsttty@uttallllla edu
Rishabh Iyer
University Texas at Dallas rishabh. Ayeerutdallas edu
Abstract
Active learning has proven to useful for minimizing labeling costs by selecting the most informative samples. However, existing active learning methods do not work well in realistic scenarios such as imbalance or rare classes, out-of-distribution data in the unlabeled set, and redundancy. In this work, we propose SIMILAR (Submodular Information Measures based actIve LeA Rning, a unified active learning framework using recently proposed submodular information measures (SIM) as acquisition functions. We argue that SIMILAR not only works in standard active learning but also easily extends the realistic settings considered above and acts real-worid datasets. Empirically, we show that SIMILAR significantly outperforms existing active learning algorithms by as much as R 5% 18% in the case of rare classes and x 5% 10% in the case of out- diitt--diittitttion data several image classification tasks like CIFAR-10. MNIST, and ImageNet. SIMILAR is available part of the DISTIL toolkit: https: // //gi/ttaa tighuu....... cow wom/decileeeeeeooeeoeeeeeeeeeeee team/ diittid
one-stop solution for active learning that is scalable to large
Introduction
Deep neural networks (DNNs) have had a lot of success in a wide variety of domains. However, they require large labeled datasets which are
9
6
7
0
0 0
0
4
4
5
9
often taxing, time-consuming, and expensive to
9
6
7
0
0
9
3
4
3
2
obtain. Active learning (AL) [12, 13, 39, 3, 9] is promising approach to solve this problem. It aims to select the most informative data points from an unlabeled dataset to be labeled in an adaptive manner with human in the loop. The
/
6
4
4
8
3 6
6
7
7
5
4 6
O
8
4
4
7
4
6
/
2
3
3
R
B
F
goal of AL to achieve maximum accuracy of the model while minimizing the number of data points required to labeled.
2
3
2
O
& 7 4
Classes
Redundancy
[c] Out-of-distribution
Figure 1: Motivating scenarios for realistic active
Current AL methods have been tested rel- atively simple, clean, and balanced datasets. However, real-world datasets are not clean and have number of characteristics that makes learning from them challenging [10, 46, 47, 38, 1, 8J. Firstly, these real-world datasets are im- balanced, and some classes very (e.g., Fig (a)). Examples of this imbalance medical
learning: (a) rare classes: digits 5 and 8 are rare: (b) redundancy: digits 0 and are redundant; (c) -iitt---istriutiinn (OOD): letters A, R, B. digit classification.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).

---

imaging domains where the cancerous images are rare. Secondly, real-world data has a lot of redun- dancy (e.g., Fig I(b)). This redundancy more prominent in datasets that are created by sampling frames from videos (e.g., footage from a car driving on a freeway or surveillance camera footage). Thirdly, it is common to have out-of- of-dissribbution (OOD) (e.g., Fig I()) data, where some part of the unlabeled data is not of concern to the task at hand. Given the amount of unlabeled data, it is not realistic to assume that these datasets can be cleaned manually; hence, it is the need of the hour to have active learning methods that are robust to such scenarios. We show that current AL. approaches (including the statc-of-the tatte-of-the-art approach BADGE [3]) do not work well in the presence of the dataset biases described above. In this work, we address the following question: Can machine learning model be trained using a single unified active learning framework that works for a broad spectrum of realistic scenarios? As solution, propose SIMILAR', unified active learning framework which enables active learning for many realistic scenarios like rare classes, out-of -iut---diitriuttion (OOD) data, and redundancy.
1.1 Related Work
Active learning has enabled efficient training of complex deep neural networks by decreasing labeling costs. The most commonly used approach is to select the most uncertain items. Examples of uncertainty strategies include ENTROPY [41], LEAST CONFIDENCE [44], and MARGIN [37]. One challenge of this approach is that all the samples within a batch can be potentially similar even though they are uncertain. To overcome this problem in batch active learning, many recent works have attempted to select diverse yet informative data points. [45, 22] propose simple approach: Filter set of points using uncertainty sampling and then select a diverse subset from the filtered set. [40] propose CORESET, which forms core-sets using greedy k-center clustering while maintaining the geometric arrangement. BADGE [3], another recent approach, proposes to select data points corresponding to nigh-magnitude, diverse hypothesized gradients by using K-MEANS++ [2] initialization to distance from previously selected data points in the batch. Most existing AL approaches fail to ensure diversity across AL selection rounds and do not perform as well when there is a lot of redundancy. Sinha al. [42] used a variational autoencoder (VAE) [25] learn a feature space and an adversarial network [32] to distinguish between labeled and unlabeled data points. However, their approach is computationally expensive and requires extensive hyperparameter tuning. Similarly, BATCUB ALD [26] does not scale to larger batch sizes since their method would need large number of Monte Carlo dropout samples to obtain a significant mutual information. Such limitations reduce the scope of applying these methods realistic settings.
Closely related to our work are two recently proposed works. The first is GLISTER- BAISTER-ACTOO ACTIVE [24], which formulates the AL acquisition function by maximizing the log-likelihood on held-out validation set. This validation set could consist of examples from the rare classes or in in-distribution examples. The second approach is the work of Gudovskiy et al. [15], who study AL for biased datasets using a self-supervised FISHER kernel and pseudo-label estimators. They address this problem by explicitly minimizing the KI. divergence between training and validation sets via maximizing the FISHER kernel. Although their method shows promising results, they make multiple unrealistic assumptions: a) They use large labeled validation set, and b) they use feature representations from model pretrained using unsupervised learning on a balanced unlabeled dataset. In this work, we compare against both ALISTER-ACTIV [24] and FISHER [15] approaches in the more realistic setting of a small held-out validation set (smaller than the seed labeled set) and an imbalanced unlabeled set. Another work proposed discrete optimization method for k-NN-type algorithms in the domain shift setting [6]. However, their approach is limited to A--NS.
This work utilizes submodular information measures (SIM) by [19] and their extensions by [23]. SIMs encompass submodular conditional mutual information (SCMI), which can then be used to derive submodular mutual information (SMI); submodular conditional gain (SCG); and submodular functions (SF). We discuss these functions in detail in Sec. 2. [23] also studies these functions on the closely related problem of targeted data selection.
1.2 Our Contributions
The following are our main contributions: 1) Given the limitations of existing approaches in handling active learning in the real world, we propose SIMILAR (Sec. 3), unified active learning framework that can serve as a comprehensive solution to multiple realistic scenarios. 2) We treat SIM as a
SSbmodular Information Measures based active LeARning

---

common umbrella for realistic active learning and study the effect of different function instantiations offered under SIM for various realistic scenarios. 3) SIMILAR not only handles standard active learning but also extends to a wide range of settings which appear in the real world such as rare classes, out-of distribbitittiinnn (OOD) data, and datasets with lot of redundancy. Finally, 4) we empirically demonstrate the effectiveness of SMI-based measures for image classification (Sec. 4) in number of realistic data settings including imbalanced, out-of ditribbbbii,,,nnn, and redundant data. Specifically, in the case of imhalanced and OOD data, we show that SIMILARR achieves improvements of more than 5 to 10% on several image classification datasets.
2 Background
In this section, we enumerate the different submodular functions that are covered under SIM and the relationships between them.
We let denote the unlabeled set of n data points L/ = (1,2,11,33.00 n2 and a set function f 2 + TR. Formally, a function f is submodular [14] if for x E U. -( 2 ((BU)) - f(B), A C B C U and x & B. For a set A C U, [(A) provides a real-valued score for A. In the context of batch active learning, this is the score of an acquisition function f on batch A. Submodularity is particularly appealing because it naturally occurs in real world applications [43, 4, 5, 20] and also admits constant factor 1 [34] for cardinality constraint maximization. Additionally, variants of the greedy algorithm maximize a submodular function in near-linear time [33].
Submodular Functions.
Submodular Mutual Information (SMI. Given sets A, Q C U, the SMI [16, 19] is defined as (A; Q) = f(A) + f(2) f(A U Q). Intuitively, SMI models the similarity between Q and A, and maximizing SMI will select points similar to Q while being diverse. Q here is the query set.
Submodular Conditional Gain (SCG). Given sets A, C U, the SCG [(A(P) is the gain in function value by adding A to P. Thus, [(AAP)) ((UUP) - ((P) [19]. Intuitively, SCG models how different A is from P. and maximizing SCG functions will select data points not similar to the points in P while being diverse. We refer to P the conditioning set.
Submodular Conditional Mutual Information (SCMI) Given sets 2, S U, the SCMI is defined as (AA QIP) = ((AUP) S(QUP) - - ((AUQUU) - f(P). Intuitively, SCMI jointly models the similarity between A and Q and their dissimilarity with P.
Relationship between SIM The rela- tionship between the above measures is the key component that unifies our AL framework [19, 23]. The unification comes from the rich modeling capacity of SCMI: (A; Q)P) where Q. P C U. This facili- tates a single acquisition function that can be applied to multiple scenarios. Concretely, the submodular function ff can be obtained by setting Q + Ul and p <- O. Next, the SMI can be obtained by setting Q + Q and P < 0, while we obtain SCG by setting Q + 0, P + P. We summarize the relationships between SIM in Tab. 1.
Function Submodular
Setting
Realistic Scenario
Q x- Q-u, u,P+D00 4-0 0
Standard AL
SMI
Q - -. + 0
Imbalance, OOD Redundancy
SCG
Q Q- @-0.P. 0.P P
SCMI
P e Q,P
OOD
Table I: Relationship between SIM and their applica- tions to realistic scenarios by choices of Q and P.
Instantiations of SIM. The formulations for Facility Location (FL), Graph Cut (GC) and Log Determinant (LOGDET) are in [19, 23] and we adapt them acquisition functions for batch active learning. We use two variants for FL. FLQMI, which models pairwise similarities of only the query set Q to the unlabeled dataset. and FLYMI, which additionally considers the pairwise similarities within the unlabeled dataset U. The SCG and SCMI expressions corresponding to FL are referred as FLCG and FI.MMI respectively (see row 1 in Tab. 2a and 2b). For LOGDET, we refer to the SMI, SCG and SCMI expressions as LOGDETMI. LOGDETCG and LOGDETCMI, respectively (see row 5 in Tab. 2a and row 2 in Tab. 2b). Similarly, the SMI and SCG expressions are respectively referred to as GCMI and GCCG for GC (see row 3 in Tab. 2a and 2b). For notation in Tab. 2, the pairwise similarity matrix between items in sets A and B denoted as SA.B. Also, we denote Siy as the (i,j) entry of S.
3 SIMILAR: Our Unified Active Learning Framework
In this section, we propose unified active learning framework SIMILAR, which uses SIMs to address the limitations of the current work (see Sec. 1.1). We show that SIMILAR can be effectively applied to broad range of realistic scenarios and thus acts as one-stop solution for AL.
3

---

Table 2: Instantiations of SIM. Note how the relationships in Tab. can be applied to SCMI
instantiations to obtain SMI and SCG instantiations.
(b) Instantiations of SCG and SCMI functions.
(a) Instantiations of SMI functions.
SCG
[(A(P)
SMI
1, 2)
maxfmna S1z
max Siz, 0)
FLVMI
min(max Sizi Siz) JEA
LogDeICG
log degie SA 1.8pp5. sA,))
FLQMI
S Σ JEA
Σ
Sij
GCOG
((A) -2 Σ Sis
JER
ieddeP
GCMI
2 Σ Σ Stz
SCMI
KEASEO
LOGDETMI
det(( A) log SSt $4.0 5 5
FLCMI
(mix( max Sis max 8iz) max S)
JEG
dettl Sp o $7.0
LogDeCMI
log
2at(1- AUD A 2 Au
The basic idea behind our framework is to exploit the relationship between the SIMs (Tab. 1) such that can be applied to any real-world dataset. Particularly, we use the formulation of SCMI and appropriately choose query set Q and/or conditioning P depending on the scenario hand. Towards this end, we use the inspiration from [3] where they select data points based on diverse gradients. The SIM functions (see Tab. 2) instantiated using similarity kernels computed using pairwise similarities S.j between the gradients of the current model. Specifically, we define Siy = (0), (O), where (0) = HH(z; 0) is the loss on the ith data point. Similar to [45, 31, we use hypothesized labels for computing the gradients, and the corresponding similarity kernels. The hypothesized label for each data point is assigned as the class with the maximum probability. We then optimize SCMI function:
(, QIP)
AÇU,AAIBB
with appropriate choices of query set Q and conditioning set P. In the context of batch active learning, A is the batch and is the budget (batch size in AL). We present our unified AL framework in Algorithm and illustrate the choices of query and conditioning set for realistic scenarios in Fig. 2.
Algorithm 1 SIMILAR: Unified AL Framework
Require: Initial Labeled set of data points: L, large unlabeled dataset: U, Loss function H. for learning model M. batch size: B, number of selection rounds: N
1: for selection round i 1 N do
2: Train model M with loss H on the current labeled set and obtain parameters
3: Using model parameters 0,, compute gradients using hypothesized labels (VoH(az, ti 0), Vj E4) and obtain a similarity matrix X.
4:
Instantiate a submodular function If based on X.
5: A; + argmax 5 P)P) (Optimize SCMI with an appropriate choice of and
P. see Tab. 1)
6: Get labels L(A,) for batch A; and < M Ar
7: end for
8: Return trained model M and parameters 0.
In the scenarios below, we will discuss how this paradigm can provide unified view of active learning, handle aspects like standard active learning (Sec. 3.1,, rare classes and imbalance (Sec. 3.2), redundancy (Sec. 3.3) and, OOD/outlers in the unlabeled data (Sec. 3.4).
3.1 Standard Active Learning
We refer to standard active learning for ideal scenarios when there is no imbalance, redundancy or OOD data the unlabeled dataset. such cases, there is no requirement for having a query set and conditioning set. Hence, given a SCMI function (AA; P)P). we get T (A: 2)P) = f(A) by setting Q - U (the unlabeled dataset) and P 0. In nutshell, the standard diversified active learning ting can be seen as a special case of our proposed unified AL framework (Equ. (1)) by choosing Q, P above. Note that this approach is very similar and closely related to BADGE [3], where the authors

---

also choose points based on diverse gradients. Furthermore, the authors discuss the use of Determinan- tal Point Processes (DPP) [28] for sampling, and this very similar maximizing dog-determmnants. In the supplementary paper, we compare the choice of different submodular functions for AL..
3.2 Rare Classes
A very common and naturally occurring scenario is that of imbalanced data. This imbalance is because some classes or attributes are naturally more frequently occurring than others in the real- world. For example, in a self-driving car application, there may be very few images of pedestrians at night on highways, cyclists at night. Another example is medical imaging, where there are many rare yet important diseases (e.g., various forms of cancers), and it often the case that non-cancerous images are much more than compared to the cancerous ones. While such classes are rare, it is also critical to be able to perform well in these classes. The problem with running standard active learning algorithms in such a case is that they may not sample too many data points from these rare classes, and as a result, the model continues to perform poorly on these classes. In such cases, we can create a (small) held-out set R which contains data points from these rare classes, and try to encourage the AL by sampling more of these rare classes by maximizing the SMI function I R):
max AC
(2)
This setting is shown in Fig. 2(a). R contains small number of held-out examples of classes 5, 8.8 which are rare, and the AL acquisition function is Equ. (2). Note that this is exactly equivalent to maximizing the SCMI function with Q < R and P < 0 (i.e. Equ. (1) in Line 5 of A Algorithm 1). Fur- thermore, since the SMI functions naturally model query relevance and diversity, they will also try to pick a diverse set of data points which are relevant to R. Finally, we also point out that this setting was considered in [15] where they use a FISHER kernel based approach to sample data points. Note that for this setting to be realistic, it is critical that the size of this validation set very small - [15] uses a much larger validation set which is not very realistic (c.g., 200x our set, see Appendix B for more details).
3.3 Redundancy in Unlabeled Data
[a] Rare Classes
(b) Redundancy
[c) Outaf--isttribtion
Another commonplace scenario is where we are dealing with a lot of redundancy - e.g. frames sampled from a video, where subsequent frames visually similar. In such cases, existing AL algorithms tend to pick data points that are se- mantically similar to the ones selected in some earlier batch. This true even for the ofa the-art AL algorithm BADGE [3] that attempts to enforce diversity, but only in the current batch of data points and not the already selected labeled set. To illustrate this, consider the scenario in Fig. 2(b). The digits 0, are redundant in the unlabeled set, and they are already present in the labeled set C. Algorithms which just focus on diversity in the current batch could fail at ensuring diversity across batches. To mitigate inter-batch redundancy, we use SCG acquisition function and condition upon the already labeled set L:
R
O
S
O 0
OI
3
5
8
2 3
U
U
9
9
7
0 0
O
4
4 5
9 3
9
6
6
7
0
0
O 6
9
3
2
/
6
4 % 2 2
4
8
3
7
7
5
4
5 5 + 1)(A; A/ A:
4 3 3
4 6
7
4
6
6
/
3 2
A
B
0
2
J
O
7
4
f(ALL)
4IATTO)
AGPL
A:C A
Figure 2: An illustration of realistic scenarios where SIMILAR is applied with appropriate choices of query and conditioning sets: a) SIMILAR finds rare digits 5, 8 E U1 by optimizing the SMI func- tion (A; R) with R containing 5, 8 as queries, b) select samples from l/ which are diverse among themselves and also diverse w.r.t those in L by optimizing ((AA(C (here, we want to avoid dig- its O. E 24 altogether because they are present in C), c) select digits (in-distribution) and avoid in 2t by optimizing alphabets where I are IDD labeled points and O (A;;
max AC
f(AIC)
(3)
Notice that this a special case of our proposed unified AL framework (Equ. (1)) since the SCG function f(A is basically SCMI function with Q < and P + L.
3.4 Out of Distribution Data
are OOD points selected so far.
In real world scenarios, we often have out-of- distribution (OOD) data or irrelevant classes in the unlabeled set. Such OOD data is not useful for
5

---

the given classification task at hand. Using an acquisition function that selects lot of OOD data points will lead a waste of labeling effort and time. This because annotators have spend time in filtering out OOD data points and discard them from the training dataset. To account for OOD data, we add additional class called "OOD" in our model. Since the goal is to improve on in-distribution classes, we ignore the prediction for the OOD class at test time. For our AL acquisition function, we use the currently labeled OOD points O as the conditioning set P. and the currently labeled in-distribution (ID) points as the query set Q. In other words, our acquisition function is to optimize:
max AC
({(A
(4)
This is illustrated in Fig. 2(c), where the labeled set consists of six examples, four of them being ID data points (set Z) and two being OOD data points (set O). In Fig. 2(c), the ID data are digits
(digit classification) and the OOD examples are alphabets. This SCMI based approach will naturally pick points "close" to the ID data while avoiding the OOD points.
Another approach for designing the acquisition function is to not explicitly condition on the OOD data points. In other words, we can just optimize the SMI function:
max
(A;
(5)
ACU.
We contrast the choices of SCMI (Equ. (4)) and SMI (Equ. (5)) functions in our experiments.
3.5 Multiple Co-occurring Realistic Scenarios
We also apply SIMILAR to datasets where more than realistic scenarios are co-occurring. As illustrated in Tab. 3, we can use the formulation of SCMI and make appropriate choices of Q and P to tackle multiple realistic scenarios.
Function ((()
Setting
Realistic Scenario
o - R. O
Rare classes + OOD)
[ R) ZOuZ'
@ + R, C
Rare classes Redundancy
@ I.P++tOu
Redundancy OOD
Table 3: Choices for Q and P for multiple co-occuring realistic scenarios
We set Q <- R and P <- O and maximize 1, RJO). Intuitively, this function would pick points close to R while avoiding the OOD points. this scenario, we can also optimize an SMI function 1 (A; R) if the data points belonging to the rare classes are not similar to the OOD data points, meaning that only searching for classes may suffice. Regardless, the SCMI approach above will further reinforce the avoidance of the OOD points.
Rare classes and OOD:
Rare classes and Redundancy: We set Q < R and + L R. Here, R is the subset of data points from the labeled set L that belong to the rare classes. Intuitively, this function would pick points close to R while avoiding points already in L - R. thereby avoiding redundant data. Just focusing R by optimizing (A; R) is also feasible option because rare classes generally not redundant. As before, the SCMI approach will only reinforce the avoidance of redundant samples in any non-rare class instances selected.
This is more challenging scenario than the ones above. We start with using the SCMI formulation for the OOD scenario, i.e., T (A; OO), where I is the set of ID samples and O is the set of OOD samples. Optimizing this function will pick diverse in-distribution samples within a batch. For selecting diverse samples across different batches, we can tackle this by using an appropriate kernel for the conditioning set. For instance, consider the FL.CMI function in Tab. 2(b). On setting P +- UI, we can rewrite the FLCMI function by splitting the penalty term as follows: Σ max (min( (max Sij, max S:₁) max(max Sij, max Sty) ()). While S is computed using cosine jEO
Redundancy and OOD:
kEWl
jEA
JEZ
jEZ
similarity, we can compute S using an exponential kernel to magnify the value of S using the exponent when i and j are very similar. This exponent is hyperparameter which can be tuned to penalize selecting redundant samples from / (denoted as 7') in Tab. 3.
3.6 Realizing Realistic Scenarios in Applications
In this section, we discuss a few insights on how these realistic scenarios can be realized. To begin with, the initial labeled set used in AL usually follows the distribution of the unlabeled set. The

---

statistics of this set can be used to identify rare classes. If the initial seed set is small, the rare classes/OOD data points can be realized after few rounds of standard AL. Until such scenarios are discovered, standard AL can be done using a diversity- acquisition function like the log determinant (LOGDET). For production-level models, they go through test deployment phase. During this phase. systematically recurring errors are often found. An example is of undetected bicycles at night in an object detector (false negatives). Such recurring failure cases can be due to rare classes in the labeled set. Moreover, we as users often know whether there are rare classes or if there is redundancy from domain knowledge. For instance, in the biomedical domain, images of cancer cells are typically rarer than ones of non-cancer cells because cancer inherently is rare disease.
3.7 Scalability and Computational Aspects of SIMILAR
The computational complexity of the different SMI functions determined by (1) the kernel computation time, and (2) the time complexity of the greedy algorithm. All functions considered here are graph based functions and require computing a kernel matrix. The LOGDET functions (LOGDET, LOGDETMI, LOGDETCG, LOGDETCMI), some Fr. functions (FI., FLVMI, FLCMI), and GC, GCMI all require the n X n similarity matrix (n [44 is the number of unla- beled points) which entails a complexity of (n²) to construct the similarity kernel. Once constructed, the complexity of the greedy algorithm for LOGDET class of functions is roughly O(Bn)) [11], while the complexity of the greedy algorithm with FL. FLVMI, and FLCMI is O(Bn²) [18, 20](B is the batch size). Different from others, FLQMI does not require computing a n X n kernel, but only a x q kernel (where q = [ is the number of query points). Correspondinglyy the complexity of the greedy algorithm with FLQMI is B3) and is linear in n. In Appendix. A, we provide a detailed summary of the complexity of different SF, SMI, SCG, and SCMI functions.
Computational Complexity:
Partition Trick: The deal with the high )n²) of the LOGDET, GC, and some of the Ft. variants (except FI.QMI), we also propose the following partitioning algorithm: We randomly split the unlabeled set U into p partitions U1 Up. and we then define the corresponding function (SF SMI, SCMI, SCG) on each of the partitions and independently optimize them. In each partition, we select B/p points. The complexity of this reduces from O to O(n² /p) and with an appropriate choice of p. we can significantly reduce the computational complexity. We use this in our ImageNet experiments (see Sec. 4.1), and observe that our approaches continue performing well while being more scalable. We provide more details on partitioning in Appendix. A.
Deep models have numerous parameters leading to very high dimensional gradients. Since our kernel matrix is computed using the cosine similarity of gradients, this becomes intractable for most models. To solve this problem, we use last-layer gradient approximation by representing data points using last layer gradients. BADGE [3], CORESET [40] and GLISTER [24] are other baselines that also use this approximation. Using this representation, we compute a pairwise cosine similarity matrix instantiate acquisition functions in SIMILAR (see lines 3,4 in Algorithm 1).
Last Layer Gradients:
4 Experimental Results
In this section, we empirically evaluate the effectiveness of SIMILAR on a wide range of scenarios like rare classes (Sec. 4.1). redundancy (Sec. 4.2) and out-of out-if-distribution (Sec. 4.3). We do so by comparing the accuracy and selections of various SCMI based acquisition functions with existing AL approaches. Using these experiments, we cover the issues with the current AL methods and show that these issues can be mitigated by using a unified implementation using SCMI with appropriate choices of query and/or conditioning sets. Although this section focuses on realistic scenarios, we also study SIMILAR in standard active learning setting and show that it performs at par with current AL methods (see Appendix. C). Furthermore, we present some experiments on a real-world medical dataset in Ap- pendix. H and some experiments on multiple co-occurrngg realistic scenarios (Sec. 3.5) in Appendix. I.
Baselines in all scenarios: We compare SCMI based functions against several methods. Particularly, we compare against: (1) three uncertainty based AL algorithms: i)ENTROPY: Selects the top B data points with the highest entropy [41], ii) MARGIN: Select the bottom B data points that have the least difference in the confidence of first and the second most probable labels [37], LiASEE CONF: Select B samples with the smallest predicted class probability [44], (2) state-of-the-ar diversity based algorithms: iv) BADGE [3] v) GL.ITERR [24] vi) CORESET [40] which are all discussed in section Sec. 1.1, and, 3) RANDOM: Select B samples randomly. Additionally. in the rare classes scenario, we compare against FISHER [15] which is also discussed in Sec. 1.1.
7

---

LOGDETMI PLVMI
DIV-GCMI - PISHIER-LV BADGE
ENTROPY CORESET LEAST-CONT RANDOM
MARGIN
FLQMI GCMI
PISHTER
GLISTER
Auteewy un Bare Clesse
(1) Rare Classes Accuracy
(b)IIFAR-10: Overall Accuracy
Crmmiileivv wittar Sumplos
(c)CIFAR-10: # Raro Class Samples
Ovegnill Arcuney
125 1230
100
1105
1250
135 250 573 D0D
625
750
573
1800 1125 1250
135
250
575
500
615
7M0
573
1800
1125
1230
135
250
575
540
0.5
750
875
1000
1125
(d)MNIST: Rare Classes Accuracy
(e)MNIST: Overall Accuracy
Cuunlaaive Stumpee
(F)NNIST: Rarn Class Sumples
Accuray
300
Areurayy
60
100
125 250 375 500 eds
750 als 1000 riss 1200
125
250
355
500
025
750
als
1000 1:55 1290
125
230
375
500
625
730
675
1000
1125
(g]amageNet: Rare Clusses Accuracy
[b)anageNet:: Overall Accuracy
(IImageNet: # Rare Class Samples
10
18
30
300000
30000
102000 127000
152000
1971900 2020000
2271030
102500
1220000
152000
177950 202000 2271000
Lelocked Sei Steee
Enboked Sei Stee
Lebecaa Sct Soe
Figure 3: Active Learning with rare classes on CIFAR-10 (top row), MNIST (middle row), and ImageNet (bottom row). Left side plots (a.d.g) are rare class accuracies, center plots (b.e.h) are overall test accuracies, right plots (c,f.i) are number of rare class samples selected. The SMI functions (specifically LOGDETMI, FLQMI) outperform other baselines by more than 10% on the rare classes.
We apply our framework to CIFAR-10 [27] and MNIST [30] classification tasks. Additionally, we also evaluate our method on down sampled 32x 32 ImageNet-2011 [38] for the rare classes setting (Sec. 4.1). Due to the lack of test split on ImageNet, we used the validation split for evaluation. In the sections below, we discuss the individual splits for f. L4 R. I, and O in each realistic scenario. To ensure that all the selection algorithms that we are studying are given fair and equal treatment across all realistic scenarios, we use a common training pro- cedure and hyperparameters. We use standard augmentation techniques like random crop, horizontal flip followed by data normalization except for MNIST which does not use horizontal flip to preserve labels. For training, we use an SGD optimizer with an initial learning rate of 0.01, the momentum of 0.9, and a weight decay of Se-4. We decay the learning rate using cosine annealing [31] for each epoch. On all datasets except MNIST, we train a ResNet18 [17] model, while on MNIST we train a LeNet [29] model. For all the experiments in particular scenario (rare classes, redundancy and OOD), we start with identical initial model M and initial labeled set D. We reinitialize the model parameters at the beginning of every selection round using Xavier initialization and train the model until either the training accuracy reaches 99% the epoch count reaches 150. We run each experiment 3 CIFAR- 10 and MNIST and xx on ImageNet and provide error bars (std deviation). All experiments were run on a V100 GPU, For more details on the experimental setup, baselines, and datasets see Appendix. B.
Datasets, model architecture and experimental setup:
4.1 Rare Classes
Following [15, 24], we simulate these rare classes by creating a class imbalance. We initialize the batch active learning experiments by creating a custom dataset which is subset of the full dataset with the same marginal distribution. Given that C consists of data points from the imbalanced classes and D consists of data points from the balanced classes, we create initial labeled set L such that |Dxl = p|Cz\ and an unlabeled set |Dxy| = plCexl, I. where P is the imbalance factor. We use small and clean validation/query set R containing data points from the imbalanced classes (== 3 data points per imbalanced class). We create an imbalance CIFAR- 10 using 5 random classes, p= = 10 and for MNIST we create an imbalance using the same classes as in [15] (5 (5...9) and use p= = 20. For both datasets: ICe |Dzl 125, |Cyl + |Dul 16.5K, B 125 (AL batch size) and, [R] = 25 (size of the held out rare instances). For MNIST, we also present the results for B = 25 and p 100 in the supplementary. On ImageNet, we randomly select 500 classes out
Custom dataset:
8

---

LOCDETCG FLCC- CLISTER BADGE CORESET ENTROPY MARGIN RANDOM
(a) Overall Accuracy
(b)MNIST: Overall Accuracy
40
Samplos
(e)CTPAR-
Unique Samples Selected
2000
17.00
42
2000
20
13.17
67.08 AM
Number
4500
5000
410
50p
300 1010
1300 2003 2000 2014 53AN)
4002 E5o0 5000
30
100
150
250
204
300
350
400
454
300
1000 520
1500 2000 2000 2000 3000 4000 4500
Figure 4: Active Learning under 10x redundancy for CIFAR-10 and MNIST. The CG functions (LOGDETCG, FI.CG) pick more unique points and outperform existing algorithms including BADGE.
of 1000 classes for imbalance and p = 5 such that |C2] + |Del = 102K, Cal + |Dul = 664K' B 25K and, R 2.5K. These data splits are chosen to simulate a low initial accuracy on the rare classes and at the same time maintain the imbalance factor in the labeled and unlabeled datasets.
The results are shown in Fig. 3. We observe that SMI based functions not only consistently outperform uncertainty based methods (ENTROPY, LEAST-CONE and MARGIN) but also all the state-of-the-art diversity based methods (BADGE, GLISTER, CORESET) by ~ 5 - 10% in terms of overall accuracy and R 10 18% in terms of average accuracy on rare classes (see Fig. 3a, 3d, 3g). The reason for the same can be seen in Fig. 3c, 3f, 3i which illustrates that they fail to pick an adequate number of examples from the rare classes. Evidently, FLQMI and LOGDETMI which balance between diversity and relevance perform better than GCMI which only models relevance. Furthermore, DIv-GCMI which is a linear combination of GCMI and a diversity term performs consistently worse, which suggest that naive combination of the two may not be as effective. This suggests the need of SMI based acquisitions functions (Equ. (2)) with richer modeling capabilities like FI.QMI and LOGDETMI within SIMILAR. Furthermore, all SMI based functions also outperform the FISHER kernel based method when the validation set is small and realistic, i.e., R = 25. Since, [15) use very large validation set in their experiments, we try their method FISHER-LY -LV with a 40x larger validation set of size 1000 (which is not practical) and observe a comparable performance with the SMI functions which use a small validation set. Furthermore, we see that FISHER-LV actually picks significantly larger number of rare class instances in MNIST, but yet is comparable in performance of FLQMI and LOGDETMI. This suggests that both these methods select higher quality and diverse rare class instances. We observe that the Gc SMI variants( GCMI and DIV-GCMI) do not perform well on MNIST classification. Finally, we point out in the case of ImageNet, FLQMI performs the best and outperforms FLVMI and LOGDETMI this is because we do not need to do the partition trick for FLQMI since is already linear in time complexity. For FLVMI and LOGDETMI, we set the number of partitions = 50 for ImageNet. Finally, we do a pairwise t-test to compare the performance of the algorithms (Appendix. D) and observe that the SMI functions (and particularly FLVMI and LOGDETMI) statistically significantly outperform all AL baselines.
Results:
4.2 Redundancy
To simulate a realistic redundancy scenario we create a custom dataset by duplicat- ing 20% of the unlabeled dataset 10x. For CIFAR-10, the number of unique points in the unlabeled set U 5K, the initial labeled set |C|| 500, B 500, whereas for MNIST Z 500, | 50
Custom dataset:
and B- = 50. For MNIST, we also present the results for 5x and 20x in the Appendix. E.
SCG vs Baselines: As expected, the diversity and uncertainty based methods outperform random. Importantly, we observe that the SCG functions (FL.CG and LOGDETCG) significantly outperform all baselines by x 3 - 5% towards the end as the conditioning gets stronger with increase in L (see Fig. 4a. 4b). This implies that simply relying on model parameters for diversity and/or uncertainty is not sufficient and that conditioning on the updated labeled set L (Equ. (3)) is required in batch active learning. In Fig. 4c we show that SCG based acquisition functions select significantly more unique data points than other baselines. We also perform a pairwise t-test (Appendix. E), to prove that the SCG functions consistently and statistically significantly outperform BADGE and other baselines.
4.3 DutttrrDisti
Custom dataset: We simulated a scenario where we convert the classification problem in CIFAR-10 and MNIST to 8-class classification, where the first 8 classes represent the set IF of in-distribution

---

LOGDETCMI LOCDETMI
FLVMI
DIV-GCMI
BADGE CORESET
ENTROPY MARGIN
RANDOM
FLCMI
FLQMI
GCMI GLISTER
(a)CIFAR-10: SCMI vs Baselines
(1)CIFAR-10: SCMI
teJOOD: Variance Comparison
78
76
2,09
Acorsey
Sverge Veriowe
7451
72.17
00
1000
1000 1050 2300
2330 2030
2634 2000 3300 3000 3630
1000 1850 2300
2030 2000
2650
3100
3500 3000 Sobu
Labeled
(dJMNIST: SCMI Baselines
(e)MNIST: SCMI SMI
(f)MNIST: #ID
Nouber ID Sumples
Accunccy
Accuracy
110
12
70
00
100 120 140 160
100
200
220
100 120 140
100
180
200
320
40
60
500 120
3.80
100
380
200
Figure 5: Active Learning with OOD data in unlabeled set. Top row: CIFAR-10 results for (a) SCMI vs Baselines, (b) SCMI vs SMI, and (c) variance comparison of different baselines, bottom row: MNIST results for (d) SCMI vs Baselines, (c) SCMI vs SMI, and (f) Number of ID points selected. We see that, i) the SCMI functions consistently outperform the baselines by 5% - 10%, ii) SCMI functions outperform the corresponding SMI functions for later rounds, and (ii) SCMI functions have the least variance compared to the rest, showing that they are more robust in performance.
(II) data points and the last 2 represent the set Op of cittoffddisrrbbutionnoOO initial labeled set L consists only of ID points, i.e. OF L 0. The unlabeled set is simulated to reflect a realistic and somewhat extreme setting where the unlabeled ID data points [Zpl is much smaller than the unlabeled OOD data points Orl. Additionally, we also assume we have very small validation set of ID points Zy. For CIFAR- 10: Ic = 1.6K, [Z,| = 4K, | = 10K, [Zy = 40, B 250 whereas for MNIST which is relatively simpler task, we use a smaller initial labeled sets and keep the unlabeled sets of the same size: [ = 40, [Ip] = 400, |0,l = 10K, [Tv| = 16, B- = 20. Recall that our algorithm uses ID set I (initialized to Iv) and OOD set O which we build as follows, Every time our selection approach selects a set A, we update I = I U (An IP) and i.e. we augment the ID and OOD points in A to the sets I and O respectively.
data points. The
O = O
Since we care about the predictive performance of the ID classes, we report the ID classes accuracy. We see that SCMI based acquisition functions significantly outperform existing AL approaches by ~ 5 - 10% (see Fig. 5a, 5d). We also observe that existing acquisition functions have a high variance, which is undesirable in real-world deployment scenarios where deep models are being continuously developed. Our SCMI based acquisition functions (LOGdETCMI and FLCMI) show the lowest variance in training (see Fig. 5c). This reinforces the need of having framework like SIMILAR that facilitates query and conditioning sets.
SCMI vs Baselines:
We compare SCMI functions against SMI functions to study the effect of conditioning and observe that the SCMI functions are comparable to the SMI functions initially but in the later selection rounds of active learning, the SCMI functions consistently outperform SMI functions. In particular, we see an improvement of 2 - 3% as the conditioning becomes stronger (see Fig. 5b, 5e). We also observe the SCMI tends to select more IDD points than SMI and other baselines (see Fig. 5f), and SCMI functions have a lower variance overall compared to even the SMI functions (Fig. 5c).
SCMI vs SMI:
5 Conclusion
In this paper, we proposed a unified active learning framework SIMILAR using the submodular information functions. We showed the applicability of the framework in three realistic scenarios for active learning, namely rare classes, redundancy, and out of distribution data. In each case, observed that the functions in SIMILAR significantly outperform existing baselines in each of these tasks. Our real-world experiments on MNIST, CIFAR-10, and ImageNet show that many of the SIM functions (specifically the LOGDET and FL variants) yield 5% 18% gain compared to existing baselines, particularly in the rare class scenario and RJ 5% 10% OOD scenarios. The main limitations of our work is the dependence on good representations compute similarity. A potential negative socictal impact of this work is the use of SIMILAR to perpetuate certain biases through a malicious use of the query and conditioning set. We discuss this in more detail in Appendix. G.
10

---

Acknowledgments and Disclosure of Funding
This work is supported by the National Science Foundation under Grant No. IIS-2106937, a startup grant from UT Dallas, and by Google and Adobe research award. Any opinions, findings, and conclusions recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation, Google or Adobe.
References
[1] Sami Abu-EI-Haija, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakrishnan Varadarajan, and Sudheendra Vijayanarasimhan. Youtube-8m A large-scale video classification benchmark. arXiv preprint arXiv: urXiv:1609.000 (609.0867755 2016.
[2] David Arthur and Sergei Vassilvitskii. k-means++: the advantages of careful seeding. In SODA '07: Proceedings of the eighteenth annual ACM-SIAM symposium on Discrete algorithms, pages 1027-1035, Philadelphia, PA, USA, 2007. Society for Industrial and Applied Mathematics. ISBN 978-0-8987 167776-2224 24-5..5.
[3] Jordan T Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agar- wal. Deep batch active learning by diverse, uncertain gradient lower bounds. arXiv preprint arXiv:1906.080 03671, 2019.
[4] Francis Bach. Learning with submodular functions: A convex optimization perspective. arXiv preprint 1eXiv:///1.64533: 2011.
[5] Francis Bach. Submodular functions: from discrete to continuous domains. Matherattiaal Programming, 175(1):419-459, 2019.
[6] Christopher Berlind and Ruth Urner. Active nearest neighbors in changing environments. In International Conference on Machine Learning, pages 1870-1879. PMLR, 2015.
[7] Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency, pages 77-91. PMLR, 2018.
[8] Holger Caesar, Varun Bankiti, Alex Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 11621-11631, 2020.
[9] Colin Campbell, Nello Cristianini, Alex Smola, et al. Query learning with large margin classifiers. In ICML, volume 20, page 0, 2000.
[10] Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Kochn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint ar 131..30050 2013.
[11] Laming Chen, Guoxin Zhang, and Hanning Zhou. Fast greedy map inference for determinantal point process improve recommendation diversity. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 5627-5638, 2018.
[12] Shai Fine, Ran Bacahdah and Eli Shamir. Query by committee, linear separation and random walks. Theoretical Computer Science, 284(1):25-4) 2002.
[13] Yoav Freund, Sebastian Seung, Eli Shamir, and Naftali Tishby. Selective sampling using the query by committee algorithm. Machine learning, 1332)1333-168, 1997.
[14] Satoru Fujishige. Submodular functions and optimization. Elsevier, 2005.
[15] Denis Gudovskiy, Alec Hodgkinson, Takuya Yamaguchi, and Sotaro Tsukizawa. Deep active learning for biased datasets via fisher kernel self-supervision. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 9041-9049, 2020.
[16] Anupam Gupta and Roie Levin. The online submodular cover problem. In ACM-SIAM Symposium Discrete Algorithms, 2020.
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 770-778, 2016.
11

---

[18] Rishabh lyer and Jeffrey Bilmes. A memoization framework for scaling submodular optimiza- tion large scale problems. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 2340-2349. PMLR, 2019.
[19] Rishabh Iyer, Ninad Khargoankar, Jeff Bilmes, and Himanshu Asanani. Submodular combina- torial information measures with applications in machine learning. In Algorithmic Learning Theory, pages 722-754. PMLR, 2021.
[20] Rishabh Krishnan Iyer. Submodular optimization and machine learning: Theoretical results, unifying and scalable algorithms, and applicationn. PhD thesis, 2015.
[21] Kimmo Karkkainen and Jungseock Joo. Fairface: Face attribute dataset for balanced race, gender, and age for bias measurement and mitigation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1548-1558, 2021. 2021.
[22] Vishal Kaushal, Rishabh Iyer, Suraj Kothawade, Rohan Mahadev, Khoshrav Doctor, and Ganesh Ramakrishnan. Learning from less data: A unified data subset selection and active learning framework for computer vision. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1289-1299. IEEE, 2019.
[23] Vishal Kaushal, Suraj Kothawade, Ganesh Ramakrishnan, Jeff Bilmes, and Rishabh lyer. Prism: unified framework of parameterized submodular information measures for targeted data subset selection and summarization. arXiv preprint arXiv:2103.00/22 2021.
[24] Krishnateja Killamsetty, Durga Sivasubramanian, Ganesh Ramakrishnan, and Rishabh Iyer. Glister: Generalization based data subset selection for efficient and robust learning. arXiv
preprint arXiv:2012.106.00 2020.
[25] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv: 14, 2013.
[26] Andreas Kirsch, Joost Van Amersfoort, and Yarin Gal. Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning, arXiv preprint arXiv:1906.081 2019.
[27] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[28] Alex Kulesza and Ben Taskar. Determinantal point processes for machine learning. arXiv preprint arXiv: /207. 6083, 2012.
[29] Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4):541-551, 1989.
[30] Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. at&t labs, 2010.
[31] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv: 160..v1660.0000 033983, 2016.
[32] Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adver- sarial autoencoders. arXiv preprint 1.1.5:151..05644, 2015.
[33] Baharan Mirzasoleiman. Ashwinkumar Badanidiyuru, Amin Karbasi, Jan Vondrák, and An- dreas Krause. Lazier than lazy greedy. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
[34] George L Nemhauser, Laurence A Wolsey, and Marshall L Fisher. An analysis of approximations for maximizing submodular set functions-i. Mathematical programming, 14(1):265-294, 1978.
[35] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
[36] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury. Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, F. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019. URL www.roorrranc-deeeeeeeeeaannngn. 1iii-rrbrrrr.. 2015-pytoree ch an iwwpriitieeestteeeeeeeeeee style high www.orewaceeeeeee pdf.
12

---

[37] Dan Roth and Kevin Small. Margin based active learning for structured output spaces. In European Conference on Machine Learning, pages 413-424. Springer, 2006.
[38] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 115(3)2111-553 2015.
[39] Greg Schohn and David Cohn. Less more: Active learning with support vector machines. In
ICML, volume 2, page 6. Citeseer, 2000.
[40] Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv: 1en:::00800040 00489, 2017.
[41] Burr Settles, Active learning literature survey. 2009.
[42] Samarth Sinha, Sayna Ebrahimi, and Trevor Darrell. Variational adversarial active learning. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 5972-5981, 2019.
[43] Ehsan Tohidi, Rouhollah Amiri, Mario Coutino, David Gesbert, Geert Leus, and Amin Karbasi. Submodularity in action: From machine learning to signal processing applications. IEFE Signal Processing Magazine, (7(5):120-133) 2020.
[44] Dan Wang and Yi Shang. A new active labeling method for deep learning. In 2014 International joint conference neural networks (IJCNN), pages 1921199.. IEEE, 2014.
[45] Kai Wei, Rishabh Iyer, and Jeff Bilmes. Submodularity in data subset selection and active learning. In International Conference Machine Learning, pages 1954-1963. PMLR, 2015,
[46] Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. arXiv preprint arXiv: 1509001600 2015.
[47] Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pages 19-27, 2015.
13


Reviews:
[{"review_id": "bfVjRmL2nEW", "reviewer": "Reviewer_yRLn", "summary": "In this paper, the authors propose a new diversity based active learning algorithm by utilizing a series of information functions in submodular optimization (SO) problems. Specifically, they consider three submodular information measures (SIM) in SO: Submodular Mutual Information (SMI), Submodular Conditional Gain (SCG) and Submodular Conditional Mutual Information (SCMI) and found that SCMI ($I_f(\\mathbf{A};\\mathbf{Q}|\\mathbf{P})$) is the most expressive SIM such that one could tailor $\\mathbf{Q}$ and $\\mathbf{P}$ to represent SCG and SMI, and thus propose to use it as a “unified” acquisition function for tackling three scenarios in realistic AL: rare class due to class imbalance, redundant data and out-of-distribution(OOD) data. The basic idea is to specify $\\mathbf{Q}$ and $\\mathbf{P}$ depending on the desired/un-desired classes, such that one could condition on data region that they want to exclude (OOD, redundant data), while up-sample regions that one want to include (rare class) by considering the mutual information w.r.t that region. To instantiate SIM acquisition function, the authors borrow the utility function from three well known graph-based SO problems for diversity coverage: Facility location (FL), Graph cut (GC) and Log Determinant (LOGDET), and use the similarity kernels computed with pairwise cosine similarity of the last layer gradients w.r.t the model-predicted label for every pair of unlabeled data points. They experiment with synthetic datasets rooted from MNIST/CIFAR-10/ImageNet to test for the three special scenarios, and show promising results against several strong baselines including BADGE.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros\n* The authors provide a systematic exploration of the application of submodular information measures from graph-based submodular optimization problems on diversity based batched AL, which is quite unique and novel.\n* The proposed SCMI framework is one of the first (to this reviewer) that can be explicitly tailored for all three scenarios (rare class, OOD data, and redundancy).\n* The authors conduct quite extensive benchmark experiments on various synthetic setups and show consistent out-performance on the three special scenarios in the metrics of interest.\n* The paper is well structured and covers a lot of material.\n\nConcerns\n* The method seems to highly rely on the availability of example data for the desired/un-desired classes, which is possible in synthetic dataset but not always achievable in reality. For example, how do one know ahead of time which classes are the minor/rare class before having the labels available? How do one define OOD samples on the large space of images? Without human susceptions or prior knowledge, the proposed methods may face limitations when applied in real dataset.\n\n* The rare class and OOD setup both specify a subset of samples to compute mutual information with. For rare class it uses $\\mathit{R}$ which is a set of labeled rare class sample and for OOD it uses the currently labeled in-distribution data $\\mathit{I}$. However, neither of the subsets is guaranteed to be covering enough sample space of the full unlabeled and relevant (non-OOD) examples. Wouldn’t this introduce bias into the AL acquisition such that it will only pick samples that are similar to the subset while leave the rest unexplored?\n\n* There might be some concerns in the experimental setups.  If R consists of data from imbalanced classes $\\mathbf{C}$ which is 10 or 20 larger than $\\mathbf{D}$, how come $\\mathbf{C}$ is considered as rare class? As the SIM methods oversamples data similar to R (which represents the majority classes C), it is not surprising that SIM will outperform all baselines on these “rare classes” given that the baseline models do not have access to the additional information in R. Similarly, there could be bias in the over accuracy metric which favors SIM methods.\n\n* On the redundancy dataset, the advantage of SIM methods only starts to show up when there is large enough labeled data (as it relies on the conditioning on already-queried points to account for diversity). This means that one will have to find a sweet spot such that using the proposed method can be beneficial, which is not always possible.\n\n* Moreover, it seems that the SIMILAR methods do not perform as well in standard AL setting (appendix figure 6), indicating potential risk of applying such biased optimization. These limitations should be addressed more carefully.\n* The figures can be further improved by marking out SIMILAR-based methods as “(Ours)” as there are so many abbreviations of different methodologies and it is very easy to lose track of the result.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This reviewer did not find any potential negative societal impact.", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "YX8lBUXzTDy", "reviewer": "Reviewer_enP1", "summary": "This work attempts to address several situations that may occur during active learning processes, i.e., data imbalance, redundancy, rare classes and out-of-distribution with some submodular information measures.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "For methods: this work should be an application/extension of \"Submodular Combinatorial Information Measures with Applicationsin Machine Learning\". However, I don't agree with the application to the scenarios rare classes/data imbalance and OOD, since both rare classes/data imbalance and OOD cases be identified based on the assumption that the labels of data are known. However, in active learning processes, we assume that the labels could not be accessed at the beginning, therefore, one cannot determine if these scenarios exist. In machine learning, these exteremes are possible but in active learning, we have no idea to identify whether the unlabeled data pool containes rare classes/data imbalance, OOD scenarios.\n\nFor experiments: \n\n1) In OOD related experiments, it's unfair to exclude OOD data points in initial labelled set, using random selection would be better.\n\t\n2) In Figure. 3-5, the color choice is so poor that it’s really hard to distinguish some strategies from the others.\n\t\n3) The upper bound of labeled set size is too small in some cases, e.g., in Figure. 3 (b), it does not converge like Figure. 4 (b), the upper bound of labeled set size should be increased in case that the proposed method perform good in previous stages and not as good as baseline models in latter stages.\n\t\n4) The proposed SIMILAR method on standard AL (in Appendix), which is the \"Achilles heel\" of the proposed model, this suggests that it is difficult to put it into practical use with more general cases.\n\t\n\nTo sum up,  the submodular information measures are indeed very good ideas with strong theoretical foundations, but I'm still skeptical about applying it to active learning scenarios. Additionally, some experimental settings are not enough convincing.  \n\n\n/**************************/\nAfter reading the authors' response, I decide to keep my score. Additionally, the author mentioned that \"for instance, in medical imaging domains, images of cancer cells are often rare and are critical to classify correctly\", but they didn't have the related experiments/case study.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "LgPw_2UlbQX", "reviewer": "Reviewer_ryua", "summary": "In this paper, the authors propose methods for selecting a query set for batch active learning. These methods select a query set by maximizing a criterion based on submodular functions such as facility location function, graph cut function, and log-determinant function. The similarity matrix between data points, used in the definition of submodular functions, is defined by the inner product of the gradients of loss functions.\n\nThe authors empirically validate the efficiency of the proposed methods for batch active learning with rare classes, redundancy, and out-of-distribution data points. They adopt standard image datasets and standard deep neural network models. In each experimental setting, some of the proposed methods outperform existing methods for batch active learning.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The overall writing quality is very well. The main body is well organized, and all the descriptions are concise. The experiments are conducted thoroughly on standard datasets and models, which validate the superiority of the proposed methods.\n\nThe main contribution of this paper is criteria based on submodular functions for selecting a query set, but I do not fully support their originality and significance. The authors proposed three criteria: submodular mutual information (SMI), submodular conditional gain (SCG), and submodular conditional mutual information (SCMI). Applying these criteria to batch active learning does not seem to be completely novel. SCG is a standard criterion when a submodular utility function is used for batch active learning (e.g., Chen and Krause, 2013, Fern et al., 2017). SMI was proposed in previous work (Iyer et al., 2021). SCMI is just a combination of these two. Also, SMI and SCMI do not seem to satisfy submodularity, and then it does not seem easy to optimize it with theoretical guarantees. It loses the most prominent advantages of using submodular functions.\n\n---\n\nI read the authors' response. As the authors claimed, the novel part of this paper is to apply SMI and SCMI to batch active learning, but submodular optimization techniques cannot be used for these criteria except in some special cases. I am not fully convinced by the merit of the proposed framework, so I keep my score.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The discussion on the limitations and societal impact is adequate.", "review_text": "", "extra_scores": {"ethical_concerns": "There is no ethical concern.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "EXmMptFvyT1", "reviewer": "Reviewer_FgsV", "summary": "This paper proposes a \"unified active learning framework\". The key element of this framework is a submodular information measure, denoted by $I_f(A;Q|P)$ where $f$ is a submodular score function, $A$ is a set of unlabeled samples whose \"value\" is measured, and $P$ and $Q$ are set of samples to be specified by the algorithm as parameters. $I_f(A;Q|P)$ is supposed to measure similarity and dissimilarity among $A$, $P$, and $Q$. In each round of the proposed \"unified active learning framework\", it chooses $A$ that maximizes $I_f$. By choosing $P$ and $Q$ properly (for example, for the class imbalance task, setting $Q$ to be a small labeled sample set from rare classes), this paper shows empirically the proposed algorithm outperforms other active learning algorithms on various tasks, including standard AL, AL with class imbalance, AL with redundant data, and AL with out of distribution data.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\nIt is nice to see a unified AL framework that can address different AL tasks. This is novel as far as I can tell and can be interesting for the community. The main tool, submodular information measure (SIM), looks also a very interesting tool.\n\nCons:\n- My major concern is clarity, in particular the explanation of SIM and how it is used.\n  - It would be much clearer if the authors could walk through at least a few instantiations of SMI in table 2: what is the corresponding f? how are they derived? How can we understand them?\n  - Are there any connections between the proposed framework with any existing AL heuristics?\n  - Why, in line 146, the inner product between gradients a good similarity measure? Are there any other choices and how do they compare?\n- Experiment results look good, but seem unfair: comparisons are between *specialized* versions (adapted to specific learning scenarios by choosing proper parameters) of SIM vs other *general-purpose* AL algorithms. It would be interesting to see how the proposed method compares against known methods for rare classes, redundant data, out of distribution data. (for example, over/under sampling, tweaking loss the loss function).\n- As admitted in Section 3.5, a single iteration would take ~n^2 time where n=the number of unlabeled samples. This would be too expensive when n is not small. The paper provides some remedies, but I'm not sure how good they are (in terms of both accuracy and time). It would be more clearer if the author could provide comparison of running time in experiments to help the readers evaluate.\n\nOther issues:\n- confidence bar\n  - How are confidence bars defined and computed?\n  - Related, why for many plots the error bars shrink to a point (for example,figure 3(d), DIV-GCMI at labeled set size=1000), or even worse disappear (for example, figure 3(d) LOGDETMI after 1125)?\n  - Why don't we have them for image net dataset?\n- It would be more convincing if you could present experiment results with different choices of hyper parameters (for example, learning rates, batch sizes, etc.)\n\n--\n\nI've read the author feedback and I tend to keep my ratings for its current version. I think this paper has good potential, but it would be much better to me if it could provide more explanation for the proposed framework, and make it clear what is new and connection with existing methods. Regarding Q4, I'm not very sure if the methods you mentioned are standard baselines, but still comparing with simple (and effective IMHO) heuristics would make it more convincing.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See main review.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers appreciate the paper’s general idea in a unified AL framework that addresses different AL tasks (rare class, OOD data, and redundancy), and acknowledge the technical novelty in the acquisition function which (non-trivially) generalizes existing heuristics. However, there remains concern in the presentation clarity (e.g. justification of important assumptions such as accessibility to OoD data for certain applications, the significance of contributions w.r.t. existing work) and the experimental setup. Thus the reviewers were not convinced that the method is well justified in its current state to merit acceptance for publication.
Decision: Accept (Poster)