SIMILAR: Submodular Information Measures Based Active Learning In Realistic Scenarios

1 Introduction Deep neural networks ( DNNs ) have had a lot of success in a wide variety of domains . However , they require large labeled datasets which are often taxing , time-consuming , and expensive to obtain . Active learning ( AL ) [ 12 , 13 , 39 , 3 , 9 ] is a promising approach to solve this problem . It aims to select the most informative data points from an unlabeled dataset to be labeled in an adaptive manner with a human in the loop . The goal of AL is to achieve maximum accuracy of the model while minimizing the number of data points required to be labeled . Current AL methods have been tested in relatively simple , clean , and balanced datasets . However , real-world datasets are not clean and have a number of characteristics that makes learning from them challenging [ 10 , 46 , 47 , 38 , 1 , 8 ] . Firstly , these real-world datasets are im- balanced , and some classes are very rare ( e.g. , Fig 1 ( a ) ) . Examples of this imbalance are medical 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . imaging domains where the cancerous images are rare . Secondly , real-world data has a lot of redundancy ( e.g. , Fig 1 ( b ) ) . This redundancy is more prominent in datasets that are created by sampling frames from videos ( e.g. , footage from a car driving on a freeway or surveillance camera footage ) . Thirdly , it is common to have out-of-distribution ( OOD ) ( e.g. , Fig 1 ( c ) ) data , where some part of the unlabeled data is not of concern to the task at hand . Given the amount of unlabeled data , it is not realistic to assume that these datasets can be cleaned manually ; hence , it is the need of the hour to have active learning methods that are robust to such scenarios . We show that current AL approaches ( including the state-of-the-art approach BADGE [ 3 ] ) do not work well in the presence of the dataset biases described above . In this work , we address the following question : Can a machine learning model be trained using a single unified active learning framework that works for a broad spectrum of realistic scenarios ? As a solution , we propose SIMILAR1 , a unified active learning framework which enables active learning for many realistic scenarios like rare classes , out-of-distribution ( OOD ) data , and redundancy . 1.1 Related Work . Active learning has enabled efficient training of complex deep neural networks by decreasing labeling costs . The most commonly used approach is to select the most uncertain items . Examples of uncertainty strategies include ENTROPY [ 41 ] , LEAST CONFIDENCE [ 44 ] , and MARGIN [ 37 ] . One challenge of this approach is that all the samples within a batch can be potentially similar even though they are uncertain . To overcome this problem in batch active learning , many recent works have attempted to select diverse yet informative data points . [ 45 , 22 ] propose a simple approach : Filter a set of points using uncertainty sampling and then select a diverse subset from the filtered set . [ 40 ] propose CORESET , which forms core-sets using greedy k-center clustering while maintaining the geometric arrangement . BADGE [ 3 ] , another recent approach , proposes to select data points corresponding to high-magnitude , diverse hypothesized gradients by using K-MEANS++ [ 2 ] initialization to distance from previously selected data points in the batch . Most existing AL approaches fail to ensure diversity across AL selection rounds and do not perform as well when there is a lot of redundancy . Sinha et al . [ 42 ] used a variational autoencoder ( VAE ) [ 25 ] to learn a feature space and an adversarial network [ 32 ] to distinguish between labeled and unlabeled data points . However , their approach is computationally expensive and requires extensive hyperparameter tuning . Similarly , BATCHBALD [ 26 ] does not scale to larger batch sizes since their method would need a large number of Monte Carlo dropout samples to obtain a significant mutual information . Such limitations reduce the scope of applying these methods to realistic settings . Closely related to our work are two recently proposed works . The first is GLISTER-ACTIVE [ 24 ] , which formulates the AL acquisition function by maximizing the log-likelihood on a held-out validation set . This validation set could consist of examples from the rare classes or in-distribution examples . The second approach is the work of Gudovskiy et al . [ 15 ] , who study AL for biased datasets using a self-supervised FISHER kernel and pseudo-label estimators . They address this problem by explicitly minimizing the KL divergence between training and validation sets via maximizing the FISHER kernel . Although their method shows promising results , they make multiple unrealistic assumptions : a ) They use a large labeled validation set , and b ) they use feature representations from a model pretrained using unsupervised learning on a balanced unlabeled dataset . In this work , we compare against both GLISTER-ACTIVE [ 24 ] and FISHER [ 15 ] approaches in the more realistic setting of a small held-out validation set ( smaller than the seed labeled set ) and an imbalanced unlabeled set . Another work proposed a discrete optimization method for k-NN-type algorithms in the domain shift setting [ 6 ] . However , their approach is limited to k-NNs . This work utilizes submodular information measures ( SIM ) by [ 19 ] and their extensions by [ 23 ] . SIMs encompass submodular conditional mutual information ( SCMI ) , which can then be used to derive submodular mutual information ( SMI ) ; submodular conditional gain ( SCG ) ; and submodular functions ( SF ) . We discuss these functions in detail in Sec . 2 . [ 23 ] also studies these functions on the closely related problem of targeted data selection . 1.2 Our Contributions . The following are our main contributions : 1 ) Given the limitations of existing approaches in handling active learning in the real world , we propose SIMILAR ( Sec . 3 ) , a unified active learning framework that can serve as a comprehensive solution to multiple realistic scenarios . 2 ) We treat SIM as a 1Submodular Information Measures based actIve LeARning common umbrella for realistic active learning and study the effect of different function instantiations offered under SIM for various realistic scenarios . 3 ) SIMILAR not only handles standard active learning but also extends to a wide range of settings which appear in the real world such as rare classes , out-of-distribution ( OOD ) data , and datasets with a lot of redundancy . Finally , 4 ) we empirically demonstrate the effectiveness of SMI-based measures for image classification ( Sec . 4 ) in a number of realistic data settings including imbalanced , out-of-distribution , and redundant data . Specifically , in the case of imbalanced and OOD data , we show that SIMILAR achieves improvements of more than 5 to 10 % on several image classification datasets . 2 Background . In this section , we enumerate the different submodular functions that are covered under SIM and the relationships between them . Submodular Functions . We let U denote the unlabeled set of n data points U = { 1 , 2 , 3 , ... , n } and a set function f : 2U −→ R. Formally , a function f is submodular [ 14 ] if for x ∈ U , f ( A∪x ) −f ( A ) ≥ f ( B ∪ x ) − f ( B ) , ∀A ⊆ B ⊆ U and x /∈ B . For a set A ⊆ U , f ( A ) provides a real-valued score for A . In the context of batch active learning , this is the score of an acquisition function f on batch A. Submodularity is particularly appealing because it naturally occurs in real world applications [ 43 , 4 , 5 , 20 ] and also admits a constant factor 1− 1e [ 34 ] for cardinality constraint maximization . Additionally , variants of the greedy algorithm maximize a submodular function in near-linear time [ 33 ] . Submodular Mutual Information ( SMI ) . Given sets A , Q ⊆ U , the SMI [ 16 , 19 ] is defined as If ( A ; Q ) = f ( A ) + f ( Q ) − f ( A ∪ Q ) . Intuitively , SMI models the similarity between Q and A , and maximizing SMI will select points similar to Q while being diverse . Q here is the query set . Submodular Conditional Gain ( SCG ) . Given sets A , P ⊆ U , the SCG f ( A|P ) is the gain in function value by adding A to P . Thus , f ( A|P ) = f ( A∪ P ) − f ( P ) [ 19 ] . Intuitively , SCG models how different A is from P , and maximizing SCG functions will select data points not similar to the points in P while being diverse . We refer to P as the conditioning set . Submodular Conditional Mutual Information ( SCMI ) . Given sets A , Q , P ⊆ U , the SCMI is defined as If ( A ; Q|P ) = f ( A∪P ) + f ( Q∪P ) − f ( A∪Q∪P ) − f ( P ) . Intuitively , SCMI jointly models the similarity between A and Q and their dissimilarity with P . Relationship between SIM The relationship between the above measures is the key component that unifies our AL framework [ 19 , 23 ] . The unification comes from the rich modeling capacity of SCMI : If ( A ; Q|P ) where Q , P ⊆ U . This facilitates a single acquisition function that can be applied to multiple scenarios . Concretely , the submodular function f can be obtained by setting Q ← U and P ← ∅ . Next , the SMI can be obtained by setting Q ← Q and P ← ∅ , while we obtain SCG by setting Q ← ∅ , P ← P . We summarize the relationships between SIM in Tab . 1 . Instantiations of SIM . The formulations for Facility Location ( FL ) , Graph Cut ( GC ) and Log Determinant ( LOGDET ) are as in [ 19 , 23 ] and we adapt them as acquisition functions for batch active learning . We use two variants for FL : FLQMI , which models pairwise similarities of only the query set Q to the unlabeled dataset , and FLVMI , which additionally considers the pairwise similarities within the unlabeled dataset U . The SCG and SCMI expressions corresponding to FL are referred as FLCG and FLCMI , respectively ( see row 1 in Tab . 2a and 2b ) . For LOGDET , we refer to the SMI , SCG and SCMI expressions as LOGDETMI , LOGDETCG and LOGDETCMI , respectively ( see row 5 in Tab . 2a and row 2 in Tab . 2b ) . Similarly , the SMI and SCG expressions are respectively referred to as GCMI and GCCG for GC ( see row 3 in Tab . 2a and 2b ) . For notation in Tab . 2 , the pairwise similarity matrix S between items in sets A and B is denoted as SA , B . Also , we denote Sij as the ( i , j ) entry of S .

Summary: This paper proposes a "unified active learning framework". The key element of this framework is a submodular information measure, denoted by $I_f(A;Q|P)$ where $f$ is a submodular score function, $A$ is a set of unlabeled samples whose "value" is measured, and $P$ and $Q$ are set of samples to be specified by the algorithm as parameters. $I_f(A;Q|P)$ is supposed to measure similarity and dissimilarity among $A$, $P$, and $Q$. In each round of the proposed "unified active learning framework", it chooses $A$ that maximizes $I_f$. By choosing $P$ and $Q$ properly (for example, for the class imbalance task, setting $Q$ to be a small labeled sample set from rare classes), this paper shows empirically the proposed algorithm outperforms other active learning algorithms on various tasks, including standard AL, AL with class imbalance, AL with redundant data, and AL with out of distribution data.