Less data is more: Selecting informative and diverse subsets with balancing constraints

1 INTRODUCTION . Deep learning has shown unprecedented success in many domains , such as speech ( Hinton et al. , 2012 ) , computer vision ( Krizhevsky et al. , 2012 ; Szegedy et al. , 2015 ; He et al. , 2016 ) , and natural language processing ( Sutskever et al. , 2014 ; Devlin et al. , 2018 ) . This success generally relies on access to significant computational resources and large human-annotated datasets . For example , energy consumption and carbon footprint of developing common NLP models are shown to be comparable to the lifetime emissions of a car ( Strubell et al. , 2019 ) . Similarly , human annotation is also a time-consuming and expensive process ; Badrinarayanan et al . ( 2010 ) reports that semantic labeling of a single video frame takes around 45-60 minutes . In this paper we ask the following question : Given a large unlabeled dataset along with a small labeled ( seed ) dataset and an annotation budget , how do we select a subset of the unlabeled dataset , which , when annotated , will achieve the best performance ? While this is addressed by many classical active learning methods ( Settles , 2009 ) , in this work we focus on the modern deep learning setting with CNNs and very deep networks such as ResNets ( He et al. , 2016 ) . Typical active learning methods employ an iterative process where a single image is labeled and used to update the model at each step . The updated model is then used to pick the next image and so on . Such an approach is not feasible for deep networks . We therefore choose to study this problem in a batch setting ( Sener & Savarese , 2018 ; Zhdanov , 2019 ; Shui et al. , 2020 ; Kim et al. , 2021 ; Ghorbani et al. , 2021 ) . In this setting , we first train an initial model using the labeled seed dataset . We typically use a randomly selected 10 % subset of the full dataset as seed . After this , as done in the standard batch-mode active learning setup , our algorithm uses this initial seed model to select a subset of the full dataset for which the labels will be revealed . We then train a new model with the selected subset . As shown in Fig . 1 , our subset selection methods can identify subsets with 30 % to 40 % less data , which achieve performance similar to what we could get training with the full annotated dataset . Most existing methods for data subset selection use some notion of margin , representativeness , or diversity . For example , the classical margin-based sampling selects images for which the prediction of the class labels are most uncertain . These could , for example , be the images having smallest difference between the highest and the second highest label probabilities ( Lewis & Gale , 1994 ) . Clustering-based objectives such as k-center ( Sener & Savarese , 2018 ) or k-medoids ( Kaufman & Rousseeuw , 1987 ) select cluster centers with rich representativeness to model the entire dataset . Diverse subsets are identified by casting the problem as the maximization of submodular functions ( Wei et al. , 2015 ) , which are discrete analogues of convex functions . We see two main limitations with existing methods . First , there is no single subset selection criterion that achieves the best performance on different classification datasets . For example we observe in our experiments ( Sec . 6 ) that in certain datasets , diversity works better than margin , and vice versa . The second limitation is particularly relevant to margin-based methods where the selection is driven by the closeness of an image to a decision boundary . In a classification problem with L labels , we have ( L 2 ) possible decision boundaries . Most margin-based methods greedily select images close to decision boundaries . This often results in a selection of too many images representing a small number of classes and boundaries , while ignoring many others . To address the first limitation , we develop a unified algorithm based on maximization of a submodular function , which can combine different selection objectives . Importantly , we do this without sacrificing constant approximation guarantees . To address the second limitation , we incorporate class-balancing and boundary-balancing constraints in our optimization using intersection of matroids . Typical diversity promoting objective functions use an underlying nearest neighbor graph that models the distances between the training samples . In this work , We present an improved diversity objective function that models the simultaneous interaction of three nearby samples , while the standard approach typically considers the interaction of two nearby samples . To summarize , our contributions are : ( 1 ) We develop a unified algorithm to combine different objective functions along with class-balancing and boundary-balancing constraints using the intersection of matroids . ( 2 ) We propose an improved diversity objective function based on 3-clique interactions in the nearest neighbor graph . ( 3 ) We outperform baseline methods on three standard classification datasets ( CIFAR-10 , CIFAR-100 , and ImageNet ) and a long-tailed classification dataset ( CIFAR-100LT ) . 2 RELATED WORK . Data summarization and submodularity : Submodularity ( Nemhauser et al. , 1978 ) , often seen as the discrete analogue of convexity , can be used to model subset selection objectives associated with diversity and representativeness . While subset selection problems are mostly NP-hard , submodular optimization algorithms are efficient and come with constant approximation guarantees ( Carbonell & Goldstein , 1998 ; Simon et al. , 2007 ; Krause & Golovin ; Golovin & Krause , 2010 ; Wei et al. , 2015 ; Lin & Bilmes , 2011 ; Kaushal et al. , 2019 ; Kim et al. , 2016 ; Prasad et al. , 2014 ; Golovin & Krause , 2010 ; 2011 ; Joseph et al. , 2019 ; Zhou & Spanos , 2016 ; Qian et al. , 2017 ; Das & Kempe , 2018 ; Elhamifar , 2019 ; Kothawade et al. , 2021 ; Kaushal et al. , 2021 ) . A submodular function that promotes representativeness and diversity was utilized in document summarization ( Lin & Bilmes , 2011 ) . Discriminant point process ( DPP ) are probabilistic models capable of selecting diverse subsets . They were used for minibatch selection in Mariet et al . ( 2020 ) , and the maximum a posteriori ( MAP ) inference for DPP is submodular ( Chen et al. , 2018 ) . Uncertainty sampling and active learning : Active learning methods primarily focus on reducing the label annotation costs by selecting images to label that are more likely to yield the best model ( See Settles ( 2009 ) for a detailed survey ) . Theoretical guarantees are usually about how quickly the version space , that maintains a set of good candidate classifiers , shrinks based on the availability of new data ( Dasgupta et al. , 2008 ) . Uncertainty sampling selects the most challenging or uncertain images first , in the hope that this will eliminate the need to label other , easier ones . In a classification setting , predictions from the initial model can provide class probabilities that can help identify images with high uncertainty . Possibilities include simple uncertainty measures from the best and second best class probabilities ( Lewis & Gale , 1994 ; Scheffer et al. , 2001 ) , entropy measure ( Holub et al. , 2008 ; Joshi et al. , 2009 ) , and geometric distance to the decision boundary ( Tong & Koller , 2002 ; Brinker , 2003 ) , selection via proxies ( Coleman et al. , 2020 ) , and query by committee ( Gilad-Bachrach et al. , 2005 ; Seung et al. , 1992 ) , where a committee of models are used to identify images where they most disagree . Coresets : A coreset is a small set of points that approximates the shape of a larger point set . In an ML setting it is a small subset of images from a larger dataset such that the model trained on the coreset is competitive with the one learned from the entire dataset . It is shown in ( Sener & Savarese , 2018 ; Wolf , 2011 ) that the average loss over any given subset of the dataset and the remaining points can be bounded , and that the minimization of this bound can be mapped to the k-center problem . Clustering : Other classical techniques for subset selection include clustering algorithms such as k-medoids ( Kaufman & Rousseeuw , 1987 ; Park & Jun , 2009 ) , which minimizes the sum of dissimilarities between images belonging to a cluster and a point designated as its center . Agglomerative clustering was used to select subsets in Birodkar et al . ( 2019 ) , where it was shown that at least 10 % of images are redundant in ImageNet and CIFAR-10 training . In this work , we show that at least 30-40 % of the images in CIFAR-10 , CIFAR-100 and ImageNet are redundant . In Ash et al . ( 2020 ) , k-MEANS++ algorithm is used to find a set of diverse and uncertain samples using the gradients of the final layer of the network . In the context of interpretability , prototypes are subsets that best represent the entire dataset where a learned model performs the best , and criticisms are the images where a learned model does not do well . Both prototypes and criticisms can be modeled using maximum-mean discrepancy ( MMD ) , that measures the difference between two distributions . The resulting optimization to compute the subsets can be cast as submodular maximization ( Kim et al. , 2016 ) . Minimization of the MMD between the entire dataset and the selected subset can be seen as empirical risk minimization , and such an approach was used for batch mode active learning ( Wang & Ye , 2015 ) . Other classical techniques for prototype selection includes k-medoids ( Bien & Tibshirani , 2011 ) clustering methods . 3 PROBLEM STATEMENT . We consider a classification setting with L classes and n training images U = { x1 , . . . , xn } . Let G = { 1 , . . . , n } denote the indices in the dataset . Let ψ : X → RL be a ( trained ) neural network that maps an input image to a probability distribution over the L classes . The model ψ can be seen as the composition of an embedding function φ : X → RD that maps an input to an D-dimensional embedding , and a discriminator function h : RD → RL that maps the embedding to output probability predictions . Subset selection and model evaluation . We assume that we are given a small , annotated subset of U , which we refer to as the seed dataset ( this is 10 % of U in our experiments ) . We use this seed dataset to train an initial model ψseed . Our goal is to use this initial model to select a subset S of G , subject to a given annotation budget , such that when the images in S are annotated we can get the best classification performance training a new model with them . To this end , using the seed model , we generate embeddings and predictions for all the input images in U . Next , we construct a neighborhood graph ( G , E ) where E denotes the weighted set of edges associated with nearby training images in the embedding space RD . Our subset selection algorithm only depends on these embeddings , predictions , and the nearest-neighbor graph ( G , E ) . As in the standard active learning settings , we do not use the groundtruth labels in U while selecting subsets . For modeling subset selection , we employ a submodular set function f : 2G → R that provides a utility value for every subset of G using notions of uncertainty and diversity . We show that the class and boundary balancing constraints can be enforced using intersections of partition matroids . With this modeling , subset selection is essentially the task of finding the subset with maximum utility value while satisfying the matroid constraints . This inference problem is combinatorial in the number of training samples , and the brute-force solution is infeasible . For efficient inference , we rely on the submodular property of the objective function and the use of matroids for modeling balancing constraints . Such an approach allows us to use an efficient greedy algorithm to maximize the objective function while satisfying matroid constraints with constant approximation guarantees . Once the subset S is computed , we request the groundtruth labels for the images in S. We use the annotated subset US ⊆ U to train a model ψS and evaluate its performance on a common held-out test set .

Summary: Most batch-mode active learning strategies involve maximizing a sub-modular score function of the value of each image to be labeled. This paper demonstrates that current methods fail to sample diverse classes or images near decision boundaries, arguably one of the most important regions to obtain label information. The paper proposes a technique for incorporating class-balance and boundary-balance constraints to the sub-modular optimization problem and performs experiments over several image data sets to show the value of class and boundary balancing.