Less data is more: Selecting informative and diverse subsets with balancing constraints

1 INTRODUCTION . Deep learning has shown unprecedented success in many domains , such as speech ( Hinton et al. , 2012 ) , computer vision ( Krizhevsky et al. , 2012 ; Szegedy et al. , 2015 ; He et al. , 2016 ) , and natural language processing ( Sutskever et al. , 2014 ; Devlin et al. , 2018 ) . This success generally relies on access to significant computational resources and large human-annotated datasets . For example , energy consumption and carbon footprint of developing common NLP models are shown to be comparable to the lifetime emissions of a car ( Strubell et al. , 2019 ) . Similarly , human annotation is also a time-consuming and expensive process ; Badrinarayanan et al . ( 2010 ) reports that semantic labeling of a single video frame takes around 45-60 minutes . In this paper we ask the following question : Given a large unlabeled dataset along with a small labeled ( seed ) dataset and an annotation budget , how do we select a subset of the unlabeled dataset , which , when annotated , will achieve the best performance ? While this is addressed by many classical active learning methods ( Settles , 2009 ) , in this work we focus on the modern deep learning setting with CNNs and very deep networks such as ResNets ( He et al. , 2016 ) . Typical active learning methods employ an iterative process where a single image is labeled and used to update the model at each step . The updated model is then used to pick the next image and so on . Such an approach is not feasible for deep networks . We therefore choose to study this problem in a batch setting ( Sener & Savarese , 2018 ; Zhdanov , 2019 ; Shui et al. , 2020 ; Kim et al. , 2021 ; Ghorbani et al. , 2021 ) . In this setting , we first train an initial model using the labeled seed dataset . We typically use a randomly selected 10 % subset of the full dataset as seed . After this , as done in the standard batch-mode active learning setup , our algorithm uses this initial seed model to select a subset of the full dataset for which the labels will be revealed . We then train a new model with the selected subset . As shown in Fig . 1 , our subset selection methods can identify subsets with 30 % to 40 % less data , which achieve performance similar to what we could get training with the full annotated dataset . Most existing methods for data subset selection use some notion of margin , representativeness , or diversity . For example , the classical margin-based sampling selects images for which the prediction of the class labels are most uncertain . These could , for example , be the images having smallest difference between the highest and the second highest label probabilities ( Lewis & Gale , 1994 ) . Clustering-based objectives such as k-center ( Sener & Savarese , 2018 ) or k-medoids ( Kaufman & Rousseeuw , 1987 ) select cluster centers with rich representativeness to model the entire dataset . Diverse subsets are identified by casting the problem as the maximization of submodular functions ( Wei et al. , 2015 ) , which are discrete analogues of convex functions . We see two main limitations with existing methods . First , there is no single subset selection criterion that achieves the best performance on different classification datasets . For example we observe in our experiments ( Sec . 6 ) that in certain datasets , diversity works better than margin , and vice versa . The second limitation is particularly relevant to margin-based methods where the selection is driven by the closeness of an image to a decision boundary . In a classification problem with L labels , we have ( L 2 ) possible decision boundaries . Most margin-based methods greedily select images close to decision boundaries . This often results in a selection of too many images representing a small number of classes and boundaries , while ignoring many others . To address the first limitation , we develop a unified algorithm based on maximization of a submodular function , which can combine different selection objectives . Importantly , we do this without sacrificing constant approximation guarantees . To address the second limitation , we incorporate class-balancing and boundary-balancing constraints in our optimization using intersection of matroids . Typical diversity promoting objective functions use an underlying nearest neighbor graph that models the distances between the training samples . In this work , We present an improved diversity objective function that models the simultaneous interaction of three nearby samples , while the standard approach typically considers the interaction of two nearby samples . To summarize , our contributions are : ( 1 ) We develop a unified algorithm to combine different objective functions along with class-balancing and boundary-balancing constraints using the intersection of matroids . ( 2 ) We propose an improved diversity objective function based on 3-clique interactions in the nearest neighbor graph . ( 3 ) We outperform baseline methods on three standard classification datasets ( CIFAR-10 , CIFAR-100 , and ImageNet ) and a long-tailed classification dataset ( CIFAR-100LT ) . 2 RELATED WORK . Data summarization and submodularity : Submodularity ( Nemhauser et al. , 1978 ) , often seen as the discrete analogue of convexity , can be used to model subset selection objectives associated with diversity and representativeness . While subset selection problems are mostly NP-hard , submodular optimization algorithms are efficient and come with constant approximation guarantees ( Carbonell & Goldstein , 1998 ; Simon et al. , 2007 ; Krause & Golovin ; Golovin & Krause , 2010 ; Wei et al. , 2015 ; Lin & Bilmes , 2011 ; Kaushal et al. , 2019 ; Kim et al. , 2016 ; Prasad et al. , 2014 ; Golovin & Krause , 2010 ; 2011 ; Joseph et al. , 2019 ; Zhou & Spanos , 2016 ; Qian et al. , 2017 ; Das & Kempe , 2018 ; Elhamifar , 2019 ; Kothawade et al. , 2021 ; Kaushal et al. , 2021 ) . A submodular function that promotes representativeness and diversity was utilized in document summarization ( Lin & Bilmes , 2011 ) . Discriminant point process ( DPP ) are probabilistic models capable of selecting diverse subsets . They were used for minibatch selection in Mariet et al . ( 2020 ) , and the maximum a posteriori ( MAP ) inference for DPP is submodular ( Chen et al. , 2018 ) . Uncertainty sampling and active learning : Active learning methods primarily focus on reducing the label annotation costs by selecting images to label that are more likely to yield the best model ( See Settles ( 2009 ) for a detailed survey ) . Theoretical guarantees are usually about how quickly the version space , that maintains a set of good candidate classifiers , shrinks based on the availability of new data ( Dasgupta et al. , 2008 ) . Uncertainty sampling selects the most challenging or uncertain images first , in the hope that this will eliminate the need to label other , easier ones . In a classification setting , predictions from the initial model can provide class probabilities that can help identify images with high uncertainty . Possibilities include simple uncertainty measures from the best and second best class probabilities ( Lewis & Gale , 1994 ; Scheffer et al. , 2001 ) , entropy measure ( Holub et al. , 2008 ; Joshi et al. , 2009 ) , and geometric distance to the decision boundary ( Tong & Koller , 2002 ; Brinker , 2003 ) , selection via proxies ( Coleman et al. , 2020 ) , and query by committee ( Gilad-Bachrach et al. , 2005 ; Seung et al. , 1992 ) , where a committee of models are used to identify images where they most disagree . Coresets : A coreset is a small set of points that approximates the shape of a larger point set . In an ML setting it is a small subset of images from a larger dataset such that the model trained on the coreset is competitive with the one learned from the entire dataset . It is shown in ( Sener & Savarese , 2018 ; Wolf , 2011 ) that the average loss over any given subset of the dataset and the remaining points can be bounded , and that the minimization of this bound can be mapped to the k-center problem . Clustering : Other classical techniques for subset selection include clustering algorithms such as k-medoids ( Kaufman & Rousseeuw , 1987 ; Park & Jun , 2009 ) , which minimizes the sum of dissimilarities between images belonging to a cluster and a point designated as its center . Agglomerative clustering was used to select subsets in Birodkar et al . ( 2019 ) , where it was shown that at least 10 % of images are redundant in ImageNet and CIFAR-10 training . In this work , we show that at least 30-40 % of the images in CIFAR-10 , CIFAR-100 and ImageNet are redundant . In Ash et al . ( 2020 ) , k-MEANS++ algorithm is used to find a set of diverse and uncertain samples using the gradients of the final layer of the network . In the context of interpretability , prototypes are subsets that best represent the entire dataset where a learned model performs the best , and criticisms are the images where a learned model does not do well . Both prototypes and criticisms can be modeled using maximum-mean discrepancy ( MMD ) , that measures the difference between two distributions . The resulting optimization to compute the subsets can be cast as submodular maximization ( Kim et al. , 2016 ) . Minimization of the MMD between the entire dataset and the selected subset can be seen as empirical risk minimization , and such an approach was used for batch mode active learning ( Wang & Ye , 2015 ) . Other classical techniques for prototype selection includes k-medoids ( Bien & Tibshirani , 2011 ) clustering methods . 3 PROBLEM STATEMENT . We consider a classification setting with L classes and n training images U = { x1 , . . . , xn } . Let G = { 1 , . . . , n } denote the indices in the dataset . Let ψ : X → RL be a ( trained ) neural network that maps an input image to a probability distribution over the L classes . The model ψ can be seen as the composition of an embedding function φ : X → RD that maps an input to an D-dimensional embedding , and a discriminator function h : RD → RL that maps the embedding to output probability predictions . Subset selection and model evaluation . We assume that we are given a small , annotated subset of U , which we refer to as the seed dataset ( this is 10 % of U in our experiments ) . We use this seed dataset to train an initial model ψseed . Our goal is to use this initial model to select a subset S of G , subject to a given annotation budget , such that when the images in S are annotated we can get the best classification performance training a new model with them . To this end , using the seed model , we generate embeddings and predictions for all the input images in U . Next , we construct a neighborhood graph ( G , E ) where E denotes the weighted set of edges associated with nearby training images in the embedding space RD . Our subset selection algorithm only depends on these embeddings , predictions , and the nearest-neighbor graph ( G , E ) . As in the standard active learning settings , we do not use the groundtruth labels in U while selecting subsets . For modeling subset selection , we employ a submodular set function f : 2G → R that provides a utility value for every subset of G using notions of uncertainty and diversity . We show that the class and boundary balancing constraints can be enforced using intersections of partition matroids . With this modeling , subset selection is essentially the task of finding the subset with maximum utility value while satisfying the matroid constraints . This inference problem is combinatorial in the number of training samples , and the brute-force solution is infeasible . For efficient inference , we rely on the submodular property of the objective function and the use of matroids for modeling balancing constraints . Such an approach allows us to use an efficient greedy algorithm to maximize the objective function while satisfying matroid constraints with constant approximation guarantees . Once the subset S is computed , we request the groundtruth labels for the images in S. We use the annotated subset US ⊆ U to train a model ψS and evaluate its performance on a common held-out test set .

Summary: The paper describes a new active learning algorithm for classification problems. The authors propose a selection criteria that trades off between uncertainty and diversity. The novelty lies in formulating a three-pronged approach to diversity that still manages to maintain submodularity of the selection criteria: 1) a traditional diversity metric based on cosine similarity of embeddings 2) a new triplet/clique "loss" that penalizes the selection of nearby points 3) class-balancing and boundary-balancing constraints  By showing this three-pronged approach maintains submodularity, they are able to devise a greedy algorithm that still has a constant approximation guarantee to the optimal selection subset (as measured by the selection criteria). Experiments on CIFAR-10, CIFAR-100, ImageNet, and CIFAR-100LT (long-tailed) suggest that the new approach performs as well as or better than state-of-the-art methods.