Similarity Search for Efficient Active Learning and Search of Rare Concepts

1 INTRODUCTION . Large-scale unlabeled datasets contain millions or billions of examples spread over a wide variety of underlying concepts ( Chelba et al. , 2013 ; Zhu et al. , 2015 ; Zhang et al. , 2015 ; Wan et al. , 2019 ; Russakovsky et al. , 2015 ; Kuznetsova et al. , 2020 ; Thomee et al. , 2016 ; Abu-El-Haija et al. , 2016 ; Caesar et al. , 2019 ; Lee et al. , 2019 ) . Often , these massive datasets skew towards a relatively small number of common concepts , such as cats , dogs , and people ( Liu et al. , 2019 ; Zhang et al. , 2017 ; Wang et al. , 2017 ; Van Horn & Perona , 2017 ) . Rare concepts , such as harbor seals , may only appear in a small fraction of the data ( less than 1 % ) . However , in many settings , performance on these rare concepts is critical . For example , harmful or malicious content may comprise a small percentage of user-generated content , but it can have an outsize impact on the overall user experience ( Wan et al. , 2019 ) . Similarly , when debugging model behavior for safety-critical applications like autonomous vehicles , or when dealing with representational biases in models , obtaining data that captures rare concepts allows machine learning practitioners to combat blind spots in model performance ( Karpathy , 2018 ; Holstein et al. , 2019 ; Ashmawy et al. , 2019 ; Karpathy , 2020 ) . Even a simple prediction task like stop sign detection can be challenging given the diversity of real-world data . Stop signs may appear in a variety of conditions ( e.g. , on a wall or held by a person ) , be heavily occluded , or have modifiers ( e.g. , “ Except Right Turns ” ) ( Karpathy , 2020 ) . While large-scale datasets are core to addressing these issues , finding the relevant examples for these long-tail tasks is challenging . Active learning and search have the potential to automate the process of identifying these rare , high value data points significantly , but existing methods become intractable at this scale . Specifically , the goal of active learning is to reduce the cost of labeling ( Settles , 2012 ) . To this end , the learning algorithm is allowed to choose which data to label based on uncertainty ( e.g. , the entropy of predicted class probabilities ) or other heuristics ( Settles , 2011 ; 2012 ; Lewis & Gale , 1994 ) . Active search is a sub-area focused on finding positive examples in skewed distributions ( Garnett et al. , 2012 ) . Because of a concentrated focus on labeling costs , existing techniques , such as uncertainty sampling ( Lewis & Gale , 1994 ) or information density ( Settles & Craven , 2008 ) , perform multiple selection rounds and iterate over the entire unlabeled data to identify the optimal example or batch of examples to label and scale linearly or even quadratically with the size of the unlabeled data . Computational efficiency is becoming an impediment as the size of datasets and model complexities have increased ( Amodei & Hernandez , 2018 ) . Recent work has tried to address this problem with sophisticated methods to select larger and more diverse batches of examples in each selection round and reduce the total number of rounds needed to reach the target labeling budget ( Sener & Savarese , 2018 ; Kirsch et al. , 2019 ; Coleman et al. , 2020 ; Pinsler et al. , 2019 ; Jiang et al. , 2018 ) . Nevertheless , these approaches still scan over all of the examples to find the optimal examples to label in each round and can be intractable for large-scale unlabeled datasets . For example , running a single inference pass over 10 billion images with ResNet-50 ( He et al. , 2016 ) would take 38 exaFLOPs . In this work , we propose Similarity search for Efficient Active Learning and Search ( SEALS ) to restrict the candidates considered in each selection round and vastly reduce the computational complexity of active learning and search methods . Empirically , we find that learned representations from pre-trained models can effectively cluster many unseen and rare concepts . We exploit this latent structure to improve the computational efficiency of active learning and search methods by only considering the nearest neighbors of the currently labeled examples in each selection round . This can be done transparently for many selection strategies making SEALS widely applicable . Finding the nearest neighbors for each labeled example in unlabeled data can be performed efficiently with sublinear retrieval times ( Charikar , 2002 ) and sub-second latency on billion-scale datasets ( Johnson et al. , 2017 ) for approximate approaches . While constructing the index for similarity search requires at least a linear pass over the unlabeled data , this computational cost is effectively amortized over many selection rounds or other applications . As a result , our SEALS approach enables selection to scale with the size of the labeled data rather than the size of the unlabeled data , making active learning and search tractable on datasets with billions of unlabeled examples . We empirically evaluated SEALS for both active learning and search on three large scale computer vision datasets : ImageNet ( Russakovsky et al. , 2015 ) , OpenImages ( Kuznetsova et al. , 2020 ) , and a proprietary dataset of 10 billion images from a large internet company . We selected 611 concepts spread across these datasets that range in prevalence from 0.203 % to 0.002 % ( 1 in 50,000 ) of the training examples . We evaluated three selection strategies for each concept : max entropy uncertainty sampling ( Lewis & Gale , 1994 ) , information density ( Settles & Craven , 2008 ) , and most-likely positive ( Warmuth et al. , 2002 ; 2003 ; Jiang et al. , 2018 ) . Across datasets , selection strategies , and concepts , SEALS achieved similar model quality and nearly the same recall of the positive examples as the baseline approaches , while improving the computational complexity by orders of magnitude . On ImageNet with a budget of 2,000 binary labels per concept ( ˜0.31 % of the unlabeled data ) , all baseline and SEALS approaches were within 0.011 mAP of full supervision and recalled over 50 % of the positive examples . On OpenImages , SEALS reduced the candidate pool to 1 % of the unlabeled data on average while remaining within 0.013 mAP and 0.1 % recall of the baseline approaches . On the proprietary dataset with 10 billion images , SEALS needed an even smaller fraction of the data , about 0.1 % , to match the baseline , which allowed SEALS to run on a single machine rather than a cluster . To the best of our knowledge , no other works have performed active learning at this scale . We also applied SEALS to the NLP spoiler detection dataset Goodreads ( Wan et al. , 2019 ) , where it achieved the same recall as the baseline approaches while only considering less than 1 % of the unlabeled data . Together , these results demonstrate that SEALS ’ improvements to computational efficiency make active learning and search tractable for even billion-scale datasets . 2 RELATED WORK . Active learning ’ s iterative retraining combined with the high computational complexity of deep learning models has led to significant work on computational efficiency ( Sener & Savarese , 2018 ; Kirsch et al. , 2019 ; Pinsler et al. , 2019 ; Coleman et al. , 2020 ; Yoo & Kweon , 2019 ; Mayer & Timofte , 2020 ; Zhu & Bento , 2017 ) . One branch of recent work has focused on selecting large batches of data to minimize the amount of retraining and reduce the number of selection rounds necessary to reach a target budget ( Sener & Savarese , 2018 ; Kirsch et al. , 2019 ; Pinsler et al. , 2019 ) . These approaches introduce novel techniques to avoid selecting highly similar or redundant examples and ensure the batches are both informative and diverse . In comparison , our work aims to reduce the number of examples considered in each selection round and complements existing work on batch active learning . Many of these approaches sacrifice computational complexity to ensure diversity , and their selection methods can scale quadratically with the size of the unlabeled data . Combined with our method , these selection methods scale with the size of the labeled data rather than the unlabeled data . Outside of batch active learning , other work has tried to improve computational efficiency by either using much smaller models as cheap proxies during selection ( Yoo & Kweon , 2019 ; Coleman et al. , 2020 ) or by generating examples ( Mayer & Timofte , 2020 ; Zhu & Bento , 2017 ) . Using a smaller model reduces the amount of computation per example , but unlike our approach , it still requires making multiple passes over the entire unlabeled pool of examples . The generative approaches ( Mayer & Timofte , 2020 ; Zhu & Bento , 2017 ) , however , enable sub-linear runtime complexity like our approach . Unfortunately , they struggle to match the label-efficiency of traditional approaches because the quality of the generated examples is highly variable . Active search is a sub-area of active learning that focuses on highly skewed class distributions ( Garnett et al. , 2012 ; Jiang et al. , 2017 ; 2018 ; 2019 ) . Rather than optimizing for model quality , active search aims to find as many examples from the minority class as possible . Prior work has focused on applications such as drug discovery , where the dataset sizes are limited , and labeling costs are exceptionally high . Our work similarly focuses on skewed distributions . However , we consider novel active search settings in image and text where the available unlabeled datasets are much larger , and computational efficiency is a significant bottleneck . k nearest neighbor ( k-NN ) classifiers are popular models in active learning and search because they do not require an explicit training phase ( Joshi et al. , 2012 ; Wei et al. , 2015 ; Garnett et al. , 2012 ; Jiang et al. , 2017 ; 2018 ) . The prediction and score for each unlabeled example can be updated immediately after each new batch of labels . In comparison , our SEALS approach uses k-NN algorithms for similarity search to create and expand the candidate pool and not as a classifier . This is an important but subtle difference . While prior work avoids expensive training by using k-NN classifiers , these approaches still require evaluating all of the unlabeled examples , which can still be prohibitively expensive on large-scale datasets like the ones we consider here . SEALS targets the selection phase rather than training , presenting a novel and complementary approach . 3 METHODS . In this section , we outline the problems of active learning ( Section 3.1 ) and search ( Section 3.2 ) formally as well as the selection methods we accelerate using SEALS . For both , we examine the pool-based setting , where all of the unlabeled data is available at once , and examples are selected in batches to improve computational efficiency , as mentioned above . Then in Section 3.3 , we describe our SEALS approach and how it further improves computational efficiency in both settings . 3.1 ACTIVE LEARNING . Pool-based active learning is an iterative process that begins with a large pool of unlabeled data U = { x1 , . . . , xn } . Each example is sampled from the space X with an unknown label from the label space Y = { 1 , . . . , C } as ( xi , yi ) . We additionally assume a feature extraction function Gz to embed each xi as a latent variable Gz ( xi ) = zi and that the C concepts are unequally distributed . Specifically , there are one or more valuable rare concepts R ⊂ C that appear in less than 1 % of the unlabeled data . For simplicity , we frame this as |R| binary classification problems solved independently rather than 1 multi-class classification problem with |R| concepts . Initially , each rare concept has a small number of positive examples and several negative examples that serve as a labeled seed set L0r . The goal of active learning is to take this seed set and select up to a budget of T examples to label that produce a model ATr that achieves low error . For each round t in pool-based active learning , the most informative examples are selected according to the selection strategy φ from a pool of candidate examples Pr in batches of size b and labeled , as shown in Algorithm 1 . For the baseline approach , Pr = { Gz ( x ) | x ∈ U } , meaning that all the unlabeled examples are considered to find the global optimal according to φ . Between each round , the model Atr is trained on all of the labeled data Ltr , allowing the selection process to adapt . In this paper , we considered max entropy ( MaxEnt ) uncertainty sampling ( Lewis & Gale , 1994 ) : φMaxEnt ( z ) = − ∑ ŷ P ( ŷ|z ; Ar ) logP ( ŷ|z ; Ar ) and information density ( ID ) ( Settles & Craven , 2008 ) : φID ( z ) = φMaxEnt ( z ) × 1 |Pr| ∑ zp∈Pr sim ( z , zp ) β where sim ( z , zp ) is the cosine similarity of the embedded examples and β = 1 . Note that for binary classification , max entropy is equivalent to least confidence and margin sampling , which are also popular criteria for uncertainty sampling ( Settles , 2009 ) . While max entropy uncertainty sampling only requires a linear pass over the unlabeled data , ID scales quadratically with |U | because it weights each example ’ s informativeness by its similarity to all other examples . To improve computational performance , the average similarity score for each example can be cached after the first selection round , so subsequent rounds scale linearly . This optimization only works whenGz is fixed and would not apply to dynamic similarity calculations like those in Sener & Savarese ( 2018 ) . We explored the greedy k-centers approach from Sener & Savarese ( 2018 ) but found that it never outperformed random sampling for our experimental setup . Unlike MaxEnt and ID , k-centers does not consider the predicted labels . It tries to achieve high coverage over the entire candidate pool , of which rare concepts make up a small fraction by definition , making it ineffective for our setting . Algorithm 1 BASELINE APPROACH Input : unlabeled data U , labeled seed set L0r , feature extractor Gz , selection strategy φ ( · ) , batch size b , labeling budget T 1 : Lr = { ( Gz ( x ) , y ) | ( x , y ) ∈ L0r } 2 : Pr = { Gz ( x ) | x ∈ U and ( x , · ) 6∈ L0r } 3 : repeat 4 : Ar = train ( Lr ) 5 : for 1 to b do 6 : z∗ = argmaxz∈Pr φ ( z ) 7 : Lr = Lr ∪ { ( z∗ , label ( x∗ ) ) } 8 : Pr = Pr \ { z∗ } 9 : end for 10 : until |Lr| = T Algorithm 2 SEALS APPROACH Input : unlabeled data U , labeled seed set L0r , feature extractor Gz , selection strategy φ ( · ) , batch size b , labeling budget T , knearest neighbors implementation N ( · , · ) 1 : Lr = { ( Gz ( x ) , y ) | ( x , y ) ∈ L0r } 2 : Pr = ∪ ( z , y ) ∈LrN ( z , k ) 3 : repeat 4 : Ar = train ( Lr ) 5 : for 1 to b do 6 : z∗ = argmaxz∈Pr φ ( z ) 7 : Lr = Lr ∪ { ( z∗ , label ( x∗ ) ) } 8 : Pr = ( Pr \ { z∗ } ) ∪N ( z∗ , k ) 9 : end for 10 : until |Lr| = T

Summary: This paper proposes an active learning and active search approach that targets samples for rare classes in very large unlabeled datasets with highly imbalanced class distributions. This is a common scenario in real-world applications, where these rare situations can be critical to accurately categorize - ie endangered species. The authors propose an approach that targets these rare cases while reducing the number of overall examples sampled, and that scales with the amount of labeled data as opposed to the amount of unlabeled data which allows them to consider datasets up to billions of examples.