Active Learning with Partial Feedback

1 INTRODUCTION . Given a large set of unlabeled images , and a budget to collect annotations , how can we learn an accurate image classifier most economically ? Active Learning ( AL ) seeks to increase data efficiency by strategically choosing which examples to annotate . Typically , AL treats the labeling process as atomic : every annotation costs the same and produces a correct label . However , large-scale multi-class annotation is seldom atomic ; we can ’ t simply ask a crowd-worker to select one among 1000 classes if they aren ’ t familiar with our ontology . Instead , annotation pipelines typically solicit feedback through simpler mechanisms such as yes/no questions . For example , to construct the 1000-class ImageNet dataset , researchers first filtered candidates for each class via Google Image Search , then asking crowd-workers questions like “ Is there a Burmese cat in this image ? ” ( Deng et al. , 2009 ) . For tasks where the Google trick won ’ t work , we might exploit class hierarchies to drill down to the exact label . Costs scale with the number of questions asked . Thus , real-world annotation costs can vary per example ( Settles , 2011 ) . We propose Active Learning with Partial Feedback ( ALPF ) , asking , can we cut costs by actively choosing both which examples to annotate , and which questions to ask ? Say that for a new image , our current classifier ∗This work was done while the author was an intern at Amazon AI places 99 % of the predicted probability mass on various dog breeds . Why start at the top of the tree – “ is this an artificial object ? ” – when we can cut costs by jumping straight to dog breeds ( Figure 1 ) ? ALPF proceeds as follows : In addition to the class labels , the learner possesses a pre-defined collection of composite classes , e.g . dog⊃ bulldog , mastiff , .... At each round , the learner selects an ( example , class ) pair . The annotator responds with binary feedback , leaving the learner with a partial label . If only the atomic class label remains , the learner has obtained an exact label . For simplicity , we focus on hierarchically-organized collections—trees with atomic classes as leaves and composite classes as internal nodes . For this to work , we need a hierarchy of concepts familiar to the annotator . Imagine asking an annotator “ is this a foo ? ” where foo represents a category comprised of 500 random ImageNet classes . Determining class membership would be onerous for the same reason that providing an exact label is : It requires the annotator be familiar with an enormous list of seemingly-unrelated options before answering . On the other hand , answering “ is this an animal ? ” is easy despite animal being an extremely coarse-grained category —because most people already know what an animal is . We use active questions in a few ways . To start , in the simplest setup , we can select samples at random but then once each sample is selected , choose questions actively until finding the label : ML : “ Is it a dog ? ” Human : Yes ! ML : “ Is it a poodle ? ” Human : No ! ML : “ Is it a hound ? ” Human : Yes ! ML : “ Is it a Rhodesian ? ” Human : No ! ML : “ Is it a Dachsund ? ” Human : Yes ! In ALPF , we go one step further . Since our goal is to produce accurate classifiers on tight budget , should we necessarily label each example to completion ? After each question , ALPF learners have the option of choosing a different example for the next binary query . Efficient learning under ALPF requires ( i ) good strategies for choosing ( example , class ) pairs , and ( ii ) techniques for learning from the partially-labeled data that results when labeling examples to completion isn ’ t required . We first demonstrate an effective scheme for learning from partial labels . The predictive distribution is parameterized by a softmax over all classes . On a per-example basis , we convert the multiclass problem to a binary classification problem , where the two classes correspond to the subsets of potential and eliminated classes . We determine the total probability assigned to potential classes by summing over their softmax probabilities . For active learning with partial feedback , we introduce several acquisition functions for soliciting partial labels , selecting questions among all ( example , class ) pairs . One natural method , expected information gain ( EIG ) generalizes the classic maximum entropy heuristic to the ALPF setting . Our two other heuristics , EDC and ERC , select based on the number of labels that we expect to see eliminated from and remaining in a given partial label , respectively . We evaluate ALPF learners on CIFAR10 , CIFAR100 , and Tiny ImageNet datasets . In all cases , we use WordNet to impose a hierarchy on our labels . Each of our experiments simulates rounds of active learning , starting with a small amount of i.i.d . data to warmstart the models , and proceeding until all examples are exactly labeled . We compare models by their test-set accuracy after various amounts of annotation . Experiments show that ERC sampling performs best . On TinyImageNet , with a budget of 250k binary questions , ALPF improves in accuracy by 26 % ( relative ) and 8.1 % ( absolute ) over the i.i.d . baseline . Additionally , ERC & EDC fully annotate the dataset with just 491k and 484k examples binary questions , respectively ( vs 827k ) , a 42 % reduction in annotation cost . Surprisingly , we observe that taking disparate annotation costs into account may alter the conventional wisdom that active learners should solicit labels for hard examples . In ALPF , easy examples might yield less information , but are cheaper to annotate . 2 ACTIVE LEARNING WITH PARTIAL FEEDBACK . By x ∈ Rd and y ∈ Y for Y = { { 1 } , ... , { k } } , we denote feature vectors and labels . Here d is the feature dimension and k is the number of atomic classes . By atomic class , we mean that they are indivisible . As in conventional AL , the agent starts off with an unlabeled training set D = { x1 , ... , xn } . Composite classes We also consider a pre-specified collection of composite classes C = { c1 , ... , cm } , where each composite class ci ⊂ { 1 , ... , k } is a subset of labels such that |ci| ≥ 1 . Note that C includes both the atomic and composite classes . In this paper ’ s empirical section , we generate composite classes by imposing an existing lexical hierarchy on the class labels ( Miller , 1995 ) . Partial labels For an example i , we use partial label to describe any element ỹi ⊂ { 1 , ... , k } such that ỹi ⊃ yi . We call ỹi a partial label because it may rule out some classes , but doesn ’ t fully indicate underlying atomic class . For example , dog = { akita , beagle , bulldog , ... } is a valid partial label when the true label is { bulldog } . An ALPF learner eliminates classes , obtaining successively smaller partial labels , until only one ( the exact label ) remains . To simplify notation , in this paper , by an example ’ s partial label , we refer to the smallest partial label available based on the already-eliminated classes . At any step t and for any example i , we use ỹ ( t ) i to denote the current partial label . The initial partial label for every example is ỹ 0 = { 1 , ... , k } An exact label is achieved when the partial label ỹi = yi . Partial Feedback The set of possible questions Q = X × C includes all pairs of examples and composite classes . An ALPF learner interacts with annotators by choosing questions q ∈ Q . Informally , we pick a question q = ( xi , cj ) and ask the annotator , does xi contain a cj ? If the queried example ’ s label belongs to the queried composite class ( yi ⊂ cj ) , the answer is 1 , else 0 . Let αq denote the binary answer to question q ∈ Q . Based on the partial feedback , we can compute the new partial label ỹ ( t+1 ) according to Eq . equation 1 , ỹ ( t+1 ) = { ỹ ( t ) \ c if α = 0 ỹ ( t ) \ c if α = 1 ( 1 ) Note that here ỹ ( t ) and c are sets , α is a bit , c is a set complement , and that ỹ ( t ) \ c and ỹ ( t ) \ c are set subtractions to eliminate classes from the partial label based on the answer . Learning Process The learning process is simple : At each round t , the learner selects a pair ( x , c ) for labeling . Note that a rational agent will never select either ( i ) an example for which the exact label is known , or ( ii ) a pair ( x , c ) for which the answer is already known , e.g. , if c ⊃ ỹ ( t ) or c ∩ ỹ ( t ) = ∅ . After receiving binary feedback , the agent updates the corresponding partial label ỹ ( t ) → ỹ ( t+1 ) , using Equation 1 . The agent then re-estimates its model , using all available non-trivial partial labels and selects another question q . In batch-mode , the ALPF learner re-estimates its model once per T queries which is necessary when training is expensive ( e.g . deep learning ) . We summarize the workflow of a ALPF learner in Algorithm 1 . Objectives We state two goals for ALPF learners . First , we want to learn predictors with low error ( on exactly labeled i.i.d . holdout data ) , given a fixed annotation budget . Second , we want to fully annotate datasets at the lowest cost . In our experiments ( Section 3 ) , a ALPF strategy dominates on both tasks . Algorithm 1 Active Learning with Partial Feedback Input : X← ( x1 , . . . , xN ) , Q← ( q1 , . . . , qM ) , K , T . Input : D ← [ xi ] Ni=1 , C ← [ cj ] Mj=1 , k , T Initialize : ỹ ( 0 ) i ← { 1 , . . . , k } , θ ← θ ( 0 ) , t← 0 repeat Score every ( xi , cj ) with θ repeat Select ( xi∗ , cj∗ ) with the best score Query cj∗ on data xi∗ Receive feedback α Update ỹ ( t+1 ) i∗ according to α t← t+ 1 until ( t mod T = 0 ) or ( ∀i , |ỹ ( t ) i | = 1 ) θ ← argminθ L ( θ ) until ∀i , |ỹ ( t ) i | = 1 or t exhausts budget

Summary: This paper proposes active learning with partial feedback, which means at each step, the learner actively chooses both which example to label and which binary question to ask, then learn the multi-class classifier with these partial labels. Three different sampling strategies are used during active learning. Experimental results demonstrate that the proposed ALPF strategy outperforms existing baselines on the predicting accuracy under a limited budget.