Active Learning with Partial Feedback

1 INTRODUCTION . Given a large set of unlabeled images , and a budget to collect annotations , how can we learn an accurate image classifier most economically ? Active Learning ( AL ) seeks to increase data efficiency by strategically choosing which examples to annotate . Typically , AL treats the labeling process as atomic : every annotation costs the same and produces a correct label . However , large-scale multi-class annotation is seldom atomic ; we can ’ t simply ask a crowd-worker to select one among 1000 classes if they aren ’ t familiar with our ontology . Instead , annotation pipelines typically solicit feedback through simpler mechanisms such as yes/no questions . For example , to construct the 1000-class ImageNet dataset , researchers first filtered candidates for each class via Google Image Search , then asking crowd-workers questions like “ Is there a Burmese cat in this image ? ” ( Deng et al. , 2009 ) . For tasks where the Google trick won ’ t work , we might exploit class hierarchies to drill down to the exact label . Costs scale with the number of questions asked . Thus , real-world annotation costs can vary per example ( Settles , 2011 ) . We propose Active Learning with Partial Feedback ( ALPF ) , asking , can we cut costs by actively choosing both which examples to annotate , and which questions to ask ? Say that for a new image , our current classifier ∗This work was done while the author was an intern at Amazon AI places 99 % of the predicted probability mass on various dog breeds . Why start at the top of the tree – “ is this an artificial object ? ” – when we can cut costs by jumping straight to dog breeds ( Figure 1 ) ? ALPF proceeds as follows : In addition to the class labels , the learner possesses a pre-defined collection of composite classes , e.g . dog⊃ bulldog , mastiff , .... At each round , the learner selects an ( example , class ) pair . The annotator responds with binary feedback , leaving the learner with a partial label . If only the atomic class label remains , the learner has obtained an exact label . For simplicity , we focus on hierarchically-organized collections—trees with atomic classes as leaves and composite classes as internal nodes . For this to work , we need a hierarchy of concepts familiar to the annotator . Imagine asking an annotator “ is this a foo ? ” where foo represents a category comprised of 500 random ImageNet classes . Determining class membership would be onerous for the same reason that providing an exact label is : It requires the annotator be familiar with an enormous list of seemingly-unrelated options before answering . On the other hand , answering “ is this an animal ? ” is easy despite animal being an extremely coarse-grained category —because most people already know what an animal is . We use active questions in a few ways . To start , in the simplest setup , we can select samples at random but then once each sample is selected , choose questions actively until finding the label : ML : “ Is it a dog ? ” Human : Yes ! ML : “ Is it a poodle ? ” Human : No ! ML : “ Is it a hound ? ” Human : Yes ! ML : “ Is it a Rhodesian ? ” Human : No ! ML : “ Is it a Dachsund ? ” Human : Yes ! In ALPF , we go one step further . Since our goal is to produce accurate classifiers on tight budget , should we necessarily label each example to completion ? After each question , ALPF learners have the option of choosing a different example for the next binary query . Efficient learning under ALPF requires ( i ) good strategies for choosing ( example , class ) pairs , and ( ii ) techniques for learning from the partially-labeled data that results when labeling examples to completion isn ’ t required . We first demonstrate an effective scheme for learning from partial labels . The predictive distribution is parameterized by a softmax over all classes . On a per-example basis , we convert the multiclass problem to a binary classification problem , where the two classes correspond to the subsets of potential and eliminated classes . We determine the total probability assigned to potential classes by summing over their softmax probabilities . For active learning with partial feedback , we introduce several acquisition functions for soliciting partial labels , selecting questions among all ( example , class ) pairs . One natural method , expected information gain ( EIG ) generalizes the classic maximum entropy heuristic to the ALPF setting . Our two other heuristics , EDC and ERC , select based on the number of labels that we expect to see eliminated from and remaining in a given partial label , respectively . We evaluate ALPF learners on CIFAR10 , CIFAR100 , and Tiny ImageNet datasets . In all cases , we use WordNet to impose a hierarchy on our labels . Each of our experiments simulates rounds of active learning , starting with a small amount of i.i.d . data to warmstart the models , and proceeding until all examples are exactly labeled . We compare models by their test-set accuracy after various amounts of annotation . Experiments show that ERC sampling performs best . On TinyImageNet , with a budget of 250k binary questions , ALPF improves in accuracy by 26 % ( relative ) and 8.1 % ( absolute ) over the i.i.d . baseline . Additionally , ERC & EDC fully annotate the dataset with just 491k and 484k examples binary questions , respectively ( vs 827k ) , a 42 % reduction in annotation cost . Surprisingly , we observe that taking disparate annotation costs into account may alter the conventional wisdom that active learners should solicit labels for hard examples . In ALPF , easy examples might yield less information , but are cheaper to annotate . 2 ACTIVE LEARNING WITH PARTIAL FEEDBACK . By x ∈ Rd and y ∈ Y for Y = { { 1 } , ... , { k } } , we denote feature vectors and labels . Here d is the feature dimension and k is the number of atomic classes . By atomic class , we mean that they are indivisible . As in conventional AL , the agent starts off with an unlabeled training set D = { x1 , ... , xn } . Composite classes We also consider a pre-specified collection of composite classes C = { c1 , ... , cm } , where each composite class ci ⊂ { 1 , ... , k } is a subset of labels such that |ci| ≥ 1 . Note that C includes both the atomic and composite classes . In this paper ’ s empirical section , we generate composite classes by imposing an existing lexical hierarchy on the class labels ( Miller , 1995 ) . Partial labels For an example i , we use partial label to describe any element ỹi ⊂ { 1 , ... , k } such that ỹi ⊃ yi . We call ỹi a partial label because it may rule out some classes , but doesn ’ t fully indicate underlying atomic class . For example , dog = { akita , beagle , bulldog , ... } is a valid partial label when the true label is { bulldog } . An ALPF learner eliminates classes , obtaining successively smaller partial labels , until only one ( the exact label ) remains . To simplify notation , in this paper , by an example ’ s partial label , we refer to the smallest partial label available based on the already-eliminated classes . At any step t and for any example i , we use ỹ ( t ) i to denote the current partial label . The initial partial label for every example is ỹ 0 = { 1 , ... , k } An exact label is achieved when the partial label ỹi = yi . Partial Feedback The set of possible questions Q = X × C includes all pairs of examples and composite classes . An ALPF learner interacts with annotators by choosing questions q ∈ Q . Informally , we pick a question q = ( xi , cj ) and ask the annotator , does xi contain a cj ? If the queried example ’ s label belongs to the queried composite class ( yi ⊂ cj ) , the answer is 1 , else 0 . Let αq denote the binary answer to question q ∈ Q . Based on the partial feedback , we can compute the new partial label ỹ ( t+1 ) according to Eq . equation 1 , ỹ ( t+1 ) = { ỹ ( t ) \ c if α = 0 ỹ ( t ) \ c if α = 1 ( 1 ) Note that here ỹ ( t ) and c are sets , α is a bit , c is a set complement , and that ỹ ( t ) \ c and ỹ ( t ) \ c are set subtractions to eliminate classes from the partial label based on the answer . Learning Process The learning process is simple : At each round t , the learner selects a pair ( x , c ) for labeling . Note that a rational agent will never select either ( i ) an example for which the exact label is known , or ( ii ) a pair ( x , c ) for which the answer is already known , e.g. , if c ⊃ ỹ ( t ) or c ∩ ỹ ( t ) = ∅ . After receiving binary feedback , the agent updates the corresponding partial label ỹ ( t ) → ỹ ( t+1 ) , using Equation 1 . The agent then re-estimates its model , using all available non-trivial partial labels and selects another question q . In batch-mode , the ALPF learner re-estimates its model once per T queries which is necessary when training is expensive ( e.g . deep learning ) . We summarize the workflow of a ALPF learner in Algorithm 1 . Objectives We state two goals for ALPF learners . First , we want to learn predictors with low error ( on exactly labeled i.i.d . holdout data ) , given a fixed annotation budget . Second , we want to fully annotate datasets at the lowest cost . In our experiments ( Section 3 ) , a ALPF strategy dominates on both tasks . Algorithm 1 Active Learning with Partial Feedback Input : X← ( x1 , . . . , xN ) , Q← ( q1 , . . . , qM ) , K , T . Input : D ← [ xi ] Ni=1 , C ← [ cj ] Mj=1 , k , T Initialize : ỹ ( 0 ) i ← { 1 , . . . , k } , θ ← θ ( 0 ) , t← 0 repeat Score every ( xi , cj ) with θ repeat Select ( xi∗ , cj∗ ) with the best score Query cj∗ on data xi∗ Receive feedback α Update ỹ ( t+1 ) i∗ according to α t← t+ 1 until ( t mod T = 0 ) or ( ∀i , |ỹ ( t ) i | = 1 ) θ ← argminθ L ( θ ) until ∀i , |ỹ ( t ) i | = 1 or t exhausts budget

Summary: The paper considers a multiclass classification problem in which labels are grouped in a given number M of subsets c_j, which contain all individual labels as singletons. Training takes place through an active learning setting in which all training examples x_i are initially provided without their ground truth labels y_i. The learner issues queries of the form (x_i,c_j) where c_j is one of the given subsets of labels. The annotator only replies yes/no according to whether the true label y_i of x_i belongs to c_j or not. Hence, for each training example the learner maintains a "version space" containing all labels that are consistent with the answers received so far for that example. The active learning process consists of the following steps: (1) use the current learning model to score queries (x_i,c_j); (2) query the best (x_i,c_j); (3) update the model.