Discovering General-Purpose Active Learning Strategies

1 INTRODUCTION . Modern supervised machine learning ( ML ) methods require large annotated datasets for training purposes and the cost of producing them can easily become prohibitive . Active learning ( AL ) mitigates the problem by selecting intelligently and adaptively a subset of the data to be annotated . To do so , AL typically relies on informativeness measures that identify unlabelled datapoints whose labels are most likely to help to improve the performance of the trained model . As a result , good performance is achieved using far fewer annotations than by randomly labelling data . Most AL selection strategies are hand-designed either on the basis of researcher ’ s expertise and intuition or by approximating theoretical criteria ( Settles , 2012 ) . They are often tailored for specific applications and empirical studies show that there is no single strategy that consistently outperforms others in all datasets ( Baram et al. , 2004 ; Ebert et al. , 2012 ) . Furthermore , they only represent a small subset of all possible strategies . To overcome these limitations , it has recently been proposed to design the strategies themselves in a data-driven fashion by learning them from prior experience with AL ( Konyushkova et al. , 2017 ; Bachman et al. , 2017 ) . This meta approach makes it possible to go beyond human intuition and potentially to discover completely new strategies by accounting for the state of the trained ML model when selecting the data to annotate . However , many of these methods are still limited to either learning from closely related domains ( Bachman et al. , 2017 ; Fang et al. , 2017 ; Liu et al. , 2018 ) , or using a greedy selection that may be suboptimal ( Konyushkova et al. , 2017 ; Liu et al. , 2018 ) , or relying on properties of specific classifiers ( Konyushkova et al. , 2017 ; Bachman et al. , 2017 ; Contardo et al. , 2017 ; Ravi & Larochelle , 2018 ) . In short , even though data-driven AL methods have flourished recently , there is still no general-purpose non-myopic methods that depend neither on the kind data nor on the specific ML model used in training . In this paper , we introduce such a generic data-driven AL method that is applicable to heterogeneous datasets and to most ML models because it does not require hand-crafting model- or dataset-specific features . More specifically , we reformulate AL as a Markov Decision Process ( MDP ) and use reinforcement learning ( RL ) to find AL strategy as an optimal MDP policy . To achieve the desired generality , we incorporate two important contributions into our approach . First , we take the AL objective to be minimizing the number of annotations required to achieve a given prediction quality , which is a departure from standard AL approaches that maximize the performance given an annotation budget . In this way , we optimise what the practitioners truly want , that is , the annotation cost , independently of the specific ML model and performance measure being used . To this end , we design the reward function of MDP to reflect our AL objective . Second , we propose a procedure that can lean an AL strategy from data coming from multiple unrelated domains for which annotations are already available . The strategy then applies to domains for which this is not the case . To this end , we defined generic MDP state and action representations that can be computed for arbitrary datasets and without regard to the specific ML model . In our experiments we demonstrate the effectiveness of our approach for the purpose of binary classification by applying the learned strategies to previously unseen datasets from different domains . We show that they enable us to reach pre-defined quality thresholds with fewer annotations than several baselines , including recent meta-AL algorithms ( Hsu & Lin , 2015 ; Konyushkova et al. , 2017 ) . We also analyse the properties of our strategies to understand their behaviour and how it differs from those of more traditional ones . 2 RELATED WORK . Manually-designed AL methods differ in their underlying assumptions , computational costs , theoretical guarantees , and generalization behaviours . However , they all rely on a human designer having decided how the data points should be selected . Representative approaches to doing this are uncertainty sampling ( Lewis & Gale , 1994 ) , which works remarkably well in many cases ( Luo et al. , 2013 ; Sun et al. , 2015 ) , query-by-committee , which does not require probability estimates ( GiladBachrach et al. , 2005 ; Beluch et al. , 2018 ) , and expected model change ( Freytag et al. , 2014 ; Käding et al. , 2015 ) . However , the performance of any one of these strategies on a never seen before dataset is unpredictable ( Baram et al. , 2004 ; Ebert et al. , 2012 ) , which makes it difficult to choose one over the other . In this section , we review recent methods to addressing this difficulty . Combining AL strategies If a single manually designed method does not consistently outperform all others , it makes sense to adaptively select the best strategy or to combine them . The algorithms that do it can rely on heuristics ( Osugi et al. , 2005 ) , on bandit algorithms ( Baram et al. , 2004 ; Hsu & Lin , 2015 ; Chu & Lin , 2016 ) , or on RL to find an MDP policy ( Ebert et al. , 2012 ; Long & Hua , 2015 ) . Still , this approach remains limited to combining existing strategies instead of learning new ones . Furthermore , strategy learning happens during AL and its success depends critically on the ability to estimate the classification performance from scarce annotated data . Data-driven AL Recently , the researchers have therefore turned to so-called data-driven AL approaches that learn AL strategies from annotated data ( Konyushkova et al. , 2017 ; Bachman et al. , 2017 ; Contardo et al. , 2017 ; Ravi & Larochelle , 2018 ; Liu et al. , 2018 ; Fang et al. , 2017 ; Pang et al. , 2018 ) . They learn what kind of datapoints are the most beneficial for training the model given the current state of trained ML model . Then , past experience helps to eventually derive a more effective selection strategy . This has been demonstrated to be effective , but it suffers from a number of limitations . First , this approach is often tailored for learning only from related datasets and domains suitable for transfer or one-shot learning ( Liu et al. , 2018 ; Bachman et al. , 2017 ; Fang et al. , 2017 ; Contardo et al. , 2017 ; Ravi & Larochelle , 2018 ) . Second , many of them rely on specific properties of the ML models , be they standard classifiers ( Konyushkova et al. , 2017 ) or few-shot learning models ( Bachman et al. , 2017 ; Contardo et al. , 2017 ; Ravi & Larochelle , 2018 ) , which restricts their generality . Finally , in some approached the resulting strategy is greedy—for example when supervised ( Konyushkova et al. , 2017 ) or imitation learning ( Liu et al. , 2018 ) is used—that might lead to suboptimal data selection . MDP formulation in data-driven AL is used both for pool-based AL , where datapoints are selected from a large pool of unlabelled data , and for stream-based AL , where datapoints come from a stream and AL decides to annotate a datapoint or not as it appears . In stream-based AL , actions—to annotate or not to—are discrete and Q-learning ( Watkins & Dayan , 1992 ) is the RL method of choice to look for an op ( Woodward & Finn , 2016 ; Fang et al. , 2017 ) . By contrast , in pool-based AL , the action selection concerns all potential datapoints that can be annotated and it is natural to characterise them by continuous vectors that makes it not suitable for Q-learning . So , policy gradient ( Williams , 1992 ; Sutton et al. , 2000 ) methods are usually used ( Bachman et al. , 2017 ; Contardo et al. , 2017 ; Ravi & Larochelle , 2018 ; Pang et al. , 2018 ) . In this paper we focus on pool-based AL but we would like to reap the benefits of Q-learning that is , lower variance and better data-complexity thanks to bootstrapping . To this end , we take advantage of the fact that although actions in pool-based AL are continuous , their number is finite . Thus , we can adapt Q-learning for our purposes . Most data-driven AL methods stipulate a specific objective function that is being maximised . However , the methods are not always evaluated in a way that is consistent with the objective that is optimized . Sometimes , the metric used for evaluation differs from the objective ( Bachman et al. , 2017 ; Konyushkova et al. , 2017 ; Pang et al. , 2018 ) . Sometimes , the learning objective may include additional factors like discounting ( Woodward & Finn , 2016 ; Fang et al. , 2017 ; Pang et al. , 2018 ) or may combine several objectives ( Woodward & Finn , 2016 ; Contardo et al. , 2017 ) . By contrast , our approach uses our evaluation criterion—minimization of the time spent annotating for a given performance level—directly in the strategy learning process . Among data-driven AL , the approach of Pang et al . ( 2018 ) achieves generality by using multiple training datasets to learn strategies , as we do . However , this approach is more complex than ours , relies on policy-gradient RL , and uses a standard AL objective . By contrast , our approach does not require a complex state and action embedding , needs fewer RL episodes for training thanks to using Q-learning , and explicitly maximizes what practitioners care about , that is , reduced annotation cost . 3 METHOD . We formulate the AL process as a Markov decision process ( MDP ) and use reinforcement learning ( RL ) to find an optimal strategy . In this section , we first outline our design philosophy . We then formalize AL in MDP terms and finally describe our approach to finding an optimal MDP policy . For simplicity , we present our approach in the context of binary classification . However , an almost identical AL problem formulation can be used for other ML tasks and a separate selection policy can be trained for each one . 3.1 APPROACH . Our goal is to advance data-driven AL towards general-purpose strategy learning . Desirable strategies should have two key properties . They should be transferable across unrelated datasets and have sufficient flexibility to be applied in conjunction with different ML models . Our design decisions are therefore geared towards learning such strategies . The iterative structure of AL is naturally suited for an MDP formulation : For every state of an AL problem , an agent takes an action that defines the datapoint to annotate and it receives a reward that depends on the quality of the model that is re-trained using the new label . An AL strategy then becomes an MDP policy that maps a state into an action . To achieve seamless transferability and flexibility , our task is therefore to design the states , actions , and rewards to be generic . To this end , we represent states and actions as vectors that are independent from specific dataset feature representations and can be computed for a wide variety of ML models . For example , the probability that the classifier assigns to a datapoint suits this purpose because most classifiers estimate this value . By contrast , the number of support vectors in a support vector machine ( SVM ) or the number of layers of a neural network ( NN ) are not suitable because they are model-specific . Raw feature representations of data are similarly inappropriate because they are domain specific . A classical AL objective is to maximize the prediction quality—often expressed in terms of accuracy , AUC , F-score , or negative squared error—for a given annotation budget . For flexibility ’ s sake , we prefer an objective that is not directly linked to a specific performance measure . We therefore consider the dual objective of minimizing the number of annotations required for a given target quality value . When learning a strategy by optimizing this objective , the AL agent only needs to know if the performance is above or below this target quality , as opposed to its exact value . Therefore , the procedure is less tied to a specific performance measure or setting . Our MDP reward function expresses this objective by penalizing the agent until the target quality is achieved . This motivates the agent to minimize its “ suffering ” by driving the amount of requested annotations down . Having formulated the AL problem as a MDP , we can learn a strategy using RL . We simulate the annotation process on data from a collection of unrelated labelled datasets , that ensures the transferability to new unlabelled datasets . Our approach to finding the optimal policy is based on the deep Q-network ( DQN ) method of Mnih et al . ( 2013 ) . To apply DQN with pool-based AL , we modify it in two ways . First , we make it work with MDP where actions are represented by vectors corresponding to individual datapoints instead of being discrete . Second , we deal with the set of actions At that change between iterations t as it makes sense to annotate a datapoint only once .

Summary: This paper studies the recently problem of learning active learning (LAL). It sets up a MDP where the the state is determined by the labeled, unlabelled datasets and classifier, the acton is to query a point, the reward is linked to classifier test set performance improvement and the transition is to update the base classifier. Recent Q-learning algorithms are used to perform the optimisation. The results show that it outperforms some classic handcrafted AL algorithms and some prior LAL algorithms. A feature of this paper is that the method is relatively simple compared to some prior LAL methods, and also that it learns policies that can transfer successfully across diverse heterogenous datasets.