Learning to Actively Learn: A Robust Approach

This work proposes a procedure for designing algorithms for specific adaptive data collection tasks like active learning and pure-exploration multi-armed bandits . Unlike the design of traditional adaptive algorithms that rely on concentration of measure and careful analysis to justify the correctness and sample complexity of the procedure , our adaptive algorithm is learned via adversarial training over equivalence classes of problems derived from information theoretic lower bounds . In particular , a single adaptive learning algorithm is learned that competes with the best adaptive algorithm learned for each equivalence class . Our procedure takes as input just the available queries , set of hypotheses , loss function , and total query budget . This is in contrast to existing meta-learning work that learns an adaptive algorithm relative to an explicit , user-defined subset or prior distribution over problems which can be challenging to define and be mismatched to the instance encountered at test time . This work is particularly focused on the regime when the total query budget is very small , such as a few dozen , which is much smaller than those budgets typically considered by theoretically derived algorithms . We perform synthetic experiments to justify the stability and effectiveness of the training procedure , and then evaluate the method on tasks derived from real data including a noisy 20 Questions game and a joke recommendation task . 1 INTRODUCTION . Closed-loop learning algorithms use previous observations to inform what measurements to take next in a closed-loop in order to accomplish inference tasks far faster than any fixed measurement plan set in advance . For example , active learning algorithms for binary classification have been proposed that under favorable conditions require exponentially fewer labels than passive , random sampling to identify the optimal classifier ( Hanneke et al. , 2014 ) . And in the multi-armed bandits literature , adaptive sampling techniques have demonstrated the ability to identify the “ best arm ” that optimizes some metric with far fewer experiments than a fixed design ( Garivier & Kaufmann , 2016 ; Fiez et al. , 2019 ) . Unfortunately , such guarantees often either require simplifying assumptions that limit robustness and applicability , or appeal to concentration inequalities that are very loose unless the number of samples is very large ( e.g. , web-scale ) . The aim of this work is a framework that achieves the best of both worlds : algorithms that learn through simulated experience to be as effective as possible with a tiny measurement budget ( e.g. , 20 queries ) , while remaining robust due to adversarial training . Our work fits into a recent trend sometimes referred to as learning to actively learn ( Konyushkova et al. , 2017 ; Bachman et al. , 2017 ; Fang et al. , 2017 ; Boutilier et al. , 2020 ; Kveton et al. , 2020 ) which tunes existing algorithms or learns entirely new active learning algorithms by policy optimization . Previous works in this area learn a policy by optimizing with respect to data observed through prior experience ( e.g. , metalearning or transfer learning ) or an assumed explicit prior distribution of problem parameters ( e.g . the true weight vector for linear regression ) . In contrast , our approach makes no assumptions about what parameters are likely to be encountered at test time , and therefore produces algorithms that do not suffer from a potential mismatch of priors . Instead , our method learns a policy that attempts to mirror the guarantees of frequentist algorithms with instance dependent sample complexities : if the problem is hard you will suffer a large loss , if it is easy you will suffer little . The learning framework is general enough to be applied to many active learning settings of interest and is intended to be used to produce novel and robust high performing algorithms . The difference is that instead of hand-crafting hard instances that witness the difficulty of the problem , we use adversarial training inspired by the robust reinforcement learning literature to automatically train minimax policies . Embracing the use of a simulator allows our learned policies to be very aggressive while maintaining robustness . Indeed , this work is particularly useful in the setting where relatively few rounds of querying can be made , where concentration inequalities of existing algorithms are vacuous . To demonstrate the efficacy of our approach we implement the framework for the ( transductive ) linear bandit problem . This paradigm includes pure-exploration combinatorial bandits ( e.g. , shortest path , matchings ) as a special case which itself reduces to active binary classification . We empirically validate our framework on a simple synthetic experiment before turning our attention to datasets derived from real data including a noisy 20 questions game and a joke recommendation task . 2 PROPOSED FRAMEWORK FOR ROBUST LEARNING TO ACTIVELY LEARN . Whether learned or defined by an expert , any algorithm for active learning can be thought of as a policy from the perspective of reinforcement learning . At time t , based on an internal state st , the policy takes action xt and receives observation yt , which then updates the state and the process repeats . In our work , at time t the state st ∈ S is a function of the history { ( xi , yi ) } t−1i=1 such as its sufficient statistics . Without loss of generality , a policy π takes a state as input and defines a probability distribution over X so that at time t we have xt ∼ π ( st ) . Fix a horizon T . For t = 1 , 2 , . . . , T • state st ∈ S is a function of the history , { ( xi , yi ) } t−1i=1 , • action xt ∈ X is drawn at random from the distribution π ( st ) defined over X , and • next state st+1 ∈ S is constructed by taking action xt in state st and observing yt ∼ f ( ·|θ∗ , st , xt ) until the game terminates at time t = T and the policy receives loss LT . Note that LT is a random variable that depends on the tuple ( π , { ( xi , yi ) } Ti=1 , θ∗ ) . We assume that f is a distribution of known parameteric form to the policy ( e.g. , f ( ·|θ , s , x ) ≡ N ( 〈x , θ〉 , 1 ) ) but the parameter θ is unknown to the policy . Let Pπ , θ , Eπ , θ denote the probability and expectation under the probability law induced by executing policy π in the game with θ∗ = θ to completion . Note that Pπ , θ includes any internal randomness of the policy π and the random observations yt ∼ f ( ·|θ , st , xt ) . Thus , Pπ , θ assigns a probability to any trajectory { ( xi , yi ) } Ti=1 . For a given policy π and θ∗ = θ , the metric of interest we wish to minimize is the expected loss ` ( π , θ ) : = Eπ , θ [ LT ] where LT as defined above is the loss observed at the end of the episode . For a fixed policy π , ` ( π , θ ) defines a loss surface over all possible values of θ . This loss surface captures the fact that some values of θ are just intrinsically harder than others , but also that a policy may be better suited for some values of θ versus others . Example : In active binary classification , T is a label budget , X could be a set of images such that we can query the label of example image xt ∈ X , yt ∈ { −1 , 1 } is the requested binary label , and the loss LT is the classification error of a trained classifier on these collected labels . Finally , θx = p ( y = 1|x ) for all x ∈ X . More examples can be found in Appendix A . 2.1 INSTANCE DEPENDENT PERFORMANCE METRIC . We now define the sense in which we wish to evaluate a particular policy . For any fixed value of θ one could clearly design an algorithm that would maximize performance on θ , but then it might have very poor performance on some other value θ′ 6= θ . Thus , we would ideally like π to perform uniformly well over a set of θ ’ s that are all equivalent in a certain sense . Define a positive function C : Θ → ( 0 , ∞ ) that assigns a score to each θ ∈ Θ that intuitively captures the “ difficulty ” of a particular θ , and can be used as a partial ordering of Θ . Ideally , C ( θ ) is a monotonic transformation of ` ( π̃ , θ ) for some “ best ” policy π̃ that we will define shortly . We give the explicit C ( θ ) for the active binary classification example in Section 3 , further description of C in Section 2.2 , and more examples in Appendix A . For any set of problem instances Θ define ` ( π , Θ ) : = sup θ∈Θ ` ( π , θ ) . And for any r ≥ 0 , define Θ ( r ) = { θ : C ( θ ) ≤ r } . The quantity ` ( π , Θ ( r ) ) − infπ′ ` ( π′ , Θ ( r ) ) is then a function of r that describes the sub-optimality gap of a given policy π relative to an r-dependent baseline policy trained specifically for each r. For a fixed rk > 0 , a policy π that aims to minimize just ` ( π , Θ ( r ) ) might focus just on the hard instances ( i.e. , those with C ( θ ) close to r ) and there may exist a different policy π′ that performs far better than π on easier instances ( i.e. , those with C ( θ ) r ) . To avoid this , assuming supr ` ( π , Θ ( r ) ) − infπ′ ` ( π ′ , Θ ( r ) ) < ∞ , we define π∗ : = arg inf π sup r > 0 ( ` ( π , Θ ( r ) ) − inf π′ ` ( π′ , Θ ( r ) ) ) ( 1 ) as the policy that minimizes the worst case sub-optimality gap over all r > 0 . Figure 1 illustrates these definitions . Instead of computing infπ′ ` ( π′ , Θ ( r ) ) for all r , in practice we define a grid with an increasing sequence { rk } Kk=1 , to find an approximation to π∗ . We are now ready to state the goal of this work : Objective : Given an increasing sequence r1 < · · · < rK that indexes nested sets of problem instances of increasing difficulty , Θ ( r1 ) ⊂ Θ ( r2 ) ⊂ · · · ⊂ Θ ( rK ) , we wish to identify a policy π̂ that minimizes the maximum sub-optimality gap with respect to this sequence . Explicitly , we seek to learn π̂ : = arginf π max k≤K ( ` ( π , Θ ( rk ) ) − inf π′ ` ( π′ , Θ ( rk ) ) ) ( 2 ) where ` ( π , Θ ) : = sup θ∈Θ ` ( π , θ ) and ` ( π , θ ) is the expected loss incurred by policy π on instance θ . Note that as K → ∞ and supk rk+1 rk → 1 , ( 1 ) and ( 2 ) are essentially equivalent under benign smoothness conditions on C ( θ ) , in which case π̂ → π∗ . In practice , we choose a finite K where ΘrK contains all problems that can be solved within the budget T relatively accurately , and a small > 0 , where maxk rk+1 rk = 1 + . Furthermore , the objective in ( 2 ) is equivalent with π̂ = arginf π max k≤K ( ` ( π , Θ ( rk ) ) − ` ( πk , Θ ( rk ) ) ) where πk ∈ arg inf π sup θ : C ( θ ) ≤rk ` ( π , θ ) . We can efficiently solve this objective by first computing πk for all k ∈ [ K ] to obtain ` ( πk , Θ ( rk ) ) as benchmarks , and then use these benchmarks to train π̂ . 2.2 PICKING THE COMPLEXITY FUNCTION C ( θ ) We have defined an optimal policy in terms of a function C ( θ ) that determines a partial ordering over instances θ . This function can come from a heuristic that intuitively captures the difficulty of an instance . Or it can be defined and motivated from information theoretic lower bounds that often describe a general ordering , but are typically very loose relative to empirical performance . For example , consider the standard multi-armed bandit game where an agent has access to K distributions and in each round t ∈ [ T ] she chooses a distribution It ∈ [ K ] and observes a random variable in [ 0 , 1 ] with mean θIt . If her strategy is described by a policy π , once t reaches T she receives loss LT = maxi∈ [ K ] ∑T t=1 θi−θIt with expectation ` ( π , θ ) = E [ LT ] where the expectation is taken with respect to the randomness in the observations , and potentially any randomness of the policy . Under benign conditions , it is known that any policy must suffer ` ( π , θ ) & min { √ KT , ∑ i 6=∗ ( θ∗− θi ) −1 } where θ∗ = maxi∈ [ k ] θi ( Lattimore & Szepesvári , 2018 ) . Such a lower bound is an ideal candidate for C ( θ ) . We define a different C ( θ ) for our particular experiments of interest , and others are described in Appendix A . The bottom line is that any function C ( θ ) works , but if it happens to correspond to an information theoretic lower bound , the resulting policy will match the lower bound if it is achievable .

Summary: The paper "Learning to Actively Learn" proposes a differentiable procedure to design algorithms for adaptive data collection tasks. The framework is based on the idea of making use of a measure of problem convexity (each problem is parametrized by a parameter theta) to solve solving a min-max objective over policies. The rationale behind this objective is that the resulting policy of solving this min-max objective should be robust to the problem complexity. The algorithm then proceeds to sample problem instances and making use of a differentiable objective to find a policy parametrized by a parameter psi, which could be parametrizing a neural network.