A Boosting Approach to Reinforcement Learning

1 INTRODUCTION . The field of reinforcement learning , formally modelled as learning in Markov decision processes ( MDP ) , models the mechanism of learning from rewards , as opposed to examples . Although the case of tabular MDPs is well understood , the main difficulty in applying RL to practice is the size of the state space . Various techniques have been suggested and applied to cope with very large MDPs . The most common of which is function approximation of either the value or the transition function of the underlying MDP , many times using deep neural networks . Training deep neural networks in the supervised learning model is known to be computationally hard . Therefore reinforcement learning with neural function approximation is also computationally hard in general , and for this reason lacks provable guarantees . This challenge of finding efficient and provable algorithms for MDPs with large state space is the focus of our study . Previous approaches can be categorized in terms of the structural assumptions made on the MDP to circumvent the computational hardness . Some studies focus on structured dynamics , whereas others on structured value function or policy classes w.r.t . to the dynamics . In this paper we study another methodology to derive provable algorithms for reinforcement learning : ensemble methods for aggregating weak or approximate algorithms into substantially more accurate solutions . Our method can be thought of as extending the methodology of boosting from supervised learning ( Schapire & Freund , 2012 ) to reinforcement learning . Interestingly , however , our resulting aggregation of weak learners is not linear . In order to circumvent the computational hardness of solving general MDPs with function approximation , we assumes access to a weak learner : an efficient sample-based procedure that is capable of generating an approximate solution to any linear optimization objective over the space of policies . We describe an algorithm that iteratively calls this procedure on carefully constructed new objectives , and aggregates the solution into a single policy . We prove that after sufficiently many iterations , our resulting policy is provably near-optimal . 1.1 CHALLENGES AND TECHNIQUES . Reinforcement learning is quite different from supervised learning and several difficulties have to be circumvented for boosting to work . Amongst the challenges that the reinforcement learning setting presents , consider the following , ( a ) The value function is not a convex or concave function of the policy . This is true even in the tabular case , and even more so if we use a parameterized policy class . ( b ) The transition matrix is unknown , or prohibitively large to manipulate for large state spaces . This means that even evaluation of a policy can not be exact , and can only be computed approximately . ( c ) It is unrealistic to expect a weak learner that attains near-optimal value for a given linear objective over the policy class . At most one can hope for a multiplicative and/or additive approximation of the overall value . Our approach overcomes these challenges by applied several new as well as recently developed techniques . To overcome the nonconvexity of the value function , we use a novel variant of the Frank-Wolfe optimization algorithm that simultaneously delivers on two guarantees . First , it finds a first order stationary point with near-optimal rate . Secondly , if the objective happens to admit a certain gradient domination property , an important generalization of convexity , it also guarantees near optimal value . The application of the nonconvex Frank-Wolfe method is justified due to previous recent investigation of the policy gradient algorithm ( Agarwal et al. , 2019 ; 2020a ) , which identified conditions under which the value function is gradient dominated . The second information-theoretic challenge of the unknown transition function is overcome by careful algorithmic design : our boosting algorithm requires only samples of the transitions and rewards . These are obtained by rollouts on the MDP . The third challenge is perhaps the most difficult to overcome . Thus far , the use of the Frank-Wolfe method in reinforcement learning did not include a multiplicative approximation , which is critical for our application . Luckily , recent work in the area of online convex optimization ( Hazan & Singh , 2021 ) studies boosting with a multiplicative weak learner . We make critical use of this new technique which includes a non-linear aggregation ( using a 2-layer neural network ) of the weak learners . This aspect is perhaps of general interest to boosting algorithm design , which is mostly based on linear aggregation . 1.2 OUR CONTRIBUTIONS . Our main contribution is a novel efficient boosting algorithm for reinforcement learning . The input to this algorithm is a weak learning method capable of approximately optimizing a linear function over a certain policy class . The output of the algorithm is a policy which does not belong to the original class considered . It is rather a non-linear aggregation of policies from the original class , according to a two-layer neural network . This is a result of the two-tier structure of our algorithm : an outer loop of non-convex Frank-Wolfe method , and an inner loop of online convex optimization boosting . The final policy comes with provable guarantees against the class of all possible policies . Our algorithm and guarantees come in four flavors , depending on the mode of accessing the MDP ( two options ) , and the boosting methodology for the inner online convex optimization problem ( two options ) . It is important to point out that we study the question from an optimization perspective , and hence , assume the availability of an efficient exploration scheme – either via access to a reset distribution that has some overlap with the state distribution of the optimal policy , or constraining the policy class to policies that explore sufficiently . Such considerations also arise when reducing reinforcement learning to a sequence of supervised learning problems , e.g . Conservative Policy Iteration ( Kakade & Langford , 2002 ) assumes the former . One contribution we make here is to quantitatively differentiate between these two modes of exploration in terms of the rates of convergence they enable for the boosting setting . 1.3 RELATED WORK . To cope with prohibitively large MDPs , the method of choice to approximate the policy and transition space are deep neural networks , dubbed “ deep reinforcement learning '' . Deep RL gave rise to beyond human performance in games such as Go , protein folding , as well as near-human level autonomous driving . In terms of provable methods for deep RL , there are two main lines of work . The first is a robust analysis of the policy gradient algorithm ( Agarwal et al. , 2019 ; 2020a ) . Importantly , the gradient domination property of the value function established in this work is needed in order to achieve global convergence guarantees of our boosting method . The other line of work for provable approaches is policy iteration , which uses a restricted policy class , making incremental updates , such as Conservative Policy Iteration ( CPI ) ( Kakade & Langford , 2002 ; Scherrer & Geist , 2014 ) , and Policy Search by Dynamic Programming ( PSDP ) ( Bagnell et al. , 2003 ) . Our boosting approach for provable deep RL builds on the vast literature of boosting for supervised learning ( Schapire & Freund , 2012 ) , and recently online learning ( Leistner et al. , 2009 ; Chen et al. , 2012 ; 2014 ; Beygelzimer et al. , 2015 ; Jung et al. , 2017 ; Jung & Tewari , 2018 ) . One of the crucial techniques important for our application is the extension of boosting to the online convex optimization setting , with bandit information ( Brukhim & Hazan , 2021 ) , and critically with a multiplicative weak learner ( Hazan & Singh , 2021 ) . This latter technique implies a non-linear aggregation of the weak learners . Non-linear boosting was only recently investigated in the context of classification ( Alon et al. , 2020 ) , where it was shown to potentially enable significantly more efficient boosting . Perhaps the closest work to ours is boosting in the context of control of dynamical systems ( Agarwal et al. , 2020b ) . However , this work critically requires knowledge of the underlying dynamics ( transitions ) , which we do not , and can not cope with a multiplicative approximate weak learner . The Frank-Wolfe algorithm is extensively used in machine learning , see e.g . ( Jaggi , 2013 ) , references therein , and recent progress in stochastic Frank-Wolfe methods ( Hassani et al. , 2017 ; Mokhtari et al. , 2018 ; Chen et al. , 2018 ; Xie et al. , 2019 ) . Recent literature has applied a variant of this algorithm to reinforcement learning in the context of state space exploration ( Hazan et al. , 2019 ) . 2 PRELIMINARIES . Optimization . We say that a differentiable function f : K 7→ R over some domain K is L-smooth with respect to some norm ‖ · ‖∗ if for every x , y ∈ K we have ∣∣f ( y ) − f ( x ) −∇f ( x ) > ( y − x ) ∣∣ ≤ L 2 ‖x− y‖2∗ . For constrained optimization ( such as over ∆A ) , the projection Γ : R|A| → ∆A of a point x to onto a domain ∆A is Γ [ x ] = arg min y∈∆A ‖x− y‖ . An important generalization of convex function we use henceforth is that of gradient domination , Definition 1 ( Gradient Domination ) . A function f : K → R is said to be ( κ , τ , K1 , K2 ) -locally gradient dominated ( around K1 by K2 ) if for all x ∈ K1 , it holds that max y∈K f ( y ) − f ( x ) ≤ κ× max y∈K2 { ∇f ( x ) > ( y − x ) } + τ. Markov decision process . An infinite-horizon discounted Markov Decision Process ( MDP ) M = ( S , A , P , r , γ , d0 ) is specified by : a state space S , an action space A , a transition model P where P ( s′|s , a ) denotes the probability of immediately transitioning to state s′ upon taking action a at state s , a reward function r : S ×A→ [ 0 , 1 ] where r ( s , a ) is the immediate reward associated with taking action a at state s , a discount factor γ ∈ [ 0 , 1 ) ; a starting state distribution d0 over S. For any infinite-length state-action sequence ( hereafter , called a trajectory ) , we assign the following value V ( τ = ( s0 , a0 , s1 , a1 , . . . ) ) = ∞∑ t=0 γtr ( st , at ) . The agent interacts with the MDP through the choice of stochastic policy π : S → ∆A it executes , where ∆A denotes the probability simplex over A . The execution of such a policy induces a distribution over trajectories τ = ( s0 , a0 , . . . ) as P ( τ |π ) = d0 ( s0 ) ∞∏ t=0 ( P ( st+1|st , at ) π ( at|st ) ) . ( 1 ) Using this description we can associate a state V π ( s ) and state-action Qπ ( s , a ) value function with any policy π . For an arbitrary distrbution d over S , define : Qπ ( s ) = E [ ∞∑ t=0 γtr ( st , at ) ∣∣∣ π , s0 = s , a0 = a ] , V π ( s ) = Ea∼π ( ·|s ) [ Qπ ( s , a ) |π , s ] , V πd = Es0∼d [ V π ( s ) |π ] . Here the expectation is with respect to the randomness of the trajectory induced by π in M . When convenient , we shall use V π to denote V πd0 , and V ∗ to denote maxπ V π . Similarly , to any policy π , one may ascribe a ( discounted ) state-visitation distribution dπ = dπd0 . dπd ( s ) = ( 1− γ ) ∞∑ t=0 γt ∑ τ : st=s P ( τ |π , s0 ∼ d ) Modes of Accessing the MDP . We henceforth consider two modes of accessing the MDP , that are standard in the reinforcement learning literature , and provide different results for each . The first natural access model is called the episodic rollout setting . This mode of interaction allows us to execute a policy , stop and restart at any point , and do this multiple times . Another interaction model we consider is called rollout with ν-restarts . This is similar to the episodic setting , but here the agent may draw from the MDP a trajectory seeded with an initial state distribution ν 6= d0 . This interaction model was considered in prior work on policy optimization Kakade & Langford ( 2002 ) ; Agarwal et al . ( 2019 ) . The motivation for this model is two-fold : first , ν can be used to incorporate priors ( or domain knowledge ) about the state coverage of the optimal policy ; second , ν provides a mechanism to incorporate exploration into policy optimization procedures .

Summary: In this paper, the authors study boosting in RL, i.e., how to convert weak learners into effective policies. The authors provide an algorithm that improves the accuracy of the weak learners iteratively, and the sample complexity and running time do not explicitly depend on the number of states.   In order to overcome the non-convexity of the value function (with respect to the policy space), the authors use a non-convex variant of the Frank-Wolfe method together with recent advances in gradient boosting.