A Boosting Approach to Reinforcement Learning

1 INTRODUCTION . The field of reinforcement learning , formally modelled as learning in Markov decision processes ( MDP ) , models the mechanism of learning from rewards , as opposed to examples . Although the case of tabular MDPs is well understood , the main difficulty in applying RL to practice is the size of the state space . Various techniques have been suggested and applied to cope with very large MDPs . The most common of which is function approximation of either the value or the transition function of the underlying MDP , many times using deep neural networks . Training deep neural networks in the supervised learning model is known to be computationally hard . Therefore reinforcement learning with neural function approximation is also computationally hard in general , and for this reason lacks provable guarantees . This challenge of finding efficient and provable algorithms for MDPs with large state space is the focus of our study . Previous approaches can be categorized in terms of the structural assumptions made on the MDP to circumvent the computational hardness . Some studies focus on structured dynamics , whereas others on structured value function or policy classes w.r.t . to the dynamics . In this paper we study another methodology to derive provable algorithms for reinforcement learning : ensemble methods for aggregating weak or approximate algorithms into substantially more accurate solutions . Our method can be thought of as extending the methodology of boosting from supervised learning ( Schapire & Freund , 2012 ) to reinforcement learning . Interestingly , however , our resulting aggregation of weak learners is not linear . In order to circumvent the computational hardness of solving general MDPs with function approximation , we assumes access to a weak learner : an efficient sample-based procedure that is capable of generating an approximate solution to any linear optimization objective over the space of policies . We describe an algorithm that iteratively calls this procedure on carefully constructed new objectives , and aggregates the solution into a single policy . We prove that after sufficiently many iterations , our resulting policy is provably near-optimal . 1.1 CHALLENGES AND TECHNIQUES . Reinforcement learning is quite different from supervised learning and several difficulties have to be circumvented for boosting to work . Amongst the challenges that the reinforcement learning setting presents , consider the following , ( a ) The value function is not a convex or concave function of the policy . This is true even in the tabular case , and even more so if we use a parameterized policy class . ( b ) The transition matrix is unknown , or prohibitively large to manipulate for large state spaces . This means that even evaluation of a policy can not be exact , and can only be computed approximately . ( c ) It is unrealistic to expect a weak learner that attains near-optimal value for a given linear objective over the policy class . At most one can hope for a multiplicative and/or additive approximation of the overall value . Our approach overcomes these challenges by applied several new as well as recently developed techniques . To overcome the nonconvexity of the value function , we use a novel variant of the Frank-Wolfe optimization algorithm that simultaneously delivers on two guarantees . First , it finds a first order stationary point with near-optimal rate . Secondly , if the objective happens to admit a certain gradient domination property , an important generalization of convexity , it also guarantees near optimal value . The application of the nonconvex Frank-Wolfe method is justified due to previous recent investigation of the policy gradient algorithm ( Agarwal et al. , 2019 ; 2020a ) , which identified conditions under which the value function is gradient dominated . The second information-theoretic challenge of the unknown transition function is overcome by careful algorithmic design : our boosting algorithm requires only samples of the transitions and rewards . These are obtained by rollouts on the MDP . The third challenge is perhaps the most difficult to overcome . Thus far , the use of the Frank-Wolfe method in reinforcement learning did not include a multiplicative approximation , which is critical for our application . Luckily , recent work in the area of online convex optimization ( Hazan & Singh , 2021 ) studies boosting with a multiplicative weak learner . We make critical use of this new technique which includes a non-linear aggregation ( using a 2-layer neural network ) of the weak learners . This aspect is perhaps of general interest to boosting algorithm design , which is mostly based on linear aggregation . 1.2 OUR CONTRIBUTIONS . Our main contribution is a novel efficient boosting algorithm for reinforcement learning . The input to this algorithm is a weak learning method capable of approximately optimizing a linear function over a certain policy class . The output of the algorithm is a policy which does not belong to the original class considered . It is rather a non-linear aggregation of policies from the original class , according to a two-layer neural network . This is a result of the two-tier structure of our algorithm : an outer loop of non-convex Frank-Wolfe method , and an inner loop of online convex optimization boosting . The final policy comes with provable guarantees against the class of all possible policies . Our algorithm and guarantees come in four flavors , depending on the mode of accessing the MDP ( two options ) , and the boosting methodology for the inner online convex optimization problem ( two options ) . It is important to point out that we study the question from an optimization perspective , and hence , assume the availability of an efficient exploration scheme – either via access to a reset distribution that has some overlap with the state distribution of the optimal policy , or constraining the policy class to policies that explore sufficiently . Such considerations also arise when reducing reinforcement learning to a sequence of supervised learning problems , e.g . Conservative Policy Iteration ( Kakade & Langford , 2002 ) assumes the former . One contribution we make here is to quantitatively differentiate between these two modes of exploration in terms of the rates of convergence they enable for the boosting setting . 1.3 RELATED WORK . To cope with prohibitively large MDPs , the method of choice to approximate the policy and transition space are deep neural networks , dubbed “ deep reinforcement learning '' . Deep RL gave rise to beyond human performance in games such as Go , protein folding , as well as near-human level autonomous driving . In terms of provable methods for deep RL , there are two main lines of work . The first is a robust analysis of the policy gradient algorithm ( Agarwal et al. , 2019 ; 2020a ) . Importantly , the gradient domination property of the value function established in this work is needed in order to achieve global convergence guarantees of our boosting method . The other line of work for provable approaches is policy iteration , which uses a restricted policy class , making incremental updates , such as Conservative Policy Iteration ( CPI ) ( Kakade & Langford , 2002 ; Scherrer & Geist , 2014 ) , and Policy Search by Dynamic Programming ( PSDP ) ( Bagnell et al. , 2003 ) . Our boosting approach for provable deep RL builds on the vast literature of boosting for supervised learning ( Schapire & Freund , 2012 ) , and recently online learning ( Leistner et al. , 2009 ; Chen et al. , 2012 ; 2014 ; Beygelzimer et al. , 2015 ; Jung et al. , 2017 ; Jung & Tewari , 2018 ) . One of the crucial techniques important for our application is the extension of boosting to the online convex optimization setting , with bandit information ( Brukhim & Hazan , 2021 ) , and critically with a multiplicative weak learner ( Hazan & Singh , 2021 ) . This latter technique implies a non-linear aggregation of the weak learners . Non-linear boosting was only recently investigated in the context of classification ( Alon et al. , 2020 ) , where it was shown to potentially enable significantly more efficient boosting . Perhaps the closest work to ours is boosting in the context of control of dynamical systems ( Agarwal et al. , 2020b ) . However , this work critically requires knowledge of the underlying dynamics ( transitions ) , which we do not , and can not cope with a multiplicative approximate weak learner . The Frank-Wolfe algorithm is extensively used in machine learning , see e.g . ( Jaggi , 2013 ) , references therein , and recent progress in stochastic Frank-Wolfe methods ( Hassani et al. , 2017 ; Mokhtari et al. , 2018 ; Chen et al. , 2018 ; Xie et al. , 2019 ) . Recent literature has applied a variant of this algorithm to reinforcement learning in the context of state space exploration ( Hazan et al. , 2019 ) . 2 PRELIMINARIES . Optimization . We say that a differentiable function f : K 7→ R over some domain K is L-smooth with respect to some norm ‖ · ‖∗ if for every x , y ∈ K we have ∣∣f ( y ) − f ( x ) −∇f ( x ) > ( y − x ) ∣∣ ≤ L 2 ‖x− y‖2∗ . For constrained optimization ( such as over ∆A ) , the projection Γ : R|A| → ∆A of a point x to onto a domain ∆A is Γ [ x ] = arg min y∈∆A ‖x− y‖ . An important generalization of convex function we use henceforth is that of gradient domination , Definition 1 ( Gradient Domination ) . A function f : K → R is said to be ( κ , τ , K1 , K2 ) -locally gradient dominated ( around K1 by K2 ) if for all x ∈ K1 , it holds that max y∈K f ( y ) − f ( x ) ≤ κ× max y∈K2 { ∇f ( x ) > ( y − x ) } + τ. Markov decision process . An infinite-horizon discounted Markov Decision Process ( MDP ) M = ( S , A , P , r , γ , d0 ) is specified by : a state space S , an action space A , a transition model P where P ( s′|s , a ) denotes the probability of immediately transitioning to state s′ upon taking action a at state s , a reward function r : S ×A→ [ 0 , 1 ] where r ( s , a ) is the immediate reward associated with taking action a at state s , a discount factor γ ∈ [ 0 , 1 ) ; a starting state distribution d0 over S. For any infinite-length state-action sequence ( hereafter , called a trajectory ) , we assign the following value V ( τ = ( s0 , a0 , s1 , a1 , . . . ) ) = ∞∑ t=0 γtr ( st , at ) . The agent interacts with the MDP through the choice of stochastic policy π : S → ∆A it executes , where ∆A denotes the probability simplex over A . The execution of such a policy induces a distribution over trajectories τ = ( s0 , a0 , . . . ) as P ( τ |π ) = d0 ( s0 ) ∞∏ t=0 ( P ( st+1|st , at ) π ( at|st ) ) . ( 1 ) Using this description we can associate a state V π ( s ) and state-action Qπ ( s , a ) value function with any policy π . For an arbitrary distrbution d over S , define : Qπ ( s ) = E [ ∞∑ t=0 γtr ( st , at ) ∣∣∣ π , s0 = s , a0 = a ] , V π ( s ) = Ea∼π ( ·|s ) [ Qπ ( s , a ) |π , s ] , V πd = Es0∼d [ V π ( s ) |π ] . Here the expectation is with respect to the randomness of the trajectory induced by π in M . When convenient , we shall use V π to denote V πd0 , and V ∗ to denote maxπ V π . Similarly , to any policy π , one may ascribe a ( discounted ) state-visitation distribution dπ = dπd0 . dπd ( s ) = ( 1− γ ) ∞∑ t=0 γt ∑ τ : st=s P ( τ |π , s0 ∼ d ) Modes of Accessing the MDP . We henceforth consider two modes of accessing the MDP , that are standard in the reinforcement learning literature , and provide different results for each . The first natural access model is called the episodic rollout setting . This mode of interaction allows us to execute a policy , stop and restart at any point , and do this multiple times . Another interaction model we consider is called rollout with ν-restarts . This is similar to the episodic setting , but here the agent may draw from the MDP a trajectory seeded with an initial state distribution ν 6= d0 . This interaction model was considered in prior work on policy optimization Kakade & Langford ( 2002 ) ; Agarwal et al . ( 2019 ) . The motivation for this model is two-fold : first , ν can be used to incorporate priors ( or domain knowledge ) about the state coverage of the optimal policy ; second , ν provides a mechanism to incorporate exploration into policy optimization procedures .

Summary: This paper proposes a new approach for solving RL problems with sample complexity independent of the number of states. Rather than imposing structural assumptions, the authors consider access to weak learners and propose a way to combine these weak learners effectively to generate a near optimal policy. The sample complexity result is competitive and does not depend on the number of states, under the assumption of access to weak learners.