Convergent Reinforcement Learning with Function Approximation: A Bilevel Optimization Perspective

1 INTRODUCTION . In reinforcement learning ( Sutton & Barto , 1998 ) , the agent aims to make optimal decisions by interacting with the environment and learning from the experiences , where the environment is modeled as a Markov Decision Process ( MDP ) . With the recent advancement of deep learning , reinforcement learning has achieved extraordinary empirical success in solving complicated decision making problems , such as the game of Go ( Silver et al. , 2016 ; 2017 ) , navigation ( Banino et al. , 2018 ) , and dialogue systems ( Li et al. , 2016 ) . Despite its great empirical success , there exists a gap between the theory and practice of reinforcement learning . Specifically , in terms of theory , most existing works focus on either the tabular case or the case with linear function approximation . In the former case , both the state and action spaces are finite , while in the latter the value function is assumed to be linear in a given feature mapping . Using tools for convex optimization and linear regression , the statistical and computational properties of the reinforcement learning algorithms are well-understood under these restrictive settings . Moreover , when it comes to nonlinear function approximation , theoretical analysis becomes intractable as it involves solving a highly nonconvex statistical optimization problems . Moreover , it is shown in Tsitsiklis & Van Roy ( 1997 ) that simple method such as TD ( 0 ) might fail to converge when nonlinear value functions are applied . However , using deep learning techniques , methods such as deep Q-network ( DQN ) ( Mnih et al. , 2015 ) and asynchronous advantage actor-critic ( A3C ) ( Mnih et al. , 2016 ) become baseline algorithms for artificial intelligence for practical applications . To bridge such a gap in DRL , we make the first attempt to study the convergence of online reinforcement learning algorithms with nonlinear function approximation in general . In particular , we propose online Q-learning ( Watkins & Dayan , 1992 ) and actor-critic algorithms with two-timescale updates Konda & Tsitsiklis ( 2000 ) , i.e. , the iterative algorithm consists of two parameters that updates with different learning rates . Although the statistical and computational properties of these methods with linear function approximation are well-studied , convergence of reinforcement learning methods with nonlinear function approximation is fundamentally more challenging due to the lack of convexity . In this case , we can only expect convergence to local minima or stationary points . Furthermore , we bring both the problems of value function and policy learning under the unified framework of bilevel optimization , which motivates two-timescale updating rules in our algorithms . Specifically , bilevel optimization problems consist of two subproblems that are intertwined with each other . Such phenomenon appears in both Q-learning and actor-critic algorithms . For Q-learning with function approximation , the updates of the target function ( also known as target network for DQN ) and the value function are coupled together ; the same is the updates of the policy and value function in the actor-critic algorithm . Since the two variables in bilevel optimization have different roles , one servers as the constraint of the other , two-timescale updates appears naturally for these problems . In particular , the parameter of the lower optimization problem needs to be updated using a larger stepsize . In addition , as a byproduct , from the view of bilevel optimization , the target network in DQN appears to be the parameter of the upper level optimization , which justifies the trick of using a target network in DQN . The analysis of our algorithms utilizes stochastic approximation ( Borkar , 2008 ; Kushner & Yin , 2003 ) , which approximates the asymptotic behavior of the iterates using ordinary differential equations . Specifically , under certain assumptions , we show that the online Q-learning algorithm converges almost surely to a local minimizer of the mean squared Bellman error . In addition , for the actor-critic algorithm , we show that the limit points of the actor and critic updates can be characterized by a set of equations . Our contributions are three-fold . First , we unify the problems of value function estimation and policy learning using the framework of bilevel optimization , which leads to the online Q-learning and actor-critic algorithms using two-timescale updates . Second , using stochastic approximation , the convergence of the proposed algorithms are established with nonlinear function approximation . Third , as a byproduct , the formulation of Q-learning via bilevel optimization justifies the techniques of target network used in DQN , which can be viewed as the parameter of the upper level optimization subproblem . Related Work . Our work is closely related to the literature on the convergence of online reinforcement learning algorithms with function approximation . For policy evaluation , most existing works focus on algorithms with linear function approximation . Tsitsiklis & Van Roy ( 1997 ) study the convergence of the on-policy TD ( λ ) algorithm based on temporal-difference ( TD ) error . To handle off-policy sampling , Maei et al . ( 2010 ) ; Sutton et al . ( 2009 ; 2016 ) ; Yu ( 2015 ) ; Hallak & Mannor ( 2017 ) propose various TD-learning methods with convergence guarantees . Utilizing two-timescale stochastic approximation in Borkar ( 2008 ) and strong convexity , they establish global convergence results for the proposed methods . The finite-sample analysis of these methods are recently established in Dalal et al . ( 2017b ; a ) . Moreover , using a duality formulation , Liu et al . ( 2015 ) ; Du et al . ( 2017 ) study the finite-sample performance of TD-learning algorithms with primal-dual updates . As for nonlinear function approximation , to the best of our knowledge , the only convergent algorithm is the nonlinear-GTD algorithm proposed in Bhatnagar et al . ( 2009a ) , which is more pertinent to our results . Their analysis also depends on two-timescale stochastic approximation , where the variable in the faster timescale essentially solves a linear equation , thus converges to the global optimum . However , in our problem , both the two variables solves nonconvex problems , which adds more challenge in the analysis . In addition , we mainly consider the problem of Q-learning and actor-critic , whereas their results only focus on policy evaluation . Moreover , their algorithm involves the Hessian of the value function and their theory requires the value function to be three times differentiable , whereas our algorithm only requires the gradient of the value function with less stringent assumptions . Thus , our results are not directly comparable with this work . Furthermore , for Q-learning and the actor-critic algorithm , existing results all consider linear function approximation . For Q-learning , Melo et al . ( 2008 ) ; Prashanth et al . ( 2014 ) study the convergence using stochastic approximation . In addition , the actor-critic algorithm is introduced in Sutton et al . ( 2000 ) ; Konda & Tsitsiklis ( 2000 ) , which also study its convergence with linear function approximation . Later , Kakade ( 2002 ) ; Peters & Schaal ( 2008 ) propose the natural actor-critic algorithm which updates the policy function using natural gradient descent ( Amari , 1998 ) . Convergence of actor-critic and natural actor-critic algorithms with linear function approximation are studied in Bhatnagar et al . ( 2009b ; 2008 ) ; Castro & Meir ( 2010 ) ; Bhatnagar ( 2010 ) ; Maei ( 2018 ) . 2 NOTATION AND BACKGROUND . In this section , we first lay out the notation that will be followed throughout this paper and then introduce some background knowledge on reinforcement learning . We stick to the following notation . For a measurable space with domain S , we denote by B ( S ) the set of bounded measurable functions on S. Let Lip ( S , L ) be the set of Lipschitz continuous functions on S with Lipschitz constant L. Let P ( S ) be the set of all probability measures over S. For any ν ∈M ( S ) and any measurable function f : S → R , we denote by ‖f‖ν , p the ` p-norm of f with respect to measure ν . For an integer k , we use ∆k to denote the probability simplex in Rk , and let [ k ] = { 1 , 2 , ... , k } . In reinforcement learning , the problem is modeled by a Markov decision process . A discounted MDP is defined by a tuple ( S , A , P , R , p0 , γ ) , where S is the set of states , A is the set of all possible actions , P : S ×A → P ( S ) is the Markov transition kernel , R : S ×A → R is the reward function , p0 ∈ P ( S ) is the distribution of the initial state s0 , and γ ∈ ( 0 , 1 ) is the discount factor . At any state s ∈ S , suppose action a ∈ A is executed , we use P ( · |s , a ) ∈ P ( S ) to denote the probability distribution of the next state , and let R ( s , a ) be the the immediate reward . For regularity , we assume that the rewards are uniformly bounded by Rmax . By taking action at at state st for each time step t ≥ 0 , the agent receives the discounted cumulative reward R = ∑ t≥0 γ t ·R ( st , at ) . Moreover , a policy π : S → P ( A ) is a action selection rule . Specifically , π ( a | s ) is the probability of selecting action a at state s. We define the state value function and the action value function of policy π as V π ( s ) = Eπ ( R | s0 = s , π ) and Qπ ( s , a ) = Eπ ( R | s0 = s , a0 = a ) , respectively . Here we use Eπ to indicate that the actions { at } t≥1 follow policy π . We define Bellman operator of T π : B ( S ) → B ( S ) by T πV ( s ) = E [ r ( st , at ) + γ · V ( st+1 ) | st = s ] , where at ∼ π ( · | st ) . By definition , we have V π = T πV π for any policy π . Since T ∗ is γ-contractive , V π is the unique fixed point of T π . The problem of estimating V π for a fixed policy π is called policy evaluation . Furthermore , in reinforcement learning , the goal is to obtain the policy that yields the maximal cumulative reward in expectation . To this end , we define the optimal value functionQ∗ : S×A → R by Q∗ ( s , a ) = supπ Q π ( s , a ) , ∀ ( s , a ) ∈ S × A , where the supremum is taken over all possible policies . Then it is known that the greedy policy with respect to Q∗ is the optimal policy . To estimate Q∗ with accuracy , most existing methods utilize the fact that Q∗ is the unique fixed point of the Bellman optimality operator T ∗ , which is defined by ( T ∗Q ) ( s , a ) =r ( s , a ) + γ · E [ max a∈A Q ( st+1 , a ) ∣∣ st = s , at = a ] . ( 2.1 ) The most well-known method for estimating Q∗ is the Q-learning algorithm ( Watkins & Dayan , 1992 ) , based upon a large family of algorithms are derived . In addition to methods that focus on estimating the value functions , another class of algorithms in reinforcement learning directly search for a good policy over a parametrized policy class { πω : ω ∈ Rp } directly , where ω is the parameter . Specifically , the goal is to maximize J ( ω ) = Eπω ( R ) = Es0∼p0 [ V πω ( s0 ) ] . The basis of these methods are provided by the policy gradient theorem ( Baxter & Bartlett , 2000 ) , which states that ( 1− γ ) · ∇ωJ ( ω ) = Es∼ρω , a∼πω ( · | s ) [ ∇ω log πω ( a | s ) ·Q πω ( s , a ) ] = Es∼ρθ , a∼πω ( · | s ) { ∇ω log πω ( a | s ) · [ R ( s , a ) + γ · V πω ( s′ ) − V πω ( s ) ] } , ( 2.2 ) where s′ ∼ P ( · | s , a ) is the next state of the MDP given ( s , a ) ∈ S ×A , and ρω ∈ P ( S ) is the state occupancy measure of πω , i.e. , ρω ( s ) = ( 1− γ ) · ∑ t≥0 γ t · Pπω ( st = s ) . We note that ( 2.2 ) brings the value function and the policy together . By updating θ using ( 2.2 ) with the value function Qπω or V πω estimated using temporal difference learning methods , we obtain the actor-critic algorithms ( Konda & Tsitsiklis , 2000 ) .

Summary: The paper casts the problems of value learning and policy optimization, which can be problematic the non-linear setting, into the bilevel optimization framework. It proposes two novel algorithms with convergence guarantees. Although other works with similar guarantees exist, these algorithms are very appealing for their simplicity. A limited empirical evaluation is provided for the value-based method in Acrobot and Mountain Car and in the Atari games Pong and Breakout for the proposed bilevel Actor Critic.