Convergent Reinforcement Learning with Function Approximation: A Bilevel Optimization Perspective

1 INTRODUCTION . In reinforcement learning ( Sutton & Barto , 1998 ) , the agent aims to make optimal decisions by interacting with the environment and learning from the experiences , where the environment is modeled as a Markov Decision Process ( MDP ) . With the recent advancement of deep learning , reinforcement learning has achieved extraordinary empirical success in solving complicated decision making problems , such as the game of Go ( Silver et al. , 2016 ; 2017 ) , navigation ( Banino et al. , 2018 ) , and dialogue systems ( Li et al. , 2016 ) . Despite its great empirical success , there exists a gap between the theory and practice of reinforcement learning . Specifically , in terms of theory , most existing works focus on either the tabular case or the case with linear function approximation . In the former case , both the state and action spaces are finite , while in the latter the value function is assumed to be linear in a given feature mapping . Using tools for convex optimization and linear regression , the statistical and computational properties of the reinforcement learning algorithms are well-understood under these restrictive settings . Moreover , when it comes to nonlinear function approximation , theoretical analysis becomes intractable as it involves solving a highly nonconvex statistical optimization problems . Moreover , it is shown in Tsitsiklis & Van Roy ( 1997 ) that simple method such as TD ( 0 ) might fail to converge when nonlinear value functions are applied . However , using deep learning techniques , methods such as deep Q-network ( DQN ) ( Mnih et al. , 2015 ) and asynchronous advantage actor-critic ( A3C ) ( Mnih et al. , 2016 ) become baseline algorithms for artificial intelligence for practical applications . To bridge such a gap in DRL , we make the first attempt to study the convergence of online reinforcement learning algorithms with nonlinear function approximation in general . In particular , we propose online Q-learning ( Watkins & Dayan , 1992 ) and actor-critic algorithms with two-timescale updates Konda & Tsitsiklis ( 2000 ) , i.e. , the iterative algorithm consists of two parameters that updates with different learning rates . Although the statistical and computational properties of these methods with linear function approximation are well-studied , convergence of reinforcement learning methods with nonlinear function approximation is fundamentally more challenging due to the lack of convexity . In this case , we can only expect convergence to local minima or stationary points . Furthermore , we bring both the problems of value function and policy learning under the unified framework of bilevel optimization , which motivates two-timescale updating rules in our algorithms . Specifically , bilevel optimization problems consist of two subproblems that are intertwined with each other . Such phenomenon appears in both Q-learning and actor-critic algorithms . For Q-learning with function approximation , the updates of the target function ( also known as target network for DQN ) and the value function are coupled together ; the same is the updates of the policy and value function in the actor-critic algorithm . Since the two variables in bilevel optimization have different roles , one servers as the constraint of the other , two-timescale updates appears naturally for these problems . In particular , the parameter of the lower optimization problem needs to be updated using a larger stepsize . In addition , as a byproduct , from the view of bilevel optimization , the target network in DQN appears to be the parameter of the upper level optimization , which justifies the trick of using a target network in DQN . The analysis of our algorithms utilizes stochastic approximation ( Borkar , 2008 ; Kushner & Yin , 2003 ) , which approximates the asymptotic behavior of the iterates using ordinary differential equations . Specifically , under certain assumptions , we show that the online Q-learning algorithm converges almost surely to a local minimizer of the mean squared Bellman error . In addition , for the actor-critic algorithm , we show that the limit points of the actor and critic updates can be characterized by a set of equations . Our contributions are three-fold . First , we unify the problems of value function estimation and policy learning using the framework of bilevel optimization , which leads to the online Q-learning and actor-critic algorithms using two-timescale updates . Second , using stochastic approximation , the convergence of the proposed algorithms are established with nonlinear function approximation . Third , as a byproduct , the formulation of Q-learning via bilevel optimization justifies the techniques of target network used in DQN , which can be viewed as the parameter of the upper level optimization subproblem . Related Work . Our work is closely related to the literature on the convergence of online reinforcement learning algorithms with function approximation . For policy evaluation , most existing works focus on algorithms with linear function approximation . Tsitsiklis & Van Roy ( 1997 ) study the convergence of the on-policy TD ( λ ) algorithm based on temporal-difference ( TD ) error . To handle off-policy sampling , Maei et al . ( 2010 ) ; Sutton et al . ( 2009 ; 2016 ) ; Yu ( 2015 ) ; Hallak & Mannor ( 2017 ) propose various TD-learning methods with convergence guarantees . Utilizing two-timescale stochastic approximation in Borkar ( 2008 ) and strong convexity , they establish global convergence results for the proposed methods . The finite-sample analysis of these methods are recently established in Dalal et al . ( 2017b ; a ) . Moreover , using a duality formulation , Liu et al . ( 2015 ) ; Du et al . ( 2017 ) study the finite-sample performance of TD-learning algorithms with primal-dual updates . As for nonlinear function approximation , to the best of our knowledge , the only convergent algorithm is the nonlinear-GTD algorithm proposed in Bhatnagar et al . ( 2009a ) , which is more pertinent to our results . Their analysis also depends on two-timescale stochastic approximation , where the variable in the faster timescale essentially solves a linear equation , thus converges to the global optimum . However , in our problem , both the two variables solves nonconvex problems , which adds more challenge in the analysis . In addition , we mainly consider the problem of Q-learning and actor-critic , whereas their results only focus on policy evaluation . Moreover , their algorithm involves the Hessian of the value function and their theory requires the value function to be three times differentiable , whereas our algorithm only requires the gradient of the value function with less stringent assumptions . Thus , our results are not directly comparable with this work . Furthermore , for Q-learning and the actor-critic algorithm , existing results all consider linear function approximation . For Q-learning , Melo et al . ( 2008 ) ; Prashanth et al . ( 2014 ) study the convergence using stochastic approximation . In addition , the actor-critic algorithm is introduced in Sutton et al . ( 2000 ) ; Konda & Tsitsiklis ( 2000 ) , which also study its convergence with linear function approximation . Later , Kakade ( 2002 ) ; Peters & Schaal ( 2008 ) propose the natural actor-critic algorithm which updates the policy function using natural gradient descent ( Amari , 1998 ) . Convergence of actor-critic and natural actor-critic algorithms with linear function approximation are studied in Bhatnagar et al . ( 2009b ; 2008 ) ; Castro & Meir ( 2010 ) ; Bhatnagar ( 2010 ) ; Maei ( 2018 ) . 2 NOTATION AND BACKGROUND . In this section , we first lay out the notation that will be followed throughout this paper and then introduce some background knowledge on reinforcement learning . We stick to the following notation . For a measurable space with domain S , we denote by B ( S ) the set of bounded measurable functions on S. Let Lip ( S , L ) be the set of Lipschitz continuous functions on S with Lipschitz constant L. Let P ( S ) be the set of all probability measures over S. For any ν ∈M ( S ) and any measurable function f : S → R , we denote by ‖f‖ν , p the ` p-norm of f with respect to measure ν . For an integer k , we use ∆k to denote the probability simplex in Rk , and let [ k ] = { 1 , 2 , ... , k } . In reinforcement learning , the problem is modeled by a Markov decision process . A discounted MDP is defined by a tuple ( S , A , P , R , p0 , γ ) , where S is the set of states , A is the set of all possible actions , P : S ×A → P ( S ) is the Markov transition kernel , R : S ×A → R is the reward function , p0 ∈ P ( S ) is the distribution of the initial state s0 , and γ ∈ ( 0 , 1 ) is the discount factor . At any state s ∈ S , suppose action a ∈ A is executed , we use P ( · |s , a ) ∈ P ( S ) to denote the probability distribution of the next state , and let R ( s , a ) be the the immediate reward . For regularity , we assume that the rewards are uniformly bounded by Rmax . By taking action at at state st for each time step t ≥ 0 , the agent receives the discounted cumulative reward R = ∑ t≥0 γ t ·R ( st , at ) . Moreover , a policy π : S → P ( A ) is a action selection rule . Specifically , π ( a | s ) is the probability of selecting action a at state s. We define the state value function and the action value function of policy π as V π ( s ) = Eπ ( R | s0 = s , π ) and Qπ ( s , a ) = Eπ ( R | s0 = s , a0 = a ) , respectively . Here we use Eπ to indicate that the actions { at } t≥1 follow policy π . We define Bellman operator of T π : B ( S ) → B ( S ) by T πV ( s ) = E [ r ( st , at ) + γ · V ( st+1 ) | st = s ] , where at ∼ π ( · | st ) . By definition , we have V π = T πV π for any policy π . Since T ∗ is γ-contractive , V π is the unique fixed point of T π . The problem of estimating V π for a fixed policy π is called policy evaluation . Furthermore , in reinforcement learning , the goal is to obtain the policy that yields the maximal cumulative reward in expectation . To this end , we define the optimal value functionQ∗ : S×A → R by Q∗ ( s , a ) = supπ Q π ( s , a ) , ∀ ( s , a ) ∈ S × A , where the supremum is taken over all possible policies . Then it is known that the greedy policy with respect to Q∗ is the optimal policy . To estimate Q∗ with accuracy , most existing methods utilize the fact that Q∗ is the unique fixed point of the Bellman optimality operator T ∗ , which is defined by ( T ∗Q ) ( s , a ) =r ( s , a ) + γ · E [ max a∈A Q ( st+1 , a ) ∣∣ st = s , at = a ] . ( 2.1 ) The most well-known method for estimating Q∗ is the Q-learning algorithm ( Watkins & Dayan , 1992 ) , based upon a large family of algorithms are derived . In addition to methods that focus on estimating the value functions , another class of algorithms in reinforcement learning directly search for a good policy over a parametrized policy class { πω : ω ∈ Rp } directly , where ω is the parameter . Specifically , the goal is to maximize J ( ω ) = Eπω ( R ) = Es0∼p0 [ V πω ( s0 ) ] . The basis of these methods are provided by the policy gradient theorem ( Baxter & Bartlett , 2000 ) , which states that ( 1− γ ) · ∇ωJ ( ω ) = Es∼ρω , a∼πω ( · | s ) [ ∇ω log πω ( a | s ) ·Q πω ( s , a ) ] = Es∼ρθ , a∼πω ( · | s ) { ∇ω log πω ( a | s ) · [ R ( s , a ) + γ · V πω ( s′ ) − V πω ( s ) ] } , ( 2.2 ) where s′ ∼ P ( · | s , a ) is the next state of the MDP given ( s , a ) ∈ S ×A , and ρω ∈ P ( S ) is the state occupancy measure of πω , i.e. , ρω ( s ) = ( 1− γ ) · ∑ t≥0 γ t · Pπω ( st = s ) . We note that ( 2.2 ) brings the value function and the policy together . By updating θ using ( 2.2 ) with the value function Qπω or V πω estimated using temporal difference learning methods , we obtain the actor-critic algorithms ( Konda & Tsitsiklis , 2000 ) .

Summary: This paper interprets the fitted Q-learning, policy evaluation and actor-critic as a bi-level optimization problem. Then, it uses two-timescale stochastic approximation to prove their convergence under nonlinear function approximation. It provides interesting view of the these existing popular reinforcement learning algorithms that is widely used in DRL. However, there are several points to be addressed in the revision, which are mainly in some of its claims.