Continuous Deep Q-Learning in Optimal Control Problems: Normalized Advantage Functions Analysis

1 INTRODUCTION . The standard reinforcement learning ( RL ) setup consists of an agent interacting with an environment ( Sutton & Barto , 2018 ) . At each step of the interaction , the agent determines an action based on its policy and its current state , gets a reward , and makes a transition to the next state . An aim of the agent is to learn the policy that maximizes the sum of rewards . Q-learning ( Watkins & Dayan , 1992 ) is one of the most widespread algorithms for solving RL problems . According to this algorithm , the optimal action-value function ( Q-function ) is being found as a solution of the Bellman optimality equation . After the learning , the agent can act optimally by the learned Q-function . Initially , the Q-learning algorithm was applied for solving RL problems with finite state and action spaces . In this case , the Q-function can be represented by a finite table . For the case of a large or continuous state space , Q-learning has recently been extended to Deep Q-learning algorithm ( Mnih et al. , 2015 ) that allows to look for the approximate Q-function in the class of neural networks by means of the stochastic gradient descent . Deep Q-learning and its modifications have shown the efficiency for a range of challenging tasks ( Wang et al. , 2016 ; van Hasselt et al. , 2016 ; Schaul et al. , 2016 ; Hessel et al. , 2018 ) , however , note that this algorithm can not be directly applied for solving RL problems with continuous action spaces . The reason is that Deep Q-learning involves a maximizing of an approximate Q-function by the action variable on each step of the learning , which is a complex problem for continuous action spaces . Among various approaches to overcome this problem ( Lillicrap et al. , 2016 ; Haarnoja et al. , 2017 ; Kalashnikov et al. , 2018 ; Lim et al. , 2019 ; Ryu et al. , 2020 ; Lutter et al. , 2021 ) , we focus on an idea of the normalized advantage functions ( NAF ) algorithm ( Gu et al. , 2016 ) . This idea consists in the approximation of the Q-function by functions quadratic with respect to the action variable . It allows to get the maximum quite fast and precisely and solve some challenging control problems ( Gu et al. , 2017 ; Dong et al. , 2018 ; Ikemoto & Ushio , 2021 ) , but on the other hand , it brings up the question of classes of RL problems in which this approximation is acceptable . The presented paper describes one of the possible answers to this question . Note that the class of LQR problems ( Bradtke et al. , 1994 ) has the Q-functions quadratic with respect to the action variable . However , this class , being quite special , is not suitable for the description of complex controlled processes . In the paper , we consider a wider ( in some sense ) class of RL prob- lems . We consider RL problems which are obtained by the discretization of certain optimal control problems ( Bardi & Dolcetta , 1997 ) . The rationale for the consideration of such RL problems is that a lot of RL problems with continuous action spaces arise from control problems for mechanical or robotic systems ( Lillicrap et al. , 2016 ; Gu et al. , 2016 ; Haarnoja et al. , 2017 ; Gu et al. , 2017 ; Kalashnikov et al. , 2018 ) , whose dynamic are , in fact , described by ordinary differential equations . For the considered class of problems , based on the idea of NAF , we present a new family of quadratic functions and prove that , first , this family is sufficiently rich to approximately solve the Bellman optimality equation ( Theorem 1 ) , and second , any sufficiently accurate solution of the Bellman optimality equation allows to approximately obtain the optimal policy in the corresponding optimal control problem ( Theorem 2 ) . Moreover , we prove that it is impossible to get the same results for the original family of functions from Gu et al . ( 2016 ) ( Theorem 3 ) . From the obtained theoretical statements , we get some additional knowledge about the Q-function approximation by our family of quadratic functions and also provide several ways to use this knowledge in order to improve NAF . The experimental results confirm the efficiency of our improvements . 2 BACKGROUND . The standard reinforcement learning ( RL ) setup consists of an agent interacting with an environment ( Sutton & Barto , 2018 ) . This interaction is described by a Markov Decision Process ( MDP ) , which is a tuple ( S , U , P , R , ρ0 , γ ) , where S is a state space , U is an action space , P ( s′|s , u ) is a transition distribution , R ( s , u ) is a reward function , ρ0 ( s ) is an initial state distribution , and γ ∈ [ 0 , 1 ] is a discount factor . An aim of the agent is to learn its optimal policy µ∗ ( s ) that maximizes the value J ( µ ) = E [ ∞∑ i=0 γiR ( si , ui ) | s0 ∼ ρ0 ( s0 ) , ui = µ ( si ) , si+1 ∼ P ( si+1|si , ui ) , i = 0 , 1 , 2 , . . . ] . In the general statement of reinforcement learning problems , a policy of the agent can be stochastic , however , within this paper , we assume that the policy is deterministic . One of the most effective algorithms for solving RL problems is Q-learning ( Watkins & Dayan , 1992 ) . According to this algorithm , the agent explores the environment and looks for the optimal action-value function ( Q-function ) Q∗ ( s , u ) = sup µ E [ ∞∑ i=0 γiR ( si , ui ) | s0 = s , u0 = u , si+1 ∼ P ( si+1|si , ui ) , ui+1 = µ ( si+1 ) , i = 0 , 1 , 2 , . . . ] . as a solution of the Bellman optimality equation Q∗ ( s , u ) = E [ R ( s , u ) + γmax u′∈U Q∗ ( s ′ , u′ ) | s′ ∼ P ( s′|s , u ) ] . In other words , the agent solves the following minimization problem : sup s∈S , u∈U ∣∣∣Q ( s , u ) − E [ R ( s , u ) + γmax u′∈U Q ( s′ , u′ ) | s′ ∼ P ( s′|s , u ) ] ∣∣∣→ inf Q . ( 1 ) If the agent knows the function Q∗ ( s , a ) , it can act optimally by the greedy policy µ∗ ( s ) ∈ argmaxu∈U Q∗ ( s , u ) . Initially , the Q-learning algorithm was applied for solving RL problems with finite state and action spaces . In this case , the Q-function is represented by a finite table and problem ( 1 ) is finitedimensional . For the case of a large or continuous state space , Q-learning has recently been extended to Deep Q-learning algorithm ( Mnih et al. , 2015 ) that allows to look for the approximate Q-function in the class of neural networks Q ( x , u|θQ ) , where θQ is the parameter vector of the neural network . During the learning , the experiences ( si , ui , ri , si+1 ) are stored in the buffer D and simultaneously the parameter vector θQ is updated by means of the stochastic gradient descent minimizing the loss function L ( θQ ) = E [ ( Q ( s , u|θQ ) − y ) 2 | ( s , u , r , s′ ) ∼ U ( D ) ] , y = r + γmax u′∈U Q ( s′ , u′|θQ ) . ( 2 ) where U ( D ) is the uniform distribution on D. Deep Q-learning and its modifications are effective for a range of challenging tasks ( Wang et al. , 2016 ; van Hasselt et al. , 2016 ; Schaul et al. , 2016 ; Hessel et al. , 2018 ) , however , note that this algorithm can not be directly applied for solving RL problems with continuous action spaces . The reason is that Deep Q-learning involves the maximizing in ( 2 ) on each step of the learning , which is a complex problem for continuous U . Among various approaches to overcome this problem ( Lillicrap et al. , 2016 ; Haarnoja et al. , 2017 ; Kalashnikov et al. , 2018 ; Lim et al. , 2019 ; Ryu et al. , 2020 ; Lutter et al. , 2021 ) , we focus on an idea of the normalized advantage functions ( NAF ) algorithm ( Gu et al. , 2016 ) . This idea consists in the approximation of the Q-function by the following quadratic with respect to u functions : Q ( s , u|θQ ) = V ( s|θV ) +A ( s , u|θA ) , A ( s , u|θA ) = −1 2 ( u− µ ( s|θµ ) ) TP ( s|θP ) ( u− µ ( s|θµ ) ) , ( 3 ) where V ( s|θV ) , µ ( s|θµ ) , and P ( s|θP ) are neural networks with parameters θV , θµ , and θP , respectively ; P ( s|θP ) is a positive-definite square matrix for each s and θP ; θA = { θµ , θP } and θQ = { θA , θV } . Under the condition µ ( s|θµ ) ∈ U , ( 4 ) it allows to get the maximum and argmaximum values directly by values of V ( s|θV ) and µ ( s|θµ ) : max u∈U Q ( s , u|θQ ) = V ( s|θV ) , Argmax u∈U Q ( s , u|θQ ) = µ ( s|θµ ) , ( 5 ) but on the other hand , it brings up the question of classes of RL problems in which quadratic approximations is acceptable . Below , we describes one such class . 3 PROBLEM STATEMENT . In this section , we consider a certain class of optimal control problems and show that discrete approximations of these problems can be formalized as RL problems . Consider the following optimal control problem : it is required to maximize the functional J ( u ( · ) ) = σ ( x ( T ) ) − ∫ T 0 ( q ( t , x ( t ) ) + u ( t ) T r ( t , x ( t ) ) u ( t ) ) dt , ( 6 ) over all u ( · ) , where x ( · ) is the solution ( Filippov , 1988 , §1 ) of the differential equation d dt x ( t ) = f ( t , x ( t ) ) + g ( t , x ( t ) ) u ( t ) , t ∈ [ 0 , T ] , ( 7 ) under the initial condition x ( 0 ) = z . ( 8 ) Here t is the time variable , T > 0 is the terminal instant of time , x ( t ) ∈ Rn is the current state vector , u ( t ) ∈ U is the current control action vector forming the measurable function u ( · ) , U ⊂ Rm is the nonempty compact set , z ∈ Rn is the fixed initial state vector , f ( t , x ) ∈ Rn , g ( t , x ) ∈ Rn×m , q ( t , x ) ∈ R , r ( t , x ) ∈ Rm×m , ( t , x ) ∈ [ 0 , T ] × Rn are continuous with respect to t and continuously differentiable with respect to x functions , r ( t , x ) is the positive-definite matrix for each ( t , x ) ∈ [ 0 , T ] ×Rn , and σ ( x ) ∈ R , x ∈ Rn is the continuous function . We assume that there exists a constant cfg > 0 such that ∥f ( t , x ) + g ( t , x ) u∥ ≤ ( 1 + ∥x∥ ) cfg , ( t , x ) ∈ [ 0 , T ] × Rn , u ∈ U . ( 9 ) Note that , under these conditions , for each function u ( · ) , there exists a unique solution x ( · ) of equation ( 7 ) under the initial condition ( 8 ) ( Filippov , 1988 , §1 ) . Define the value function in optimal control problem ( 6 ) , ( 7 ) by V∗ ( t∗ , x∗ ) = sup u ( · ) ( σ ( x ( T ) ) − ∫ T t∗ ( q ( t , x ( t ) ) +u ( t ) T r ( t , x ( t ) ) u ( t ) ) dt ) , ( t∗ , x∗ ) ∈ [ 0 , T ] ×Rn , ( 10 ) where , for each u ( · ) , x ( · ) is the solution of equation ( 7 ) on the interval [ t∗ , T ] under the initial condition x ( t∗ ) = x∗ . Define the sets S = { ( t , x ) ∈ [ 0 , T ] × Rn : ∥x∥ ≤ ( 1 + ∥z∥ ) ecfgt − 1 } , S ( t ) = { x ∈ Rn : ( t , x ) ∈ S } . ( 11 ) Let k ∈ N , ∆tk = T/k , and ti = i∆tk , i ∈ 0 , k. Consider the corresponding discrete optimal control problem : it is required to maximize the function Jk ( u0 , u1 , . . . uk−1 ) = σ ( xk ) −∆tk k−1∑ i=0 ( q ( ti , xi ) + u T i r ( ti , xi ) ui ) , ( 12 ) over all ui ∈ U , i ∈ 0 , k − 1 , where ( x0 , x1 , . . . , xk ) is defined by x0 = z , xi+1 = xi + ( f ( ti , xi ) + g ( ti , xi ) ui ) ∆tk , i ∈ 0 , k − 1 . ( 13 ) Let us show that problem ( 12 ) , ( 13 ) can be formalized as the RL problem . First , we define the state and actions spaces , the initial state distribution , and the discount factor as follows : S = ∪ki=0 ( { ti } × S ( ti ) ) ∪ sT , U = U , ρ0 ( s0 ) = δ ( s0 = ( 0 , z ) ) , γ = 1 . ( 14 ) Here sT is some fictional terminal state , δ is Dirac delta distribution . Next , for every i ∈ 0 , k − 1 , x ∈ S ( ti ) , and u ∈ U , we define the transition distribution and the reward function by P ( s′|s = ( ti , x ) , u ) = δ ( s′ = ( ti+1 , x′ ) ) , R ( s = ( ti , x ) , u ) = − ( q ( ti , x ) + u T r ( ti , x ) u ) ∆tk , ( 15 ) where x′ = x + ( f ( ti , x ) + g ( ti , x ) u ) ∆tk . Taking into account ( 9 ) and ( 11 ) , one can prove the inclusion ( ti+1 , x′ ) ∈ S. Hence , the transition distribution P is well-defined . For i = k , we set P ( s′|s = ( tk , x ) , u ) = δ ( s′ = sT ) , R ( s = ( tk , x ) , u ) = σ ( x ) , x ∈ S ( tk ) , u ∈ U . ( 16 ) In order to make dynamical processes ( 13 ) formally infinite , we put P ( s′|sT , u ) = δ ( s′ = sT ) , R ( sT , u ) = 0 , u ∈ U . ( 17 ) Thus , we define MDP which describes the RL problem corresponding to problem ( 12 ) , ( 13 ) . Next , we show that such RL problems is suitable for using quadratic approximations of the Q-function .

Summary: -The authors prove that a discrete approximation of a class of optimal control problems can be recasted as an RL MDP. -The authors prove that the original NAF formulation of the Q-function cannot approximately solve the MDP defined above, and hence propose a new quadratic formulation of the Q-value. They apply their new formulation via Bounded, Reward-based, and Gradient-based BNAF. -The authors evaluate their proposed agent over 4 optimal control environments.