CAQL: Continuous Action Q-Learning

1 INTRODUCTION . Reinforcement learning ( RL ) has shown success in a variety of domains such as games ( Mnih et al. , 2013 ) and recommender systems ( RSs ) ( Gauci et al. , 2018 ) . When the action space is finite , valuebased algorithms such as Q-learning ( Watkins & Dayan , 1992 ) , which implicitly finds a policy by learning the optimal value function , are often very efficient because action optimization can be done by exhaustive enumeration . By contrast , in problems with a continuous action spaces ( e.g. , robotics ( Peters & Schaal , 2006 ) ) , policy-based algorithms , such as policy gradient ( PG ) ( Sutton et al. , 2000 ; Silver et al. , 2014 ) or cross-entropy policy search ( CEPS ) ( Mannor et al. , 2003 ; Kalashnikov et al. , 2018 ) , which directly learn a return-maximizing policy , have proven more practical . Recently , methods such as ensemble critic ( Fujimoto et al. , 2018 ) and entropy regularization ( Haarnoja et al. , 2018 ) have been developed to improve the performance of policy-based RL algorithms . Policy-based approaches require a reasonable choice of policy parameterization . In some continuous control problems , Gaussian distributions over actions conditioned on some state representation is used . However , in applications such as RSs , where actions often take the form of high-dimensional item-feature vectors , policies can not typically be modeled by common action distributions . Furthermore , the admissible action set in RL is constrained in practice , for example , when actions must lie within a specific range for safety ( Chow et al. , 2018 ) . In RSs , the admissible actions are often random functions of the state ( Boutilier et al. , 2018 ) . In such cases , it is non-trivial to define policy parameterizations that handle such factors . On the other hand , value-based algorithms are wellsuited to these settings , providing potential advantage over policy methods . Moreover , at least with linear function approximation ( Melo & Ribeiro , 2007 ) , under reasonable assumptions , Q-learning converges to optimality , while such optimality guarantees for non-convex policy-based methods are generally limited ( Fazel et al. , 2018 ) . Empirical results also suggest that value-based methods are more data-efficient and less sensitive to hyper-parameters ( Quillen et al. , 2018 ) . Of course , with large action spaces , exhaustive action enumeration in value-based algorithms can be expensive—one solution is to represent actions with continuous features ( Dulac-Arnold et al. , 2015 ) . The main challenge in applying value-based algorithms to continuous-action domains is selecting optimal actions ( both at training and inference time ) . Previous work in this direction falls into three broad categories . The first solves the inner maximization of the ( optimal ) Bellman residual loss using global nonlinear optimizers , such as the cross-entropy method ( CEM ) for QT-Opt ( Kalashnikov et al. , 2018 ) , gradient ascent ( GA ) for actor-expert ( Lim et al. , 2018 ) , and action discretization ( Uther & Veloso , 1998 ; Smart & Kaelbling , 2000 ; Lazaric et al. , 2008 ) . However , these approaches do not guarantee optimality . The second approach restricts the Q-function parameterization so that the optimization problem is tractable . For instance , one can discretize the state and action spaces and use a tabular Q-function representation . However , due to the curse of dimensionality , discretizations must generally be coarse , often resulting in unstable control . Millán et al . ( 2002 ) circumvents this issue by averaging discrete actions weighted by their Q-values . Wire-fitting ( Gaskett et al. , 1999 ; III & Klopf , 1993 ) approximates Q-values piecewise-linearly over a discrete set of points , chosen to ensure the maximum action is one of the extreme points . The normalized advantage function ( NAF ) ( Gu et al. , 2016 ) constructs the state-action advantage function to be quadratic , hence analytically solvable . Parameterizing the Q-function with an input-convex neural network ( Amos et al. , 2017 ) ensures it is concave . These restricted functional forms , however , may degrade performance if the domain does not conform to the imposed structure . The third category replaces optimal Q-values with a “ soft ” counterpart ( Haarnoja et al. , 2018 ) : an entropy regularizer ensures that both the optimal Q-function and policy have closed-form solutions . However , the sub-optimality gap of this soft policy scales with the interval and dimensionality of the action space ( Neu et al. , 2017 ) . Motivated by the shortcomings of prior approaches , we propose Continuous Action Q-learning ( CAQL ) , a Q-learning framework for continuous actions in which the Q-function is modeled by a generic feed-forward neural network.1 Our contribution is three-fold . First , we develop the CAQL framework , which minimizes the Bellman residual in Q-learning using one of several “ plug-andplay ” action optimizers . We show that “ max-Q ” optimization , when the Q-function is approximated by a deep ReLU network , can be formulated as a mixed-integer program ( MIP ) that solves max-Q optimally . When the Q-function has sufficient representation power , MIP-based optimization induces better policies and is more robust than methods ( e.g. , CEM , GA ) that approximate the max-Q solution . Second , to improve CAQL ’ s practicality for larger-scale applications , we develop three speed-up techniques for computing max-Q values : ( i ) dynamic tolerance ; ( ii ) dual filtering ; and ( iii ) clustering . Third , we compare CAQL with several state-of-the-art RL algorithms on several benchmark problems with varying degrees of action constraints . Value-based CAQL is generally competitive , and outperforms policy-based methods in heavily constrained environments , sometimes significantly . We also study the effects of our speed-ups through ablation analysis . 2 PRELIMINARIES . We consider an infinite-horizon , discounted Markov decision process ( Puterman , 2014 ) with states X , ( continuous ) action space A , reward function R , transition kernel P , initial state distribution β and discount factor γ ∈ [ 0 , 1 ) , all having the usual meaning . A ( stationary , Markovian ) policy π specifies a distribution π ( ·|x ) over actions to be taken at state x . Let ∆ be the set of such policies . The expected cumulative return of π ∈ ∆ is J ( π ) : = E [ ∑∞ t=0 γ trt | P , R , x0 ∼ β , π ] . An optimal policy π∗ satisfies π∗ ∈ arg maxπ∈∆ J ( π ) . The Bellman operator F [ Q ] ( x , a ) = R ( x , a ) + γ ∑ x′∈X P ( x ′|x , a ) maxa′∈AQ ( x′ , a′ ) over state-action value function Q has unique fixed point Q∗ ( x , a ) ( Puterman , 2014 ) , which is the optimal Q-function Q∗ ( x , a ) = E [ ∑∞ t=0 γ tR ( xt , at ) | x0 = x , a0 = a , π∗ ] . An optimal ( deterministic ) policy π∗ can be extracted from Q∗ : π∗ ( a|x ) = 1 { a = a∗ ( x ) } , where a∗ ( x ) ∈ arg maxaQ∗ ( x , a ) . For large or continuous state/action spaces , the optimal Q-function can be approximated , e.g. , using a deep neural network ( DNN ) as in DQN ( Mnih et al. , 2013 ) . In DQN , the value function Qθ is updated using the value label r + γmaxa′ Qθtarget ( x′ , a′ ) , where Qθtarget is a target Q-function . Instead of training these weights jointly , θtarget is updated in a separate iterative fashion using the previous θ for a fixed number of training steps , or by averaging θtarget ← τθ + ( 1 − τ ) θtarget for some small momentum weight τ ∈ [ 0 , 1 ] ( Mnih et al. , 2016 ) . DQN is off-policy—the target is valid no matter how the experience was generated ( as long as it is sufficiently exploratory ) . Typically , the loss is minimized over mini-batches B of past data ( x , a , r , x′ ) sampled from a large experience replay bufferR ( Lin & Mitchell , 1992 ) . One common loss function for trainingQθ∗ is mean squared Bellman error : minθ ∑|B| i=1 ( Qθ ( xi , ai ) − ri − γmaxa′ Qθtarget ( x′i , a′ ) ) 2 . Under this loss , RL can be viewed as ` 2-regression ofQθ ( · , · ) w.r.t . target labels r+γmaxa′ Qθtarget ( x′ , a′ ) . We augment DQN , using double Q-learning for more stable training ( Hasselt et al. , 2016 ) , whose loss is : min θ |B|∑ i=1 ( ri + γQθtarget ( x ′ i , arg max a′ Qθ ( x ′ i , a ′ ) ) −Qθ ( xi , ai ) ) 2 . ( 1 ) A hinge loss can also be used in Q-learning , and has connections to the linear programming ( LP ) formulation of the MDP ( Puterman ( 2014 ) ) . The optimal Q-network weights can be specified as : minθ 1 |B| ∑|B| i=1Qθ ( xi , ai ) + λ ( ri + γmaxa′∈AQθ ( x ′ i , a ′ ) −Qθ ( xi , ai ) ) + , where λ > 0 is a tun- 1Results can be extended to handle convolutional NNs , but are omitted for brevity . able penalty w.r.t . constraint : r+γmaxa′∈AQθ ( x′ , a′ ) ≤ Qθ ( x , a ) , ∀ ( x , a , r , x′ ) ∈ B . To stabilize training , we replace the Q-network of the inner maximization with the target Q-network and the optimal Q-value with the double-Q label , giving ( see Appendix A for details ) : min θ 1 |B| |B|∑ i=1 Qθ ( xi , ai ) + λ ( ri + γQθtarget ( x ′ i , arg max a′ Qθ ( x ′ i , a ′ ) ) −Qθ ( xi , ai ) ) + . ( 2 ) In this work , we assume the Q-function approximation Qθ to be a feed-forward network . Specifically , let Qθ be a K-layer feed-forward NN with state-action input ( x , a ) ( where a lies in a ddimensional real vector space ) and hidden layers arranged according to the equations : z1 = ( x , a ) , ẑj = Wj−1zj−1 + bj−1 , zj = h ( ẑj ) , j = 2 , . . . , K , Qθ ( x , a ) : = c > ẑK , 2 ( 3 ) where ( Wj , bj ) are the multiplicative and bias weights , c is the output weight of the Q-network , θ = ( c , { ( Wj , bj ) } K−1j=1 ) are the weights of the Q-network , ẑj denotes pre-activation values at layer j , and h ( · ) is the ( component-wise ) activation function . For simplicity , in the following analysis , we restrict our attention to the case when the activation functions are ReLU ’ s . We also assume that the action space A is a d-dimensional ` ∞-ball B∞ ( a , ∆ ) with some radius ∆ > 0 and center a . Therefore , at any arbitrary state x ∈ X the max-Q problem can be re-written as q∗x : = maxa∈AQθ ( x , a ) = max { ẑj } Kj=2 , { zj } K−1 j=2 { c > ẑK : z1 = ( x , a ) , a ∈ B∞ ( a , ∆ ) , eqs . ( 3 ) } . While the above formulation is intuitive , the nonlinear equality constraints in the neural network formulation ( 3 ) makes this problem non-convex and NP-hard ( Katz et al. , 2017 ) .

Summary: The paper proposes a novel value-based continuous control algorithm by formulating the problem as mixed-integer programming. With this formulation, the optimal action (corresponding to the maximum action value) can be found by solving the optimization problem at each time step. To reduce the time complexity of the optimization, the author proposes several variants to approximately solve the problem. Results on robotics control are presented. The proposed looks interesting and could be useful in practice.