Success at any cost: value constrained model-free continuous control

1 INTRODUCTION . Deep Reinforcement Learning ( RL ) has achieved great successes over the last couple of years , enabling learning of effective policies from high-dimensional input , such as pixels , on complicated tasks . However , compared to problems with discrete action spaces , continuous control problems with high-dimensional continuous state-action spaces – as often encountered in robotics – have proven much more challenging . Beyond the issue of exploration in high-dimensional continuous action spaces , RL algorithms rarely learn policies that produce smooth control signals when just optimizing for success . Instead , policies often exhibit control signals that switch between extreme values at high-frequency , often colloquially referred to as bang-bang control . Smoothness , however , is a desirable property in most real-world control problems . Unnecessary oscillations are not only energy inefficient , they also exert stress on a physical system by exciting second-order dynamics and increasing wear and tear on structural elements and actuators . To regularize the behavior , one can add penalties to the reward function . As a result , the reward function is composed of positive reward for achieving the goal and negative reward ( penalties ) for control action discontinuities or high energy use . This effectively casts the problem into a multiobjective optimization setting , where – depending on the ratio between the reward and the different penalties – different behaviors may be achieved . While every ratio will have its optimal policy , finding the ratio that results in the desired behavior , i.e . smooth control while still achieving an acceptable task success rate , is not always trivial and requires excessive hyperparameter tuning . Often , one must find different hyperparameter settings for different reward-penalty trade-offs or tasks . The process of finding these parameters is tedious and cumbersome , and may prevent robust 1Videos available at https : //sites.google.com/view/minitauriclr2019 general solutions . In this paper we rephrase the problem : instead of trying to find the right ratios between reward and penalties , we regularize the optimization problem by adding constraints , thereby reducing its effective dimensionality . More specifically , we propose to minimize the penalty with respect to a lower bound on the success rate of the task . Using a Lagrangian relaxation technique , we introduce cost coefficients for each of the imposed constraints that are tuned automatically during the optimization process . In this way we can find the optimal trade-off between reward and costs ( that also satisfies the imposed constraints ) automatically . By making the cost multipliers state-dependent , and adapting them alongside the policy , we can not only impose constraints on expected reward or cost , but also on their instantaneous values . Such point-wise constraints allow for much tighter control over the behavior of the policy , since a constraint that is satisfied only in overall expectation could still be violated momentarily . Finally , the entire constrained optimization procedure that we introduce can furthermore be conditioned on the constraint bounds themselves in order to learn a single , bound-conditioned policy that is able dynamically trade-off reward and penalties . This allows us to , for example , learn energy-efficient locomotion at a range of different velocities . The contributions of this work are ( i ) we regress state-dependent Lagrangian multipliers with a neural network in order to generalize across states , ( ii ) we impose structure to the critic by simultaneously learning both reward and value estimates as well as the coefficient to trade them off in a single model , and finally ( iii ) we train a bound-conditioned policy that is optimized for a range of bounds . Our approach , as described in more detail in Section 3 , is general and flexible in that it can be applied to any value-based RL algorithm and any number of constraints . We evaluate our approach on continuous control problems in Section 4 using tasks from the DM Control Suite ( Tassa et al. , 2018 ) and a ( precisely simulated ) locomotion task with the Minitaur quadruped . 2 BACKGROUND AND RELATED WORK . We consider the classical Markov Decision Process ( MDP ) setting ( Sutton & Barto , 1998 ) , where an agent sequentially interacts with an environment . More precisely , the agent observes the state of the environment s and decides on which action to take according to a policy a ∼ π ( s | s ) . Executing the action in the environment , then , causes a state transition . Each transition has an associated reward defined by some utility function r ( s , a ) . The goal of the agent is to maximize the expected sum of rewards , also known as the return , maxπ Es , a∼π [ ∑ t r ( st , at ) ] . While some tasks have a welldefined reward , such as the increase in score when playing a game , for many others the objective is not as easily defined . Designing reward functions that produce a desired behavior policy can thus be extremely difficult , even in the single-objective case ( e.g . Popov et al. , 2017 ; Amodei et al. , 2016 ) . Multi-Objective RL ( MORL ) problems arise in many domains , including robotics , and have been covered by a rich body of literature ( see e.g . Roijers et al. , 2013 , for a recent review ) , suggesting a variety of solution strategies . For instance , Mossalam et al . ( 2016 ) devise a Deep RL algorithm that implements an outer loop method and repeatedly calls a single-objective solver . Mannor & Shimkin ( 2004 ) propose an algorithm for learning in a stochastic game setting with vector valued rewards ( their approach is based on approachability of a target set in the reward space ) . However , most of these approaches explicitly recast the multi-objective problem into a single-objective problem ( that is amenable to existing methods ) , where one aims to find the trade-off between the different objectives that yields the desired result . In contrast , we aim for a method that automatically trades off different components in the objective to achieve a particular goal . To achieve this , we cast the problem in the framework of Constrained Markov Decision Processes ( CMDPs ) ( Altman , 1999 ) . CMDPs have been considered in a variety of works , including in the robotics and control literature . For instance , Achiam et al . ( 2017 ) and Dalal et al . ( 2018 ) focus on constraints motivated by safety concerns and propose algorithms that ensure that constraints remain satisfied at all times . These works , however , assume that the initial policy already satisfies the constraint , which is not the case when the constraint involves the task success rate ; as in this work . The motivation for the work by Tessler et al . ( 2018 ) is similar to ours . In contrast to our work , their approach maximizes reward subject to a constraint on the cost and enforces constraints only in expectation . Additionally , as an advance over the existing literature , we explicitly learn separate values for the reward and cost , as well as state-dependent coefficients that enable us to trade off the two in the policy optimization . Constraint-based formulations are also used frequently in single-objective policy search algorithms where bounds on the policy divergence are employed to control the rate of change in the policy from one iteration to the next ( e.g . Peters & Mülling , 2010 ; Levine & Koltun , 2013 ; Schulman et al. , 2015 ; Abdolmaleki et al. , 2018 ) . Our use of constraints , while similar in the employed methods , can be seen as orthogonal to the idea of using constraints to bound the rate of change in a policy . While we note that our approach can be applied to any value-based off-policy method , we make use of the method described in Maximum a Posteriori Policy Optimisation ( MPO ) ( Abdolmaleki et al. , 2018 ) as the underlying policy optimization algorithm – without loss of any generality of our method . MPO is an actor-critic algorithm that is known to yield robust policy improvement . In each policy improvement step , for each state sampled from replay buffer , MPO creates a population of actions . Subsequently , these actions are re-weighted based on their estimated values such that better actions will have higher weights . Finally , MPO uses a supervised learning step to fit a new policy in continuous state and action space . See Abdolmaleki et al . ( 2018 ) and Appendix A for more details . 3 CONSTRAINED OPTIMIZATION FOR CONTROL . We consider MDPs where we have both a reward and cost , r ( s , a ) and c ( s , a ) , which are functions of state s and action a . The goal is to automatically find a probabilistic policy π ( a|s ; θ ) ( with parameter θ ) that trades-off between maximizing the ( expected ) reward and minimizing the cost – in order to achieve the desired behavior . In the case of continuous control , desirable behavior would be solving the task ( e.g . stable swing up in cart-pole ) while minimizing other quantities , such as control effort or energy . In effect we want to optimize the total return subject to a penalty proportional to the total cost , i.e . maxπ Es , a∼π [ ∑ t r ( st , at ) − α · c ( st , at ) ] , where we take maxπ to mean maximizing the objective with respect to the policy parameters θ . The expectation over states s is with respect to the state visitation probability under the policy pπ ( s ) . The problem of finding the right trade-off then becomes a matter of finding a good value for α . Finding this trade-off is often non-trivial . An alternative way of looking at this dilemma is to take a multi-objective optimization perspective . Instead of fixing α , we can optimize for it simultaneously and can obtain different Pareto-optimal solutions for different values of α . In addition , to ease the definition of a desirable regime for α , one can consider imposing hard constraints on the cost to reduce dimensionality ( Deb , 2014 ) , instead of linearly combining the different objectives . Defining such hard constraints is often more intuitive than trying to manually tune coefficients . For example , in locomotion , it is easier to define desired behavior in terms of a lower bound on speed or an upper bound on an energy cost . 3.1 CONSTRAINED MDPS . The constrained perspective outlined above can be formalized as CMDPs ( Altman , 1999 ) . While a constraint can be placed on either the reward or the cost , in this work we consider a lower bound on the expected total return ( although the theory derived below equivalently applies to constraints on cost ) , i.e . minπ Es , a∼π [ ∑ t c ( st , at ) ] , s.t . Es , a∼π [ ∑ t r ( st , at ) ] ≥ R̄ , where R̄ is the minimum desired return . In the case of an infinite horizon with a given stationary state distribution , the constraint can instead be formulated for the per-step reward , i.e . Es , a∼π [ r ( s , a ) ] ≥ r̄ . In practice one often optimizes the γ-discounted return in both cases . To apply model-free RL methods to this problem we first define an estimate of the expected discounted return for a given policy as the action-value function Qr ( s , a ) = Es , a∼π [ ∑ t γ t · r ( st , at ) |s0 = s , a0 = a ] . Further , let Qc ( s , a ) denote the similarly constructed expected discounted cost action-value function . Equipped with these value functions , we can then recast the CMDP in value-space , where V̄r = r̄/ ( 1− γ ) ( i.e . scaling the desired reward r̄ with the limit of the converging sum over discounts ) : min π Es , a∼π [ Qc ( s , a ) ] , s.t . Es , a∼π [ Qr ( s , a ) ] ≥ V̄r . ( 1 )

Summary: This paper proposes an approach for mitigating issues associated with high-frequency/amplitude control signals that may be obtained when one applies reinforcement learning algorithms to continuous control tasks. The approach taken by the paper is to solve a constrained optimization problem, where the constraint imposes a (potentially state-dependent) lower bound on the reward. This is done by using a Lagrangian relaxation that learns the parameters of a control policy that satisfies the desired constraints (and also learns the Lagrange multipliers). The presented approach is demonstrated on a cart-pole swing-up task as well as a quadruped locomotion task.