A Reduction Approach to Constrained Reinforcement Learning

1 INTRODUCTION . Contemporary approaches in reinforcement learning ( RL ) largely focus on optimizing the behavior of an agent against a single reward function . RL algorithms like value function methods ( Zou et al. , 2019 ; Zheng et al. , 2018 ) or policy optimization methods ( Chen et al. , 2019 ; Zhao et al. , 2017 ) are widely used in real-world tasks . This can be sufficient for simple tasks . However , for complicated applications , designing a reward function that implicitly defines the desired behavior can be challenging . For instance , applications concerning risk ( Geibel & Wysotzki , 2005 ; Chow & Ghavamzadeh , 2014 ; Chow et al. , 2017 ) , safety ( Chow et al. , 2018 ) or budget ( Boutilier & Lu , 2016 ; Xiao et al. , 2019 ) are naturally modelled by augmenting the RL problem with orthant constraints . Exploration suggestions , such as to visit all states as evenly as possible , can be modelled by using a vector to measure the behavior of the agent , and to find a policy whose measurement vector lies in a convex set ( Miryoosefi et al. , 2019 ) . To solve RL problem under constraints , existing methods either ensure convergence only on a specific family of RL algorithms , or treat the underlying RL algorithms as a black box oracle to find individual policy , and look for mixed policy that randomizes among these individual policies . Though the second group of methods has the advantage of working with arbitrary RL algorithms that best suit the underlying problem , existing methods have practically infeasible memory requirement . To get an -approximate solution , they require storing O ( 1/ ) individual policies , and an exact solution requires storing infinitely many policies . This limits the prevalence of such methods , especially when the individual policy uses deep neural networks . In this paper , we propose a novel reduction approach for the general convex constrained RL ( C2RL ) problem . Our approach has the advantage of the second group of methods , yet requires storing at most constantly many policies . For a vector-valued Markov Decision Process ( MDP ) and any given target convex set , our method finds a mixed policy whose measurement vector lies in the target convex set , using any off-the-shelf RL algorithm that optimizes a scalar reward as a RL oracle . To do so , the C2RL problem is reduced to a distance minimization problem between a polytope and a convex set , and a novel variant of Frank-Wolfe type algorithm is proposed to solve this distance minimization problem . To find an -approximate solution in an m-dimensional vector-valued MDP , our method only stores at most m + 1 policies , which improves from infinitely many O ( 1/ ) ( Le et al. , 2019 ; Miryoosefi et al. , 2019 ) to a constant . We also show this m + 1 constant is worstcase optimal to ensure convergence of RL algorithms using deterministic policies . Moreover , our method introduces no extra hyper-parameter , which is favorable for practical usage . A preliminary experimental comparison demonstrates the performance of the proposed method and the sparsity of the policy found . 2 RELATED WORK . For high dimensional constrained RL , one line of approaches incorporates the constraint as a penalty signal into the reward function , and makes updates in a multiple time-scale scheme ( Tessler et al. , 2018 ; Chow & Ghavamzadeh , 2014 ) . When used with policy gradient or actor-critic algorithms ( Sutton & Barto , 2018 ) , this penalty signal guides the policy to converge to a constraint satisfying one ( Paternain et al. , 2019 ; Chow et al. , 2017 ) . However , the convergence guarantee requires the RL algorithm can find a single policy that satisfies the constraint , hence ruling out methods that search for deterministic policies , such as Deep Q-Networks ( DQN ) ( Mnih et al. , 2013 ) , Deep Deterministic Policy Gradient ( DDPG ) ( Lillicrap et al. , 2015 ) and their variants ( Van Hasselt et al. , 2015 ; Wang et al. , 2016 ; Fujimoto et al. , 2018 ; Barth-Maron et al. , 2018 ) . Another line of approaches uses a game-theoretic framework , and does not tie to specific families of RL algorithm . The constrained problem is relaxed to a zero-sum game , whose equilibrium is solved by online learning ( Agarwal et al. , 2018 ) . The game is played repeatedly , each time any RL algorithm can be used to find a best response policy to play against a no-regret online learner . The mixed policy that uniformly distributed among all played policies can be shown to converge to an optimal policy of the constrained problem ( Freund & Schapire , 1999 ; Abernethy et al. , 2011 ) . Taking this approach , Le et al . ( 2019 ) uses Lagrangian relaxation to solve the orthant constraint case , and Miryoosefi et al . ( 2019 ) uses conic duality to solve the convex constraint case . However , since the convergence is established by the no-regret property , the policy found by these methods requires randomization among policies found during the learning process , which limits their prevalence . Different from the game-theoretic approaches , we reduce the C2RL to a distance minimization problem and propose a novel variant of Frank-Wolfe ( FW ) algorithm to solve it . Our result builds on recent finding that the standard FW algorithm emerges as computing the equilibrium of a special convex-convave zero sum game ( Abernethy & Wang , 2017 ) . This connects our approach with previous approaches from game-theoretic framework ( Agarwal et al. , 2018 ; Le et al. , 2019 ; Miryoosefi et al. , 2019 ) . The main advantage of our reduction approach is that the convergence of FW algorithm does not rely on the no-regret property of an online learner . Hence there is no need to introduce extra hyper-parameters , such as learning rate of the online learner , and intuitively , we can eliminate unnecessary policies to achieve better sparsity . To do so , we extend Wolfe ’ s method for minimum norm point problem ( Wolfe , 1976 ) to solve our distance minimization problem . Throughout the learning process , we maintain an active policy set , and constantly eliminate policies whose measurement vector are affinely dependent of others . Unlike norm function in Wolfe ’ s method , our objective function is not strongly convex . Hence we can not achieve the linear convergence of Wolfe ’ s method as shown in Lacoste-Julien & Jaggi ( 2015 ) . Instead , we analyze the complexity of our method based on techniques from Chakrabarty et al . ( 2014 ) . A theoretical comparison between our method and various approaches in constrained RL is provided in Table 1 . 3 PRELIMINARIES . A vector-valued Markov decision process can be identified by a tuple { S , A , β , P , c } , where S is a set of states , A is the set of actions and β is the initial state distribution . At the start of each episode , an initial state s0 is drawn following the distribution β . Then , at each step t = 0 , 1 , . . . , the agent observes a state st ∈ S and makes a decision to take an action at . After at is chosen , at the next observation the state evolves to state st+1 ∈ S with probability P ( st+1|st , at ) . However , instead of a scalar reward , in our setting , the agent receives an m-dimensional vector ct ∈ Rm that may implicitly contain measurements of reward , risk or violation of other constraints . The episode ends after a certain number of steps , called the horizon , or when a terminate state is reached . Actions are typically selected according to a policy π , where π ( s ) is a distribution over actions for any s ∈ S . Policies that take a single action for any state are deterministic policies , and can be identified by the mapping π : S 7→ A . The set of all deterministic policies is denoted by Π . For a discount factor γ ∈ [ 0 , 1 ) , the discounted long-term measurement vector of a policy π ∈ Π is defined as c ( π ) : = E ( T∑ t=0 γtct ( st , π ( st ) ) ) , ( 1 ) where the expectation is over trajectories generated by the described random process . Unlike unconstrained setting , for a constrained RL problem , it is possible that all feasible policies are non-deterministic ( see Appendix D for an example ) . This limits the usage of RL algorithms that search for deterministic policies in the setting of constrained RL problem . One workaround is to use mixed policies . For a set of policies U , a mixed policy is a distribution over U , and the set of all mixed policies over U is denoted by ∆ ( U ) . To execute a mixed policy µ ∈ ∆ ( U ) , we first select a policy π ∈ U according to π ∼ µ ( π ) , and then execute π for the entire episode . Altman ( 1999 ) shows that any c ( · ) achievable can be achieved by some mixed deterministic policies µ ∈ ∆ ( Π ) . Therefore , though an off-shelves RL algorithm may not converge to any constraint-satisfying policy , it can be used as a subroutine to find individual policies ( possibly deterministic ) , and a randomization among these policies can converge to a feasible policy . The discounted long-term measurement vector of a mixed policy µ ∈ ∆ ( Π ) is defined similarly c ( µ ) : = Eπ∼µ ( c ( π ) ) = ∑ π∈Π µ ( π ) c ( π ) . ( 2 ) For a mixed policy µ ∈ ∆ ( U ) , its active set is defined to be the set of policies with non-zero weights A : = { π ∈ U|µ ( π ) > 0 } . The memory requirement of storing µ , is then proportional to the size of its active set . Since a mixed policy can be interpreted as a convex combination of policies in its active set , in the following , the term sparsity of a mixed policy refers to the sparsity of this combination . Our learning problem , the convex constrained reinforcement learning ( C2RL ) , is to find a policy whose expected long-term measurement vector lies in a given convex set ; i.e. , for a given convex target set C ⊂ Rm , our target is to find µ∗ such that c ( µ∗ ) ∈ Ω ( C2RL ) . ( 3 ) Any policy µ∗ that satisfies c ( µ∗ ) ∈ Ω is called a feasible policy , and a C2RL problem is feasible if there exists some feasible policies . In the following , we assume the C2RL problem is feasible . 4 APPROACH , ALGORITHM AND ANALYSIS . We now show how the C2RL ( 3 ) can be reduced to a distance minimization problem ( 7 ) between a polytope and a convex set . A novel variant of Frank-Wolfe-type algorithm is then proposed to solve the distance minimization problem , followed by theoretic analysis about convergence and sparsity of the proposed method .

Summary: This paper presents a reduction approach to tackle the optimization problem of constrained RL. They propose a Frank-Wolfe type algorithm for the task, which avoids many shortcomings of previous methods, such as the memory complexity. They prove that their algorithm can find an $\epsilon$-approximate solution with $O(1/\epsilon)$ invocation. They also show the power of their algorithm with experiments in a grid-world navigation task, though the tasks looks relatively simple.