Reward is enough for convex MDPs

1 Introduction . In reinforcement learning ( RL ) , an agent learns how to map situations to actions so as to maximize a cumulative scalar reward signal . The learner is not told which actions to take , but instead must discover which actions lead to the most reward [ 64 ] . Mathematically , the RL problem can be written as finding a policy whose state occupancy has the largest inner product with a reward vector [ 55 ] , i.e. , the goal of the agent is to solve RL : max dπ∈K ∑ s , a r ( s , a ) dπ ( s , a ) , ( 1 ) where dπ is the state-action stationary distribution induced by policy π and K is the set of admissible stationary distributions ( see Definition 1 ) . A significant body of work is dedicated to solving the RL problem efficiently in challenging domains [ 45 , 62 ] . However , not all decision making problems of interest take this form . In particular we consider the more general convex MDP problem , Convex MDP : min dπ∈K f ( dπ ) , ( 2 ) where f : K → R is a convex function . Sequential decision making problems that take this form include Apprenticeship Learning ( AL ) , pure exploration , and constrained MDPs , among others ; see Table 1 . In this paper we prove the following claim : We can solve Eq . ( 2 ) by using any algorithm that solves Eq . ( 1 ) as a subroutine . In other words , any algorithm that solves the standard RL problem can be used to solve the more general convex MDP problem . More specifically , we make the following contributions . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Firstly , we adapt the meta-algorithm of Abernethy and Wang [ 3 ] for solving Eq . ( 2 ) . The key idea is to use Fenchel duality to convert the convex MDP problem into a two-player zero-sum game between the agent ( henceforth , policy player ) and an adversary that produces rewards ( henceforth , cost player ) that the agent must maximize [ 3 , 6 ] . From the agent ’ s point of view , the game is bilinear , and so for fixed rewards produced by the adversary the problem reduces to the standard RL problem with non-stationary reward ( Fig . 1 ) . Secondly , we propose a sample efficient policy player that uses a standard RL algorithm ( eg , [ 35 , 60 ] ) , and computes an optimistic policy with respect to the non-stationary reward at each iteration . In other words , we use algorithms that were developed to achieve low regret in the standard RL setup , to achieve low regret as policy players in the min-max game we formulate to solve the convex MDP . Our main result is that the average of the policies produced by the policy player converges to a solution to the convex MDP problem ( Eq . ( 2 ) ) . Inspired by this principle , we also propose a recipe for using deep-RL ( DRL ) agents to solve convex MDPs heuristically : provide the agent non-stationary rewards from the cost player . We explore this principle in our experiments . Finally , we show that choosing specific algorithms for the policy and cost players unifies several disparate branches of RL problems , such as apprenticeship learning , constrained MDPs , and pure exploration into a single framework , as we summarize in Table 1 . 2 Reinforcement Learning Preliminaries . In RL an agent interacts with an environment over a number of time steps and seeks to maximize its cumulative reward . We consider two cases , the average reward case and the discounted case . The Markov decision process ( MDP ) is defined by the tuple ( S , A , P , R ) for the average reward case and by the tuple ( S , A , P , R , γ , d0 ) for the discounted case . We assume an infinite horizon , finite state-action problem where initially , the state of the agent is sampled according to s0 ∼ d0 , then at each time t the agent is in state st ∈ S , selects action at ∈ A according to some policy π ( st , · ) , receives reward rt ∼ R ( st , at ) and transitions to new state st+1 ∈ S according to the probability distribution P ( · , st , at ) . The two performance metrics we consider are given by Javgπ = lim T→∞ 1 T E T∑ t=1 rt , J γ π = ( 1− γ ) E ∞∑ t=1 γtrt , ( 3 ) for the average reward case and discounted case respectively . The goal of the agent is to find a policy that maximizes Javgπ or J γ π . Any stationary policy π induces a state-action occupancy measure dπ , which measures how often the agent visits each state-action when following π . Let Pπ ( st = · ) be the probability measure over states at time t under policy π , then davgπ ( s , a ) = lim T→∞ 1 T E T∑ t=1 Pπ ( st = s ) π ( s , a ) , dγπ ( s , a ) = ( 1− γ ) E ∞∑ t=1 γtPπ ( st = s ) π ( s , a ) , for the average reward case and the discounted case respectively . With these , we can rewrite the RL objective in Eq . ( 3 ) in terms of the occupancy measure using the following well-known result , which for completeness we prove in Appendix B . Proposition 1 . For both the average and the discounted case , the agent objective function Eq . ( 3 ) can be written in terms of the occupancy measure as Jπ = ∑ s , a r ( s , a ) dπ ( s , a ) . Given an occupancy measure it is possible to recover the policy by setting π ( s , a ) = dπ ( s , a ) / ∑ a dπ ( s , a ) if ∑ a dπ ( s , a ) > 0 , and π ( s , a ) = 1/|A| otherwise . Accordingly , in this paper we shall formulate the RL problem using the state-action occupancy measure , and both the standard RL problem ( Eq . ( 1 ) ) and the convex MDP problem ( Eq . ( 2 ) ) are convex optimization problems in variable dπ . For the purposes of this manuscript we do not make a distinction between the average and discounted settings , other than through the convex polytopes of feasible occupancy measures , which we define next . Definition 1 ( State-action occupancy ’ s polytope [ 55 ] ) . For the average reward case the set of admissible state-action occupancies is Kavg = { dπ | dπ ≥ 0 , ∑ s , a dπ ( s , a ) = 1 , ∑ a dπ ( s , a ) = ∑ s′ , a′ P ( s , s′ , a′ ) dπ ( s ′ , a′ ) ∀s ∈ S } , and for the discounted case it is given by Kγ = { dπ | dπ ≥ 0 , ∑ a dπ ( s , a ) = ( 1− γ ) d0 ( s ) + γ ∑ s′ , a′ P ( s , s′ , a′ ) dπ ( s ′ , a′ ) ∀s ∈ S } . We note that being a polytope implies that K is a convex and compact set . The convex MDP problem is defined for the tuple ( S , A , P , f ) in the average cost case and ( S , A , P , f , γ , d0 ) in the discounted case . This tuple is defining a state-action occupancy ’ s polytope K ( Definition 1 ) , and the problem is to find a policy π whose state occupancy dπ is in this polytope and minimizes the function f ( Eq . ( 2 ) ) . 3 A Meta-Algorithm for Solving Convex MDPs via RL . To solve the convex MDP problem ( Eq . ( 2 ) ) we need to find an occupancy measure dπ ( and associated policy ) that minimizes the function f . Since both f : K → R and the set K are convex this is a convex optimization problem . However , it is a challenging one due to the nature of learning about the environment through stochastic interactions . In this section we show how to reformulate the convex MDP problem ( Eq . ( 2 ) ) so that standard RL algorithms can be used to solve it , allowing us to harness decades of work on solving vanilla RL problems . To do that we will need the following definition . Definition 2 ( Fenchel conjugate ) . For a function f : Rn → R ∪ { −∞ , ∞ } , its Fenchel conjugate is denoted f∗ : Rn → R ∪ { −∞ , ∞ } and defined as f∗ ( x ) : = supy x · y − f ( y ) . Remark 1 . The Fenchel conjugate function f∗ is always convex ( when it exists ) even if f is not . Furthermore , the biconjugate f∗∗ : = ( f∗ ) ∗ equals f if and only if f is convex and lower semicontinuous . Using this we can rewrite the convex MDP problem ( Eq . ( 2 ) ) as fOPT = min dπ∈K f ( dπ ) = min dπ∈K max λ∈Λ ( λ · dπ − f∗ ( λ ) ) = max λ∈Λ min dπ∈K ( λ · dπ − f∗ ( λ ) ) ( 4 ) where Λ is the closure of ( sub- ) gradient space { ∂f ( dπ ) |dπ ∈ K } , which is a convex set [ 3 , Theorem 4 ] . As both sets are convex , this is a convex-concave saddle-point problem and a zero-sum two-player game [ 54 , 49 ] , and we were able to swap the order of minimization and maximization using the minimax theorem [ 71 ] . With this we define the Lagrangian as L ( dπ , λ ) : = λ · dπ − f∗ ( λ ) . For a fixed λ ∈ Λ , minimizing the Lagrangian is a standard RL problem of the form of Eq . ( 1 ) , i.e. , equivalent to maximizing a reward r = −λ . Thus , one might hope that by producing an optimal dual variable λ ? we could simply solve d ? π = argmindπ∈K L ( · , λ ? ) for the optimal occupancy measure . However , the next lemma states that this is not possible in general . Lemma 1 . There exists an MDP M and convex function f for which there is no stationary reward r ∈ RS×A such that arg maxdπ∈K dπ · r = arg mindπ∈K f ( dπ ) . To see this note that for any reward r there is a deterministic policy that optimizes the reward [ 55 ] , but for some choices of f no deterministic policy is optimal , eg , when f is the negative entropy function . This result tells us that even if we have access to an optimal dual-variable we can not simply use it to recover the stationary distribution that solves the convex MDP problem in general . To overcome this issue we develop an algorithm that generates a sequence of policies { πk } k∈N such that the average converges to an optimal policy for Eq . ( 2 ) , i.e. , ( 1/K ) ∑K k=1 d k π → d ? π ∈ arg mindπ∈K f ( dπ ) . The algorithm we develop is described in Algorithm 1 and is adapted from the meta-algorithm described in Abernethy and Wang [ 3 ] . It is referred to as a meta-algorithm since it relies on supplied sub-routine algorithms Algπ and Algλ . The reinforcement learning algorithm Algπ takes as input a reward vector and returns a state-action occupancy measure dπ . The cost algorithm Algλ can be a more general function of the entire history . We discuss concrete examples of Algπ and Algλ in Section 4 . Algorithm 1 : meta-algorithm for convex MDPs 1 : Input : convex-concave payoff L : K × Λ→ R , algorithms Algλ , Algπ , K ∈ N 2 : for k = 1 , . . . , K do 3 : λk = Algλ ( d 1 π , . . . , d k−1 π ; L ) 4 : dkπ = Algπ ( −λk ) 5 : end for 6 : Return d̄Kπ = 1 K ∑K k=1 d k π , λ̄ K = 1K ∑K k=1 λ k In order to analyze this algorithm we will need a small detour into online convex optimization ( OCO ) . In OCO , a learner is presented with a sequence of K convex loss functions ` 1 , ` 2 , . . . , ` K : K → R and at each round k must select a point xk ∈ K after which it suffers a loss of ` k ( xk ) . At time period k the learner is assumed to have perfect knowledge of the loss functions ` 1 , . . . , ` k−1 . The learner wants to minimize its average regret , defined as R̄K : = 1 K ( K∑ k=1 ` k ( xk ) −min x∈K K∑ k=1 ` k ( x ) ) . In the context of convex reinforcement learning and meta-algorithm 1 , the loss functions for the cost player are ` kλ = −L ( · , λk ) , and for the policy player are ` kπ = L ( dkπ , · ) , with associated average regrets R̄πK and R̄ λ K . This brings us to the following theorem . Theorem 1 ( Theorem 2 , [ 3 ] ) . Assume that Algπ and Algλ have guaranteed average regret bounded as R̄πK ≤ K and R̄λK ≤ δK , respectively . Then Algorithm 1 outputs d̄Kπ and λ̄K satisfying mindπ∈K L ( dπ , λ̄K ) ≥ fOPT − K − δK and maxλ∈Λ L ( d̄Kπ , λ ) ≤ fOPT + K + δK . This theorem tells us that so long as the RL algorithm we employ has guaranteed low-regret , and assuming we choose a reasonable low-regret algorithm for deciding the costs , then the meta-algorithm will produce a solution to the convex MDP problem ( Eq . ( 2 ) ) to any desired tolerance , this is because fOPT ≤ f ( d̄Kπ ) = maxλ L ( d̄Kπ , λ ) ≤ fOPT + K +δK . For example , we shall later present algorithms that have regret bounded as K = δK ≤ O ( 1/ √ K ) , in which case we have f ( d̄Kπ ) − fOPT ≤ O ( 1/ √ K ) . ( 5 ) Non-Convex f . Remark 1 implies that the game maxλ∈Λ mindπ∈K ( λ · dπ − f∗ ( λ ) ) is concaveconvex for any function f , so we can solve it with Algorithm 1 , even for a non-convex f . From weak duality the value of the Lagrangian on the output of Algorithm 1 , L ( d̄π , λ̄ ) , is a lower bound on the optimal solution fOPT . In addition , since f ( dπ ) is always an upper bound on fOPT we have both an upper bound and a lower bound on the optimal value : L ( d̄π , λ̄ ) ≤ fOPT ≤ f ( d̄π ) .

Summary: This paper addresses an extended reinforcement learning setting with general utilities, where the agent's objective can be any convex function of the state-action distribution. Especially, it recasts the problem as a min-max game between a policy player and a cost player, and it describes a principled methodology to solve the game. Finally, it provides ways to instantiate the methodology with known algorithms, and a brief experimental evaluation in simple domains.