Reward is enough for convex MDPs

1 Introduction . In reinforcement learning ( RL ) , an agent learns how to map situations to actions so as to maximize a cumulative scalar reward signal . The learner is not told which actions to take , but instead must discover which actions lead to the most reward [ 64 ] . Mathematically , the RL problem can be written as finding a policy whose state occupancy has the largest inner product with a reward vector [ 55 ] , i.e. , the goal of the agent is to solve RL : max dπ∈K ∑ s , a r ( s , a ) dπ ( s , a ) , ( 1 ) where dπ is the state-action stationary distribution induced by policy π and K is the set of admissible stationary distributions ( see Definition 1 ) . A significant body of work is dedicated to solving the RL problem efficiently in challenging domains [ 45 , 62 ] . However , not all decision making problems of interest take this form . In particular we consider the more general convex MDP problem , Convex MDP : min dπ∈K f ( dπ ) , ( 2 ) where f : K → R is a convex function . Sequential decision making problems that take this form include Apprenticeship Learning ( AL ) , pure exploration , and constrained MDPs , among others ; see Table 1 . In this paper we prove the following claim : We can solve Eq . ( 2 ) by using any algorithm that solves Eq . ( 1 ) as a subroutine . In other words , any algorithm that solves the standard RL problem can be used to solve the more general convex MDP problem . More specifically , we make the following contributions . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Firstly , we adapt the meta-algorithm of Abernethy and Wang [ 3 ] for solving Eq . ( 2 ) . The key idea is to use Fenchel duality to convert the convex MDP problem into a two-player zero-sum game between the agent ( henceforth , policy player ) and an adversary that produces rewards ( henceforth , cost player ) that the agent must maximize [ 3 , 6 ] . From the agent ’ s point of view , the game is bilinear , and so for fixed rewards produced by the adversary the problem reduces to the standard RL problem with non-stationary reward ( Fig . 1 ) . Secondly , we propose a sample efficient policy player that uses a standard RL algorithm ( eg , [ 35 , 60 ] ) , and computes an optimistic policy with respect to the non-stationary reward at each iteration . In other words , we use algorithms that were developed to achieve low regret in the standard RL setup , to achieve low regret as policy players in the min-max game we formulate to solve the convex MDP . Our main result is that the average of the policies produced by the policy player converges to a solution to the convex MDP problem ( Eq . ( 2 ) ) . Inspired by this principle , we also propose a recipe for using deep-RL ( DRL ) agents to solve convex MDPs heuristically : provide the agent non-stationary rewards from the cost player . We explore this principle in our experiments . Finally , we show that choosing specific algorithms for the policy and cost players unifies several disparate branches of RL problems , such as apprenticeship learning , constrained MDPs , and pure exploration into a single framework , as we summarize in Table 1 . 2 Reinforcement Learning Preliminaries . In RL an agent interacts with an environment over a number of time steps and seeks to maximize its cumulative reward . We consider two cases , the average reward case and the discounted case . The Markov decision process ( MDP ) is defined by the tuple ( S , A , P , R ) for the average reward case and by the tuple ( S , A , P , R , γ , d0 ) for the discounted case . We assume an infinite horizon , finite state-action problem where initially , the state of the agent is sampled according to s0 ∼ d0 , then at each time t the agent is in state st ∈ S , selects action at ∈ A according to some policy π ( st , · ) , receives reward rt ∼ R ( st , at ) and transitions to new state st+1 ∈ S according to the probability distribution P ( · , st , at ) . The two performance metrics we consider are given by Javgπ = lim T→∞ 1 T E T∑ t=1 rt , J γ π = ( 1− γ ) E ∞∑ t=1 γtrt , ( 3 ) for the average reward case and discounted case respectively . The goal of the agent is to find a policy that maximizes Javgπ or J γ π . Any stationary policy π induces a state-action occupancy measure dπ , which measures how often the agent visits each state-action when following π . Let Pπ ( st = · ) be the probability measure over states at time t under policy π , then davgπ ( s , a ) = lim T→∞ 1 T E T∑ t=1 Pπ ( st = s ) π ( s , a ) , dγπ ( s , a ) = ( 1− γ ) E ∞∑ t=1 γtPπ ( st = s ) π ( s , a ) , for the average reward case and the discounted case respectively . With these , we can rewrite the RL objective in Eq . ( 3 ) in terms of the occupancy measure using the following well-known result , which for completeness we prove in Appendix B . Proposition 1 . For both the average and the discounted case , the agent objective function Eq . ( 3 ) can be written in terms of the occupancy measure as Jπ = ∑ s , a r ( s , a ) dπ ( s , a ) . Given an occupancy measure it is possible to recover the policy by setting π ( s , a ) = dπ ( s , a ) / ∑ a dπ ( s , a ) if ∑ a dπ ( s , a ) > 0 , and π ( s , a ) = 1/|A| otherwise . Accordingly , in this paper we shall formulate the RL problem using the state-action occupancy measure , and both the standard RL problem ( Eq . ( 1 ) ) and the convex MDP problem ( Eq . ( 2 ) ) are convex optimization problems in variable dπ . For the purposes of this manuscript we do not make a distinction between the average and discounted settings , other than through the convex polytopes of feasible occupancy measures , which we define next . Definition 1 ( State-action occupancy ’ s polytope [ 55 ] ) . For the average reward case the set of admissible state-action occupancies is Kavg = { dπ | dπ ≥ 0 , ∑ s , a dπ ( s , a ) = 1 , ∑ a dπ ( s , a ) = ∑ s′ , a′ P ( s , s′ , a′ ) dπ ( s ′ , a′ ) ∀s ∈ S } , and for the discounted case it is given by Kγ = { dπ | dπ ≥ 0 , ∑ a dπ ( s , a ) = ( 1− γ ) d0 ( s ) + γ ∑ s′ , a′ P ( s , s′ , a′ ) dπ ( s ′ , a′ ) ∀s ∈ S } . We note that being a polytope implies that K is a convex and compact set . The convex MDP problem is defined for the tuple ( S , A , P , f ) in the average cost case and ( S , A , P , f , γ , d0 ) in the discounted case . This tuple is defining a state-action occupancy ’ s polytope K ( Definition 1 ) , and the problem is to find a policy π whose state occupancy dπ is in this polytope and minimizes the function f ( Eq . ( 2 ) ) . 3 A Meta-Algorithm for Solving Convex MDPs via RL . To solve the convex MDP problem ( Eq . ( 2 ) ) we need to find an occupancy measure dπ ( and associated policy ) that minimizes the function f . Since both f : K → R and the set K are convex this is a convex optimization problem . However , it is a challenging one due to the nature of learning about the environment through stochastic interactions . In this section we show how to reformulate the convex MDP problem ( Eq . ( 2 ) ) so that standard RL algorithms can be used to solve it , allowing us to harness decades of work on solving vanilla RL problems . To do that we will need the following definition . Definition 2 ( Fenchel conjugate ) . For a function f : Rn → R ∪ { −∞ , ∞ } , its Fenchel conjugate is denoted f∗ : Rn → R ∪ { −∞ , ∞ } and defined as f∗ ( x ) : = supy x · y − f ( y ) . Remark 1 . The Fenchel conjugate function f∗ is always convex ( when it exists ) even if f is not . Furthermore , the biconjugate f∗∗ : = ( f∗ ) ∗ equals f if and only if f is convex and lower semicontinuous . Using this we can rewrite the convex MDP problem ( Eq . ( 2 ) ) as fOPT = min dπ∈K f ( dπ ) = min dπ∈K max λ∈Λ ( λ · dπ − f∗ ( λ ) ) = max λ∈Λ min dπ∈K ( λ · dπ − f∗ ( λ ) ) ( 4 ) where Λ is the closure of ( sub- ) gradient space { ∂f ( dπ ) |dπ ∈ K } , which is a convex set [ 3 , Theorem 4 ] . As both sets are convex , this is a convex-concave saddle-point problem and a zero-sum two-player game [ 54 , 49 ] , and we were able to swap the order of minimization and maximization using the minimax theorem [ 71 ] . With this we define the Lagrangian as L ( dπ , λ ) : = λ · dπ − f∗ ( λ ) . For a fixed λ ∈ Λ , minimizing the Lagrangian is a standard RL problem of the form of Eq . ( 1 ) , i.e. , equivalent to maximizing a reward r = −λ . Thus , one might hope that by producing an optimal dual variable λ ? we could simply solve d ? π = argmindπ∈K L ( · , λ ? ) for the optimal occupancy measure . However , the next lemma states that this is not possible in general . Lemma 1 . There exists an MDP M and convex function f for which there is no stationary reward r ∈ RS×A such that arg maxdπ∈K dπ · r = arg mindπ∈K f ( dπ ) . To see this note that for any reward r there is a deterministic policy that optimizes the reward [ 55 ] , but for some choices of f no deterministic policy is optimal , eg , when f is the negative entropy function . This result tells us that even if we have access to an optimal dual-variable we can not simply use it to recover the stationary distribution that solves the convex MDP problem in general . To overcome this issue we develop an algorithm that generates a sequence of policies { πk } k∈N such that the average converges to an optimal policy for Eq . ( 2 ) , i.e. , ( 1/K ) ∑K k=1 d k π → d ? π ∈ arg mindπ∈K f ( dπ ) . The algorithm we develop is described in Algorithm 1 and is adapted from the meta-algorithm described in Abernethy and Wang [ 3 ] . It is referred to as a meta-algorithm since it relies on supplied sub-routine algorithms Algπ and Algλ . The reinforcement learning algorithm Algπ takes as input a reward vector and returns a state-action occupancy measure dπ . The cost algorithm Algλ can be a more general function of the entire history . We discuss concrete examples of Algπ and Algλ in Section 4 . Algorithm 1 : meta-algorithm for convex MDPs 1 : Input : convex-concave payoff L : K × Λ→ R , algorithms Algλ , Algπ , K ∈ N 2 : for k = 1 , . . . , K do 3 : λk = Algλ ( d 1 π , . . . , d k−1 π ; L ) 4 : dkπ = Algπ ( −λk ) 5 : end for 6 : Return d̄Kπ = 1 K ∑K k=1 d k π , λ̄ K = 1K ∑K k=1 λ k In order to analyze this algorithm we will need a small detour into online convex optimization ( OCO ) . In OCO , a learner is presented with a sequence of K convex loss functions ` 1 , ` 2 , . . . , ` K : K → R and at each round k must select a point xk ∈ K after which it suffers a loss of ` k ( xk ) . At time period k the learner is assumed to have perfect knowledge of the loss functions ` 1 , . . . , ` k−1 . The learner wants to minimize its average regret , defined as R̄K : = 1 K ( K∑ k=1 ` k ( xk ) −min x∈K K∑ k=1 ` k ( x ) ) . In the context of convex reinforcement learning and meta-algorithm 1 , the loss functions for the cost player are ` kλ = −L ( · , λk ) , and for the policy player are ` kπ = L ( dkπ , · ) , with associated average regrets R̄πK and R̄ λ K . This brings us to the following theorem . Theorem 1 ( Theorem 2 , [ 3 ] ) . Assume that Algπ and Algλ have guaranteed average regret bounded as R̄πK ≤ K and R̄λK ≤ δK , respectively . Then Algorithm 1 outputs d̄Kπ and λ̄K satisfying mindπ∈K L ( dπ , λ̄K ) ≥ fOPT − K − δK and maxλ∈Λ L ( d̄Kπ , λ ) ≤ fOPT + K + δK . This theorem tells us that so long as the RL algorithm we employ has guaranteed low-regret , and assuming we choose a reasonable low-regret algorithm for deciding the costs , then the meta-algorithm will produce a solution to the convex MDP problem ( Eq . ( 2 ) ) to any desired tolerance , this is because fOPT ≤ f ( d̄Kπ ) = maxλ L ( d̄Kπ , λ ) ≤ fOPT + K +δK . For example , we shall later present algorithms that have regret bounded as K = δK ≤ O ( 1/ √ K ) , in which case we have f ( d̄Kπ ) − fOPT ≤ O ( 1/ √ K ) . ( 5 ) Non-Convex f . Remark 1 implies that the game maxλ∈Λ mindπ∈K ( λ · dπ − f∗ ( λ ) ) is concaveconvex for any function f , so we can solve it with Algorithm 1 , even for a non-convex f . From weak duality the value of the Lagrangian on the output of Algorithm 1 , L ( d̄π , λ̄ ) , is a lower bound on the optimal solution fOPT . In addition , since f ( dπ ) is always an upper bound on fOPT we have both an upper bound and a lower bound on the optimal value : L ( d̄π , λ̄ ) ≤ fOPT ≤ f ( d̄π ) .

Summary: The authors propose to consider a convex optimization problem over the polytope formed by the state-visitation frequency of an MDP. They constructed algorithms based on existing tools of sub-gradient descents, such as Frank-Wolfe, FTL and OMD, in conjunction with an oracle that (approximate)-solves an MDP with scalar rewards. Altogether, they provide a meta-algorithm that translates a scalar reward MDP oracle to an algorithm for the convex MDP. The authors also highlight several applications, such as the Apprenticeship Learning problem and the constrained MDP problem, of their formulation.