Average Reward Reinforcement Learning with Monotonic Policy Improvement

1 INTRODUCTION . The goal of Reinforcement Learning ( RL ) is to build agents that can learn high-performing behaviors through trial-and-error interactions with the environment . Broadly speaking , modern RL tackles two kinds of problems : episodic tasks and continuing tasks . In episodic tasks , the agent-environment interaction can be broken into separate distinct episodes , and the performance of the agent is simply the sum of the rewards accrued within an episode . Examples of episodic tasks include training an agent to learn to play Go ( Silver et al. , 2016 ; 2018 ) or Atari video games ( Mnih et al. , 2013 ) , where the episode terminates when the game ends . In continuing tasks , such as controlling robots with long operating lifespans ( Peters & Schaal , 2008 ; Schulman et al. , 2015 ; Haarnoja et al. , 2018 ) , there is no natural separation of episodes and the agent-environment interaction continues indefinitely . The performance of an agent in a continuing task is more difficult to quantify since even for bounded reward functions , the total sum of rewards is typically infinite . One way of making the long-term reward objective meaningful for continuing tasks is to apply discounting , i.e. , we maximize the discounted sum of rewards r0 + γr1 + γ2r2 + · · · for some discount factor γ ∈ ( 0 , 1 ) . This is guaranteed to be finite for any bounded reward function . However the discounted objective biases the optimal policy to choose actions that lead to high near-term performance rather than to high long-term performance . Such an objective — while useful in certain applications — is not appropriate when the goal is optimize long-term behavior . As argued in Chapter 10 of Sutton & Barto ( 2018 ) and in Naik et al . ( 2019 ) , a more natural objective is to use the average reward received by an agent over every time-step . While the average reward setting has been extensively studied in the classical Markov Decision Process literature ( Howard , 1960 ; Blackwell , 1962 ; Veinott , 1966 ; Bertsekas et al. , 1995 ) , it is much less commonly used in reinforcement learning . An important open question is whether recent advances in RL for the discounted reward criterion can be naturally generalized to the average reward setting . One major source of difficulty with modern DRL algorithms lies in controlling the step-size for policy updates . In order to have better control over step-sizes , Schulman et al . ( 2015 ) constructed a lower bound on the difference between the expected discounted return for two arbitrary policies π and π′ . The bound is a function of the divergence between these two policies and the discount factor . Schulman et al . ( 2015 ) showed that iteratively maximizing this lower bound generates a sequence of monotonically improved policies in terms of their discounted return . In this paper , we first show that the policy improvement theorem from Schulman et al . ( 2015 ) results in a non-meaningful bound in the average reward case . We then derive a novel result which lower bounds the difference of the average rewards based on the divergence of the policies . The bound depends on the average divergence between the policies and on the so-called Kemeny constant , which measures to what degree the unichain Markov chains associated with the policies are wellconnected . We show that iteratively maximizing this lower bound guarantees monotonic average reward policy improvement . Similar to the discounted case , the problem of maximizing the lower bound can be approximated with DRL algorithms which can be optimized using samples collected in the environment . We describe in detail two such algorithms : Average Reward TRPO ( ATRPO ) and Average Cost CPO ( ACPO ) , which are average reward versions of algorithms based on the discounted criterion ( Schulman et al. , 2015 ; Achiam et al. , 2017 ) . Using the MuJoCo simulated robotic benchmark , we carry out extensive experiments with the ATRPO algorithm and show that it is more effective than their discounted counterparts for these continuing control tasks . To our knowledge , this is one of the first paper to address DRL using the long-term average reward criterion . 2 PRELIMINARIES . Consider a Markov Decision Process ( MDP ) ( Sutton & Barto , 2018 ) ( S , A , P , r , µ ) where the state space S and action space A are assumed to be finite . The transition probability is denoted by P : S ×A×S → [ 0 , 1 ] , the bounded reward function r : S ×A → [ rmin , rmax ] , and µ : S → [ 0 , 1 ] is the initial state distribution . Let π = { π ( a|s ) : s ∈ S , a ∈ A } be a stationary policy , and Π is the set of all stationary policies . Here we discuss the two objective formulations for continuing control tasks : the average reward approach and discounted reward approach . Average Reward Approach In this paper , we will focus exclusively on unichain MDPs , which is when the Markov chain corresponding to every policy contains only one recurrent class and a finite but possibly empty set of transient states . The average reward objective is defined as : ρ ( π ) : = lim N→∞ 1 N E τ∼π [ N−1∑ t=0 r ( st , at ) ] = E s∼dπ a∼π [ r ( s , a ) ] . ( 1 ) Here dπ ( s ) : = limN→∞ 1N ∑N−1 t=0 P ( st = s|π ) = limt→∞ P ( st = s|π ) is the stationary state distribution under policy π , τ = ( s0 , a0 , . . . , ) is a sample trajectory . We use τ ∼ π to indicate that the trajectory is sampled from policy π , i.e . s0 ∼ µ , at ∼ π ( ·|st ) , and st+1 ∼ P ( ·|st , at ) . In the unichain case , the average reward ρ ( π ) is state-independent for any policy π ( Bertsekas et al. , 1995 ) . We express the average-reward value function as V π ( s ) : = Eτ∼π [ ∑∞ t=0 ( r ( st , at ) − ρ ( π ) ) ∣∣∣∣s0 = s ] and action-value function as Qπ ( s , a ) : = Eτ∼π [ ∑∞ t=0 ( r ( st , at ) − ρ ( π ) ) ∣∣∣∣s0 = s , a0 = a ] . We define the average reward advantage function as Aπ ( s , a ) : = Qπ ( s , a ) − V π ( s ) . Discounted Reward Approach For some discount factor γ ∈ ( 0 , 1 ) , the discounted reward objective is defined as ργ ( π ) : = E τ∼π [ ∞∑ t=0 γtr ( st , at ) ] = 1 1− γ E s∼dπ , γ a∼π [ r ( s , a ) ] . ( 2 ) where dπ , γ ( s ) : = ( 1 − γ ) ∑∞ t=0 γ tP ( st = s|π ) is known as the future discounted state visitation distribution under policy π . Note that unlike the average reward objective , the discounted objective depends on the initial state distribution µ . It can be easily shown that dπ , γ ( s ) → dπ ( s ) for all s as γ → 1 . The discounted value function is defined as V πγ ( s ) : = Eτ∼π [ ∑∞ t=0 γ tr ( st , at ) ∣∣∣∣s0 = s ] and discounted action-value function Qπγ ( s , a ) : = Eτ∼π [ ∑∞ t=0 γ tr ( st , at ) ∣∣∣∣s0 = s , a0 = a ] . Finally , the discounted advantage function is defined as Aπγ ( s , a ) : = Q π γ ( s , a ) − V πγ ( s ) . It is well-known that limγ→1 ( 1− γ ) ργ ( π ) = ρ ( π ) , implying that the discounted and average reward objectives are equivalent in the limit as γ approaches 1 ( Blackwell , 1962 ) . We will further discuss the relationship between the discounted and average reward value functions in the supplementary materials and prove that limγ→1Aπγ ( s , a ) = A π ( s , a ) ( see Corollary A.1 ) . 3 MONTONICALLY IMPROVEMENT GUARANTEES FOR DISCOUNTED RL . In many modern RL literature ( Schulman et al. , 2015 ; 2017 ; Abdolmaleki et al. , 2018 ; Vuong et al. , 2019 ) , algorithms iteratively update policies within a local region , i.e. , at iteration k we find policy πk+1 by maximizing ργ ( π ) within some region D ( π , πk ) ≤ δ for some divergence measure D. This approach allows us to control the step-size of each update using different choices of D and δ which can lead to better sample efficiency ( Peters & Schaal , 2008 ) . Schulman et al . ( 2015 ) derived a policy improvement bound based on a specific choice of D : ργ ( πk+1 ) − ργ ( πk ) ≥ 1 1− γ E s∼dπk , γ a∼πk+1 [ Aπkγ ( s , a ) ] − C ·max s [ DTV ( πk+1 ‖ πk ) [ s ] ] ( 3 ) where DTV ( π′ ‖ π ) [ s ] : = 12 ∑ a |π′ ( a|s ) − π ( a|s ) | is the total variation divergence for policies π and π′ , and C is some constant which does not depend on the divergence term DTV . Schulman et al . ( 2015 ) showed that by choosing πk+1 such that the right hand side of ( 3 ) is maximized , we are guaranteed to have ργ ( πk+1 ) ≥ ργ ( πk ) . This provided the theoretical foundation for an entire class of scalable policy optimization algorithms based on efficiently maximizing the right-hand-side of ( 3 ) ( Schulman et al. , 2015 ; 2017 ; Wu et al. , 2017 ; Abdolmaleki et al. , 2018 ; Vuong et al. , 2019 ) . A natural question arises here is whether the iterative procedure described by Schulman et al . ( 2015 ) also guarantees improvement w.r.t . the average reward . Since the discounted and average reward objectives are equivalent when γ → 1 , one may assume that we can also lower bound the policy performance difference of the average reward objective by letting γ → 1 for the bounds in Schulman et al . ( 2015 ) . Unfortunately this results in a non-meaningful bound . We will demonstrate this through a similar policy improvement bound from Achiam et al . ( 2017 ) based on the average divergence but a similar argument can be made for the original bound from Schulman et al . ( 2015 ) ( see supplementary material for proof and discussion ) . Proposition 1 . Consider the following bound from Achiam et al . ( 2017 ) D−π , γ ( π ′ ) ≤ ργ ( π′ ) − ργ ( π ) ≤ D+π , γ ( π′ ) ( 4 ) where D±π , γ ( π ′ ) = 1 1− γ E s∼dπ a∼π [ π′ ( a|s ) π ( a|s ) Aπγ ( s , a ) ] ± 2γ γ ( 1− γ ) 2 E s∼dπ [ DTV ( π ′ ‖ π ) [ s ] ] and γ = maxs ∣∣Ea∼π′ [ Aπγ ( s , a ) ] ∣∣ . We have : lim γ→1 ( 1− γ ) D±π , γ ( π′ ) = ±∞ ( 5 ) Since limγ→1 ( 1− γ ) ( ργ ( π′ ) − ργ ( π ) ) = ρ ( π′ ) − ρ ( π ) , Proposition 1 says ( 4 ) becomes trivial when used on the average reward . This result is discouraging as it shows that the policy improvement guarantee from Schulman et al . ( 2015 ) does not appear to generalize to the average reward setting . In the next section , we will derive an alternative policy improvement bound for the average reward objective which can be used to generate monotonically improved policies w.r.t . the average reward .

Summary: This paper advocates for the use of the average reward objective in long horizon, continual reinforcement learning settings, and it examines whether a limiting argument for the discount is sufficient for extending theoretical results from the discounted setting to the average-reward setting. In the case of policy optimization, the paper shows the monotonic improvement bound from Schulman et al. (2015) becomes vacuous as gamma approaches one. The rest of the paper presents theory to support a new, non-vacuous bound, which leads to a new algorithm analogous to TRPO (A-TRPO). The paper additionally shows how their main theorem can be applied to constrained MDPs in Section  5.2. Empirical results are provided in three MuJoCo domains modified to provide non-episodic experience.