A Deeper Look at Discounting Mismatch in Actor-Critic Algorithms

1 INTRODUCTION . Actor-critic algorithms have enjoyed great success both theoretically ( Williams , 1992 ; Sutton et al. , 2000 ; Konda , 2002 ; Schulman et al. , 2015a ) and empirically ( Mnih et al. , 2016 ; Silver et al. , 2016 ; Schulman et al. , 2017 ; OpenAI , 2018 ) . There is , however , a longstanding gap between the theory behind actor-critic algorithms and how practitioners implement them . Let γ , γA , and γC be the discount factors for defining the objective , updating the actor , and updating the critic respectively . Theoretically , no matter whether γ = 1 or γ < 1 , we should always use γA = γC = γ ( Sutton et al. , 2000 ; Schulman et al. , 2015a ) or at least keep γA = γC if Blackwell optimality ( Veinott , 1969 ; Weitzman , 2001 ) 1 is considered . Practitioners , however , usually use γA = 1 and γC < 1 in their implementations ( Dhariwal et al. , 2017 ; Caspi et al. , 2017 ; Zhang , 2018 ; Kostrikov , 2018 ; Achiam , 2018 ; Liang et al. , 2018 ; Stooke & Abbeel , 2019 ) . Although this mismatch and its theoretical disadvantage have been recognized by Thomas ( 2014 ) ; Nota & Thomas ( 2020 ) , whether and why it yields benefits in practice has not been systematically studied . In this paper , we empirically investigate this mismatch from a representation learning perspective . We consider two scenarios separately . Scenario 1 : The true objective is undiscounted ( γ = 1 ) . The theory prescribes to use γA = γC = γ = 1 . Practitioners , however , usually use γA = γ = 1 but γC < 1 , introducing bias . We explain this mismatch with the following hypothesis : Hypothesis 1. γC < 1 optimizes a bias-variance-representation trade-off . It is easy to see that γC < 1 reduces the variance in bootstrapping targets . Besides this , we further provide empirical evidence showing that when γC < 1 , it may become easier to find a good representation compared to γC = 1 . Consequently , although using γC < 1 introduces bias , it can facilitate representation learning . For our empirical study , we make use of recently introduced techniques , such fixed horizon temporal different learning ( De Asis et al. , 2019 ) and distributional reinforcement learning ( Bellemare et al. , 2017 ) to disentangle the various effects the discount factor has on the learning process . Scenario 2 : The true objective function is discounted ( γ < 1 ) . Theoretically , there is a γt term for the actor update on a transition observed at time t in a trajectory ( Sutton et al. , 2000 ; Schulman 1Blackwell optimality states that , in finite MDPs , there exists a γ0 < 1 such that for all γ ≥ γ0 , the optimal policies for the γ-discounted objective are the same . et al. , 2015a ) . Practitioners , however , usually ignore this term while using a discounted critic , i.e. , γA = 1 and γC = γ < 1 are used . We explain this mismatch with the following hypothesis : Hypothesis 2 . Using γC = γ < 1 and γA = 1 is effectively similar to using γC = γA = γ < 1 plus an auxiliary loss that sometimes facilitates representation learning . Our empirical study involves implementing the auxiliary task explicitly by using an additional policy for optimizing the difference term between the loss of γA = 1 and the loss of γA < 1 . We also design new benchmarking environments where the sign of the reward function is flipped after a certain time step such that later transitions differ from earlier ones . In that setting , γA = 1 becomes harmful . 2 BACKGROUND Markov Decision Processes : We consider an infinite horizon MDP with a finite state space S , a finite action spaceA , a bounded reward function r : S → R , a transition kernel p : S × S × A → [ 0 , 1 ] , an initial state distribution µ0 , and a discount factor γ ∈ [ 0 , 1 ] .2 The initial state S0 is sampled from µ0 . At time step t , an agent in state St takes action At ∼ π ( ·|St ) , where π : A×S → [ 0 , 1 ] is the policy it follows . The agent then gets a reward Rt+1 . = r ( St ) and proceeds to the next state St+1 ∼ p ( ·|St , At ) . The return of the policy π at time step t is defined as Gt . = ∑∞ i=1 γ i−1Rt+i , which allows us to define the state value function vγπ ( S ) . = E [ Gt|St = s ] and the state-action value function qγπ ( s , a ) . = E [ Gt|St = s , At = a ] . We consider episodic tasks where we assume there is an absorbing state s∞ ∈ S such that r ( s∞ ) = 0 and p ( s∞|s∞ , a ) = 1 holds for any a ∈ A . When γ < 1 , vγπ and qγπ are always well defined . When γ = 1 , to ensure v γ π and q γ π are well defined , we further assume finite expected episode length . Let Tπs be a random variable denoting the first time step that an agent hits s∞ when following π given S0 = s. We assume Tmax . = supπ∈Π maxs E [ Tπs ] < ∞ , where π is parameterized by θ and Π is the corresponding function class . Similar assumptions are also used in stochastic shortest path problems ( e.g. , Section 2.2 of Bertsekas & Tsitsiklis ( 1996 ) ) . In our experiments , all the environments have a hard time limit of 1000 , i.e. , Tmax = 1000 . This is standard practice , classic RL environments also have an upper limit on their episode lengths ( e.g . 27k in Bellemare et al . ( 2013 , ALE ) ) . Following Pardo et al . ( 2018 ) , we add the ( normalized ) time step t in the state to keep the environment Markovian . We measure the performance of a policy π with Jγ ( π ) . = ES0∼µ0 [ vγπ ( S0 ) ] . Vanilla Policy Gradient : Sutton et al . ( 2000 ) compute∇θJγ ( π ) as ∇θJγ ( π ) . = ∑ s d γ π ( s ) ∑ a q γ π ( s , a ) ∇θπ ( a|s ) , ( 1 ) where dγπ ( s ) . = ∑∞ t=0 γ t Pr ( St = s|µ0 , p , π ) for γ < 1 and dγπ ( s ) . = E [ ∑TπS0 t=0 Pr ( St = s|S0 , p , π ) ] for γ = 1.3 Note dγπ remains well-defined for γ = 1 when Tmax < ∞ . In order to optimize the policy performance Jγ ( π ) , one can follow ( 1 ) and , at time step t , update θt as θt+1 ← θt + αγtAqγCπ ( St , At ) ∇θ log π ( At|St ) , ( 2 ) where α is a learning rate . If we replace qγCπ with a learned value function , the update rule ( 2 ) becomes an actor-critic algorithm , where the actor refers to π and the critic refers to the learned approximation of qγCπ . In practice , an estimate for v γC π instead of q γC π is usually learned . Theoretically , we should have γA = γC = γ . Practitioners , however , usually ignore the γtA term in ( 2 ) , and use γC < γA = 1 . What this update truly optimizes remains an open problem ( Nota & Thomas , 2020 ) . TRPO and PPO : To improve the stability of actor-critic algorithms , Schulman et al . ( 2015a ) propose Trust Region Policy Optimization ( TRPO ) , based on the performance improvement lemma : Lemma 1 . ( Theorem 1 in Schulman et al . ( 2015a ) ) For γ < 1 and any two policies π and π′ , Jγ ( π ′ ) ≥ Jγ ( π ) + ( ∑ s d γ π ( s ) ∑ a π ′ ( a|s ) Advγπ ( s , a ) ) − 4 maxs , a |Adv γ π ( s , a ) |γ ( π , π ′ ) ( 1−γ ) 2 , 2Following Schulman et al . ( 2015a ) , we consider r : S → R instead of r : S ×A → R for simplicity . 3Sutton et al . ( 2000 ) do not explicitly define dγπ when γ = 1 , which , however , can be easily deduced from Chapter 13.2 in Sutton & Barto ( 2018 ) . where Advγπ ( s , a ) . = Es′∼p ( ·|s , a ) [ r ( s ) + γvγπ ( s′ ) − vγπ ( s ) ] is the advantage , ( π , π′ ) . = maxsDKL ( π ( ·|s ) ||π′ ( ·|s ) ) , and DKL refers to the KL divergence . To facilitate our empirical study , we first make a theoretical contribution by extending Lemma 1 to the undiscounted setting . We have the following lemma : Lemma 2 . Assuming Tmax < ∞ , for γ = 1 and any two policies π and π′ , Jγ ( π ′ ) ≥ Jγ ( π ) + ( ∑ s d γ π ( s ) ∑ a π ′ ( a|s ) Advγπ ( s , a ) ) − 4 maxs , a |Advγπ ( s , a ) |T 2max ( π , π′ ) . The proof of Lemma 2 is provided in the appendix . A practical implementation of Lemmas 1 and 2 is to compute a new policy θ via gradient ascent on the clipped objective : L ( θ ) . = ∑∞ t=0 γ t A min { πθ ( At|St ) πθold ( At|St ) AdvγCπθold ( St , At ) , clip ( πθ ( At|St ) πθold ( At|St ) ) AdvγCπθold ( St , At ) } , ( 3 ) where St and At are sampled from πθold , and clip ( x ) . = max ( min ( x , 1 + ) , 1 − ) with a hyperparameter . Theoretically , we should have γA = γC , but practical algorithms like Proximal Policy Optimization ( Schulman et al. , 2017 , PPO ) usually use γC < γA = 1 . Policy Evaluation : We now introduce several policy evaluation techniques we use in our empirical study . Let v̂ be our estimate of vγπ . At time step t , Temporal Difference learning ( TD , Sutton ( 1988 ) ) updates v̂ as v̂ ( St ) ← v̂ ( St ) +α ( Rt+1 + γv̂ ( St+1 ) − v̂ ( St ) ) . Instead of the infinite horizon discounted returnGt , De Asis et al . ( 2019 ) propose to consider theH-step returnGHt . = ∑H i=1Rt+i . Correspondingly , the H-step value function is defined as vHπ ( s ) . = E [ GHt |St = s ] . We let v̂H be our estimate of vHπ . At time step t , De Asis et al . ( 2019 ) use the following update rule to learn v̂ H : v̂i ( St ) ← v̂i ( St ) + α ( Rt+1 + v̂i−1 ( St+1 ) − v̂i ( St ) ) ( i = 1 , . . . H ) , ( 4 ) where v̂0 ( s ) .= 0 . In other words , to learn v̂H , we need to learn { v̂i } i=1 , ... , H simultaneously . De Asis et al . ( 2019 ) call ( 4 ) Fixed Horizon Temporal Difference learning ( FHTD ) . As Gt is a random variable , Bellemare et al . ( 2017 ) propose to learn its full distribution instead of its expectation only , yielding the Distributional Reinforcement Learning ( RL ) paradigm . They use a categorical distribution with 51 atoms uniformly distributed in [ −Vmax , Vmax ] to approximate the distribution of Gt , where Vmax is a hyperparameter . In this paper , we refer to the corresponding policy evaluation algorithm as C51 . Methodology : We consider MuJoCo robot simulation tasks from OpenAI gym ( Brockman et al. , 2016 ) as our benchmark . Given its popularity in understanding deep RL algorithms ( Henderson et al. , 2017 ; Ilyas et al. , 2018 ; Engstrom et al. , 2019 ; Andrychowicz et al. , 2020 ) and designing new deep RL algorithms ( Fujimoto et al. , 2018 ; Haarnoja et al. , 2018 ) , we believe our empirical results are relevant to most practitioners . We choose PPO , a simple yet effective and widely used algorithm , as the representative actor-critic algorithm for our empirical study . PPO is usually equipped with generalized advantage estimation ( Schulman et al. , 2015b , GAE ) , which has a tunable hyperparameter γ̂ . The roles of γ and γ̂ are similar . To reduce its confounding effect , we do not use GAE in our experiments , i.e. , the advantage estimation for our actor is simply the TD errorRt+1 +γCv̂ ( St+1 ) − v̂ ( St ) . The PPO pseudocode we follow is provided in Alg . 1 in the appendix and we refer to it as the default PPO implementation . We use the standard architecture and optimizer across all tasks , in particular , the actor and the critic do not share layers . We conduct a thorough learning rate search in Ant for each algorithmic configuration ( i.e. , a curve in a figure ) and then use the same learning rate for all other tasks . When using FHTD and C51 , we also include H and Vmax in the grid search . All details are provided in the appendix . We report the average episode return of the ten most recent episodes against the number of interactions with the environment . Curves are averages over ten independent runs with shaded regions indicating standard errors .

Summary: The authors examine the commonly used paradigm of not discounting in the policy gradient objective. They propose two hypotheses relating to discounting. (1) discounting the critic improves representation learning. (2) undiscounted policy gradient is similar to discounting + an auxiliary loss. These hypotheses are studied through a series of empirical tests in the MuJoCo domain with PPO.