A Deeper Look at Discounting Mismatch in Actor-Critic Algorithms

1 INTRODUCTION . Actor-critic algorithms have enjoyed great success both theoretically ( Williams , 1992 ; Sutton et al. , 2000 ; Konda , 2002 ; Schulman et al. , 2015a ) and empirically ( Mnih et al. , 2016 ; Silver et al. , 2016 ; Schulman et al. , 2017 ; OpenAI , 2018 ) . There is , however , a longstanding gap between the theory behind actor-critic algorithms and how practitioners implement them . Let γ , γA , and γC be the discount factors for defining the objective , updating the actor , and updating the critic respectively . Theoretically , no matter whether γ = 1 or γ < 1 , we should always use γA = γC = γ ( Sutton et al. , 2000 ; Schulman et al. , 2015a ) or at least keep γA = γC if Blackwell optimality ( Veinott , 1969 ; Weitzman , 2001 ) 1 is considered . Practitioners , however , usually use γA = 1 and γC < 1 in their implementations ( Dhariwal et al. , 2017 ; Caspi et al. , 2017 ; Zhang , 2018 ; Kostrikov , 2018 ; Achiam , 2018 ; Liang et al. , 2018 ; Stooke & Abbeel , 2019 ) . Although this mismatch and its theoretical disadvantage have been recognized by Thomas ( 2014 ) ; Nota & Thomas ( 2020 ) , whether and why it yields benefits in practice has not been systematically studied . In this paper , we empirically investigate this mismatch from a representation learning perspective . We consider two scenarios separately . Scenario 1 : The true objective is undiscounted ( γ = 1 ) . The theory prescribes to use γA = γC = γ = 1 . Practitioners , however , usually use γA = γ = 1 but γC < 1 , introducing bias . We explain this mismatch with the following hypothesis : Hypothesis 1. γC < 1 optimizes a bias-variance-representation trade-off . It is easy to see that γC < 1 reduces the variance in bootstrapping targets . Besides this , we further provide empirical evidence showing that when γC < 1 , it may become easier to find a good representation compared to γC = 1 . Consequently , although using γC < 1 introduces bias , it can facilitate representation learning . For our empirical study , we make use of recently introduced techniques , such fixed horizon temporal different learning ( De Asis et al. , 2019 ) and distributional reinforcement learning ( Bellemare et al. , 2017 ) to disentangle the various effects the discount factor has on the learning process . Scenario 2 : The true objective function is discounted ( γ < 1 ) . Theoretically , there is a γt term for the actor update on a transition observed at time t in a trajectory ( Sutton et al. , 2000 ; Schulman 1Blackwell optimality states that , in finite MDPs , there exists a γ0 < 1 such that for all γ ≥ γ0 , the optimal policies for the γ-discounted objective are the same . et al. , 2015a ) . Practitioners , however , usually ignore this term while using a discounted critic , i.e. , γA = 1 and γC = γ < 1 are used . We explain this mismatch with the following hypothesis : Hypothesis 2 . Using γC = γ < 1 and γA = 1 is effectively similar to using γC = γA = γ < 1 plus an auxiliary loss that sometimes facilitates representation learning . Our empirical study involves implementing the auxiliary task explicitly by using an additional policy for optimizing the difference term between the loss of γA = 1 and the loss of γA < 1 . We also design new benchmarking environments where the sign of the reward function is flipped after a certain time step such that later transitions differ from earlier ones . In that setting , γA = 1 becomes harmful . 2 BACKGROUND Markov Decision Processes : We consider an infinite horizon MDP with a finite state space S , a finite action spaceA , a bounded reward function r : S → R , a transition kernel p : S × S × A → [ 0 , 1 ] , an initial state distribution µ0 , and a discount factor γ ∈ [ 0 , 1 ] .2 The initial state S0 is sampled from µ0 . At time step t , an agent in state St takes action At ∼ π ( ·|St ) , where π : A×S → [ 0 , 1 ] is the policy it follows . The agent then gets a reward Rt+1 . = r ( St ) and proceeds to the next state St+1 ∼ p ( ·|St , At ) . The return of the policy π at time step t is defined as Gt . = ∑∞ i=1 γ i−1Rt+i , which allows us to define the state value function vγπ ( S ) . = E [ Gt|St = s ] and the state-action value function qγπ ( s , a ) . = E [ Gt|St = s , At = a ] . We consider episodic tasks where we assume there is an absorbing state s∞ ∈ S such that r ( s∞ ) = 0 and p ( s∞|s∞ , a ) = 1 holds for any a ∈ A . When γ < 1 , vγπ and qγπ are always well defined . When γ = 1 , to ensure v γ π and q γ π are well defined , we further assume finite expected episode length . Let Tπs be a random variable denoting the first time step that an agent hits s∞ when following π given S0 = s. We assume Tmax . = supπ∈Π maxs E [ Tπs ] < ∞ , where π is parameterized by θ and Π is the corresponding function class . Similar assumptions are also used in stochastic shortest path problems ( e.g. , Section 2.2 of Bertsekas & Tsitsiklis ( 1996 ) ) . In our experiments , all the environments have a hard time limit of 1000 , i.e. , Tmax = 1000 . This is standard practice , classic RL environments also have an upper limit on their episode lengths ( e.g . 27k in Bellemare et al . ( 2013 , ALE ) ) . Following Pardo et al . ( 2018 ) , we add the ( normalized ) time step t in the state to keep the environment Markovian . We measure the performance of a policy π with Jγ ( π ) . = ES0∼µ0 [ vγπ ( S0 ) ] . Vanilla Policy Gradient : Sutton et al . ( 2000 ) compute∇θJγ ( π ) as ∇θJγ ( π ) . = ∑ s d γ π ( s ) ∑ a q γ π ( s , a ) ∇θπ ( a|s ) , ( 1 ) where dγπ ( s ) . = ∑∞ t=0 γ t Pr ( St = s|µ0 , p , π ) for γ < 1 and dγπ ( s ) . = E [ ∑TπS0 t=0 Pr ( St = s|S0 , p , π ) ] for γ = 1.3 Note dγπ remains well-defined for γ = 1 when Tmax < ∞ . In order to optimize the policy performance Jγ ( π ) , one can follow ( 1 ) and , at time step t , update θt as θt+1 ← θt + αγtAqγCπ ( St , At ) ∇θ log π ( At|St ) , ( 2 ) where α is a learning rate . If we replace qγCπ with a learned value function , the update rule ( 2 ) becomes an actor-critic algorithm , where the actor refers to π and the critic refers to the learned approximation of qγCπ . In practice , an estimate for v γC π instead of q γC π is usually learned . Theoretically , we should have γA = γC = γ . Practitioners , however , usually ignore the γtA term in ( 2 ) , and use γC < γA = 1 . What this update truly optimizes remains an open problem ( Nota & Thomas , 2020 ) . TRPO and PPO : To improve the stability of actor-critic algorithms , Schulman et al . ( 2015a ) propose Trust Region Policy Optimization ( TRPO ) , based on the performance improvement lemma : Lemma 1 . ( Theorem 1 in Schulman et al . ( 2015a ) ) For γ < 1 and any two policies π and π′ , Jγ ( π ′ ) ≥ Jγ ( π ) + ( ∑ s d γ π ( s ) ∑ a π ′ ( a|s ) Advγπ ( s , a ) ) − 4 maxs , a |Adv γ π ( s , a ) |γ ( π , π ′ ) ( 1−γ ) 2 , 2Following Schulman et al . ( 2015a ) , we consider r : S → R instead of r : S ×A → R for simplicity . 3Sutton et al . ( 2000 ) do not explicitly define dγπ when γ = 1 , which , however , can be easily deduced from Chapter 13.2 in Sutton & Barto ( 2018 ) . where Advγπ ( s , a ) . = Es′∼p ( ·|s , a ) [ r ( s ) + γvγπ ( s′ ) − vγπ ( s ) ] is the advantage , ( π , π′ ) . = maxsDKL ( π ( ·|s ) ||π′ ( ·|s ) ) , and DKL refers to the KL divergence . To facilitate our empirical study , we first make a theoretical contribution by extending Lemma 1 to the undiscounted setting . We have the following lemma : Lemma 2 . Assuming Tmax < ∞ , for γ = 1 and any two policies π and π′ , Jγ ( π ′ ) ≥ Jγ ( π ) + ( ∑ s d γ π ( s ) ∑ a π ′ ( a|s ) Advγπ ( s , a ) ) − 4 maxs , a |Advγπ ( s , a ) |T 2max ( π , π′ ) . The proof of Lemma 2 is provided in the appendix . A practical implementation of Lemmas 1 and 2 is to compute a new policy θ via gradient ascent on the clipped objective : L ( θ ) . = ∑∞ t=0 γ t A min { πθ ( At|St ) πθold ( At|St ) AdvγCπθold ( St , At ) , clip ( πθ ( At|St ) πθold ( At|St ) ) AdvγCπθold ( St , At ) } , ( 3 ) where St and At are sampled from πθold , and clip ( x ) . = max ( min ( x , 1 + ) , 1 − ) with a hyperparameter . Theoretically , we should have γA = γC , but practical algorithms like Proximal Policy Optimization ( Schulman et al. , 2017 , PPO ) usually use γC < γA = 1 . Policy Evaluation : We now introduce several policy evaluation techniques we use in our empirical study . Let v̂ be our estimate of vγπ . At time step t , Temporal Difference learning ( TD , Sutton ( 1988 ) ) updates v̂ as v̂ ( St ) ← v̂ ( St ) +α ( Rt+1 + γv̂ ( St+1 ) − v̂ ( St ) ) . Instead of the infinite horizon discounted returnGt , De Asis et al . ( 2019 ) propose to consider theH-step returnGHt . = ∑H i=1Rt+i . Correspondingly , the H-step value function is defined as vHπ ( s ) . = E [ GHt |St = s ] . We let v̂H be our estimate of vHπ . At time step t , De Asis et al . ( 2019 ) use the following update rule to learn v̂ H : v̂i ( St ) ← v̂i ( St ) + α ( Rt+1 + v̂i−1 ( St+1 ) − v̂i ( St ) ) ( i = 1 , . . . H ) , ( 4 ) where v̂0 ( s ) .= 0 . In other words , to learn v̂H , we need to learn { v̂i } i=1 , ... , H simultaneously . De Asis et al . ( 2019 ) call ( 4 ) Fixed Horizon Temporal Difference learning ( FHTD ) . As Gt is a random variable , Bellemare et al . ( 2017 ) propose to learn its full distribution instead of its expectation only , yielding the Distributional Reinforcement Learning ( RL ) paradigm . They use a categorical distribution with 51 atoms uniformly distributed in [ −Vmax , Vmax ] to approximate the distribution of Gt , where Vmax is a hyperparameter . In this paper , we refer to the corresponding policy evaluation algorithm as C51 . Methodology : We consider MuJoCo robot simulation tasks from OpenAI gym ( Brockman et al. , 2016 ) as our benchmark . Given its popularity in understanding deep RL algorithms ( Henderson et al. , 2017 ; Ilyas et al. , 2018 ; Engstrom et al. , 2019 ; Andrychowicz et al. , 2020 ) and designing new deep RL algorithms ( Fujimoto et al. , 2018 ; Haarnoja et al. , 2018 ) , we believe our empirical results are relevant to most practitioners . We choose PPO , a simple yet effective and widely used algorithm , as the representative actor-critic algorithm for our empirical study . PPO is usually equipped with generalized advantage estimation ( Schulman et al. , 2015b , GAE ) , which has a tunable hyperparameter γ̂ . The roles of γ and γ̂ are similar . To reduce its confounding effect , we do not use GAE in our experiments , i.e. , the advantage estimation for our actor is simply the TD errorRt+1 +γCv̂ ( St+1 ) − v̂ ( St ) . The PPO pseudocode we follow is provided in Alg . 1 in the appendix and we refer to it as the default PPO implementation . We use the standard architecture and optimizer across all tasks , in particular , the actor and the critic do not share layers . We conduct a thorough learning rate search in Ant for each algorithmic configuration ( i.e. , a curve in a figure ) and then use the same learning rate for all other tasks . When using FHTD and C51 , we also include H and Vmax in the grid search . All details are provided in the appendix . We report the average episode return of the ten most recent episodes against the number of interactions with the environment . Curves are averages over ten independent runs with shaded regions indicating standard errors .

Summary: In this paper, the authors focus on the discounting mismatch in the Actor-Critic algorithm. From comprehensive experiments, the authors claim that this mismatch is either a bias-variance representation tradeoff or an auxiliary task for the actor update. Since the discounting mismatch problem is a well-known gap between the theoretical analysis and the application, their work, especially the experiments, might have some impact on how to understand this gap.