A Deeper Look at Discounting Mismatch in Actor-Critic Algorithms
Authors: Shangtong Zhang
Romain Laroche
Harm van Seijen
Shimon Whiteson
Remi Tachet des Combes
Venue:  (2021)

Abstract: We investigate the discounting mismatch in actor-critic algorithm implementations from a representation learning perspective. Theoretically, actor-critic algorithms usually have discounting for both actor and critic, i.e., there is a $\gamma^t$ term in the actor update for the transition observed at time $t$ in a trajectory and the critic is a discounted value function. Practitioners, however, usually ignore the discounting ($\gamma^t$) for the actor while using a discounted critic. We investigate this mismatch in two scenarios. In the first scenario, we consider optimizing an undiscounted objective $(\gamma = 1)$ where $\gamma^t$ disappears naturally $(1^t = 1)$. We then propose to interpret the discounting in critic in terms of a bias-variance-representation trade-off and provide supporting empirical results. In the second scenario, we consider optimizing a discounted objective ($\gamma < 1$) and propose to interpret the omission of the discounting in the actor update from an auxiliary task perspective and provide supporting empirical results.

Under review
A DEEPER LOOK AT DISCOUNTING MISMATCH IN ACTOR-CRITIC ALGORITHMS
ABSTRACT
We investigate the discounting mismatch in actor-critic algorithm implementa- tions from representation learning perspective. Theoretically, actor-critic algo- rithms usually have discounting for both actor and critic, i.e., there is a 2 term in the actor update for the transition observed time in a trajectory and the critic is a discounted value function. Practitioners, however, usually ignore the discount- ing (ty") for the actor while using a discounted critic. We investigate this mismatch in two scenarios. In the first scenario, we consider optimizing an undiscounted objective (7 1) where 7 disappears naturally (1' 1). We then propose interpret the discounting in critic in terms of a www.vaaaeeeeeeeeeeeeeeeee
trade- off and provide supporting empirical results. In the second scenario, we consider optimizing discounted objective (y < 1) and propose to interpret the omission of the discounting in the actor update from an auxiliary task perspective and provide supporting empirical results.
INTRODUCTION
Actor-critic algorithms have enjoyed great success both theoretically (Williams) 1992 Sutton et al. 2000 Konda, 2002, Schulman et al. 2015a) and empirically (Mnih al. 20166] Silver et al. 2016 Schulman et al. OpenAA 2018). There is, however, longstanding gap between the theory behind actor-critic algorithms and how practitioners implement them. Let 7, 7a. and %c be the discount factors for defining the objective, updating the actor, and updating the critic respectively.
Theoretically, no matter whether 7 = or 7 < 1, we should always use 7/4 = Yo = (Sutton
al. 2000 Schulman et al. (2155a) or at least keep 7 = Tic if Blackwell optimality (Veinott 1969. Weitzman 2001) is considered. Practitioners, however, usually use % 1 and Tc < in their implementations (Dhariwal et al. [2017] Caspi et al. 2017 Zhang 2018 Kostrikov 2018 Achiam 2018 Liang et al. 20188] Stooke & Abbeel 2019). Although this mismatch and its theoretical
disadvantage have been recognized by Thomas (2014); Nota & Thomas (2020), whether and why it yields benefits in practice has not been systematically studied. In this paper, we empirically investigate this mismatch from a representation learning perspective. We consider two scenarios separately.
Scenario 1: The true objective is undiscounted (7 1). The theory prescribes to 7 = % 1. Practitioners, however, usually use = = 1 but % < 1, introducing bias. We explain this mismatch with the following hypothesis:
Hypothesis 1. To < optimizes a biaa--urrunnee-eppeesntttiii trade-off.
It is easy to that To < 1 reduces the variance in bootstrapping targets. Besides this, we further provide empirical evidence showing that when "/c < 1, it may become easier to find a good repre- sentation compared % 1. Consequently, although using "cc introduces bias, it can facilitate representation lcarning. For our empirical study, we make use of recently introduced techniques, such fixed horizon temporal different learning (De Asis et al., 2019) and distributional reinforce- ment learning (Bellemare et al. (017) to disentangle the various effects the discount factor has on the learning process.
Scenario 2: The true objective function is discounted (7 < 1). Theoretically, there is 7 term for the actor update on transition observed time in trajectory (Sutton al. 2000 Schulman
BBlacwwell optimality states that, in finite MDPs, there exists %) 1 such that for all > 7o. the optimal policies for the 7-discounted objective the

---

Under review
et al. 2015a). Practitioners, however, usually ignore this term while using discounted critic, i.e., 7/4 and % = Y < 1 are used. We explain this mismatch with the following hypothesis:
Hypothesis 2. Using "cc =y < and aa is effectively similar to using "oc 7/a = Y < plus an auxiliary loss that sometimes facilitates representation learning.
Our empirical study involves implementing the auxiliary task explicitly by using an additional policy for optimizing the difference term between the loss of 7 = 1 and the loss of % < 1. We also design new benchmarking environments where the sign of the reward function is flipped after a certain time
step such that later transitions differ from carlier ones. In that setting, 7 1 becomes harmful.
2 BACKGROUND
Markov Decision Processes: We consider an infinite horizon MDP with a finite state space S, finite action space A, a bounded reward
Y
define the objective
function r S + I, transition kernel p S x S x A + 10, 1]. an initial state distribution Mu, and a discount factor 7 E [o, 1 The initial state So is sampled from μgg At time step t. agent in state S, takes action A, ~ T-1[S)), where π : A S + [0. 1] is the policy follows. The agent then gets a reward Re+1 = r(S.) and proceeds to the next state Si+1 ~ p(. The return of the policy π at time step defined as G2- define the state value function (S) = E[G,|S, = s| and the state-action value function (s, a) = E[G,|S, = 8, A, a). We consider episodic tasks where we assume there an absorbing state 800 E S such that r(s ) = O and p a) = 1 holds for any a E A.. When 7 < 1, e and Q are always well defined. When 7 = 1, to ensure oi and o7 are well defined, we further assume finite expected episode length. Let T be random variable denoting the first time step that an agent hits xx when following π given S. = s. We assume Tmax = sup, EII max, E < 0o, where π is parameterized by 8 and II is the corresponding function class. Similar assumptions also used in stochastic shortest path problems (e-g., Section 2.2 of Bertsekas & Tsitsiklis (1966). In our experiments, all the environments have hard time limit of T000, i.e., max 1000. This is standard practice, classic RL environments also have an upper limit on their episode lengths (e.g. 27k in [Bellemare et al.) (2013] ALE)). Following Pardo et al. (2018), we add the (normalized) time step ł in the state to keep the environment Markovian. We measure the performance of a policy π with (T) = EsprHo (So)].
update the actor
Tc update the critic
Table 1: Roles of the differ- ent discount factors
Zi=l - which allows us to
Vanilla Policy Gradient: Sutton et al.
2000 compute (T) as Vody = Σ. (s) Ea TH (eT(a)s)
(1)
where di - Ei Pr(S, = s|m0, π) for 7 < 1 and (3 = EI So Pr(S; = s|Su, u. p, +)]
for y = Note df remains well-defined for y = 1 when Tmax < oc. In order to optimize the policy performance (T), one can follow (1) and, at time step t, update 0₂ as
Oct1 + 0, + 42 (St, A.) log T (Ar)S..)
where a is a learning rate. we replace o² with a learned value function, the update rule (2)
becomes actor-critic algorithm, where the actor refers to and the critic refers the learned approximation of 4Fth In practice, an estimate for enc instead of q3 is usually learned. Theoretically, we should have 7 To = y. Practitioners, however, usually ignore the A term in ( and use
Yc < <% = 1. What this update truly optimizes remains an open problem (Nota & Thomas 2020).
TRPO and PPO: To improve the stability of actor-critic algorithms, Schulman et al. (2015a) pro- pose Trust Region Policy Optimization (TRPO), based on the performance improvement lemma:
Lemma 1. (Theorem) I in Schulman et al. 2015a)) For < and any two policies म and n²,
J,(T') 2 (T) + (E. (E, (abs)))d(s
4max, [Ady (s.))(aa))))(( (1-7)²
2 Following Schulman et al. 2015a), we consider r : S + instead of S x A + R for simplicity. Sutton et al. (2000) do not explicitly define df when 7 1, which, however, can be easily deduced from Chapter 13.2 in Sutton Barto 2018).
2

---

Under review
= where A = Ex-plh s. s. (r(s) + (00((s") - [s) is the advantage, <(T, (') max, Dxr ( (11-))))(/ and DRL. refers to the Kl. divergence.
To facilitate our empirical study, we first make theoretical contribution by extending Lemma I to the undiscounted setting, We have the following lemma:
Lemma 2. Assuming Tax < oc, for and any two policies and n²,
(T()) (T) d2
(ssls))dd(ss.a)) ( max [Adv (s, a) T) f T').
T
The proof of Lemma 2 provided in the appendix. A practical implementation of Lemmas I andd] is to compute new policy 0 via gradient ascent on the clipped objective:
L(0) Ei min
=a(A,1)) Te [S.)
Ad do
(Sr, At), (s.A Te (Ae 3,T
Adv (Sr, At) Roou
[S.] Fosu
where St and A₁ are sampled from Th and c1ip(x)) = max(min(r,) 1 c), 1 c) with hyper- parameter. Theoretically, we should have TA = "cc but practical algorithms like Proximal Policy Optimization (Schulman et al. 2017, PPO) usually use T/c < 7/ 1.
Policy Evaluation: We now introduce several policy evaluation techniques we use in our empirical study. Let è be our estimate of tz.- At time step t, Temporal Difference learning (TD, Suttonn (1888) ) updates é as i(S,) t ((S,) aa + ((S)). Instead of the infinite horizon discounted return G., De Asis al. 2019) propose consider the H-step return G = Σx= Reti- Correspondingly, the H -step value function is defined as oz (s) = EE [S, s). We let 8 be our estimate of uz At time step t. De Asis et al. (2019) use the following update rule to learn 6.
('().) t (S.) + a(Re+1 + 8' (S (S.))
(i (i= H).
(4)
where (s) = O. In other words, learn H we need learn {0r (0')i=.....11 simultaneously. De Asis et al. (2019) call (4) Fixed Horizon Temporal Difference learning (FHTD).
As G, is random variable, Bellemare et al. (211)) propose to learn its full distribution instead of its expectation only, yielding the Distributional Reinforcement Learning (RL.) paradigm. They use categorical distribution with 51 atoms uniformly distributed in [- (-Vmax: Vimax] approximate the distribution of G2, where Vmax is hyperparameter. In this paper, we refer to the corresponding policy evaluation algorithm as C51.
Methodology: We consider MuJoCo robot simulation tasks from OpenAl gym (Brockman et al., 2016 as our benchmark. Given its popularity in understanding deep RL algorithms (Henderson et al. 2017 [lyas et al., 2018 EEgstrom et al. [019 (Andrychowicz et al. (020)) and designing new deep RL algorithms (Fujimoto et al. 2018 Haarnoja et al.) 2018), we believe our empirical results are relevant to most practitioners.
We choose PPO, simple yet effective and widely used algorithm, as the representative actor-critic algorithm for our empirical study. PPO is usually equipped with generalized advantage estima- tion (Schulman et al. 20155 GAE), which has a tunable hyperparameter ý. The roles of y and f are similar. To reduce its confounding effect, we do not use GAE in our experiments, i.e., the advantage estimation for our actor is simply the TD eтror Re+1 ( - o(S.). The PPO pseudocode we follow is provided in Alg. Tin in the appendix and we refer to it as the default PPO implementation.
We use the standard architecture and optimizer across all tasks, in particular, the actor and the critic do not share layers. We conduct thorough learning rate search in Ant for each algorithmic configuration (i.e., a curve a figure) and then use the same learning rate for all other tasks. When using FHTD and C51, also include H and Vmax in the grid search. All details are provided in the appendix. We report the average episode return of the ten most recent episodes against the number of interactions with the environment. Curves are averages over ten independent runs with shaded regions indicating standard errors.
3 OPTIMIZING THE UNDISCOUNTED OBJECTIVE (SCENARIO I
When our goal is to optimize the undiscounted objective (T), one theoretically grounded option to 7A %c 7 = 1. By using 74 1 and % 1, practitioners introduce bias. We first
3

---

Under review
empirically confirm that introducing bias in this way indeed has empirical advantages. A simple first hypothesis is that "c < leads to lower variance in Monte Carlo return bootstrapping targets than Yc = 1, it thus optimizes bias variincie trade-off. However, we further show that there empirical advantages from " < that cannot uniquely be explained by this bias-variance trade-off, indicating that there are additional factors beyond variance. We then show empirical evidence identifying representation learning as an additional factor, leading to the bias-varraaccc-eppreenttttii trade-off from Hypothesis
All the experiments this section use 7/A
2000
Hafthheethh
Via ker
2000
129093
1:00
yees
1000
1000
03033
500
1000
3305
500
63323
PPG
39
Jaac
Jaa23
Figure 1: The default PPO implementation with different discount factors.
1250
2003
2360
1000
1000
5c0c0
3305
500
9:0-
100
500
JM
3M
Figure 2: Comparison between PPO and PPO-TD when Yo 1.
We be24
Pospar
Humundls
Se0c0
Ba BameretSaanee
1500
23:5
2000
50000
1.00
3000
19-0-
500
$00
mort 5.00
40000
chipe
stepe
ctips
Ctope
ctpp
Figure 3: PPO-TD with different discount factors.
Bias Varvanae trade-off: To investigate the advantages of using Yo < 1, we first test default PPO with Yc E {0.95, 0.97, 0.99, 0.995, 1}. We find that the best discount factor is always with Yo < and that ?c 1 usually leads to a performance drop (Figure D. In default PPO, although the advantage is computed as the one-step TD error, the update target for updating the critic ((S)) is almost always Monte Carlo return. As there is no 7 term in the actor update, we should theoretically use % TA 1 when computing the Monte Carlo return, which usually leads to high variance. Consequently, a simple hypothesis for the empirical advantages of using "o < is a bias-variance trade-off. We find, however, that there is more at play.
Beyond bias-variance trade-off: To reduce the effect of ?e in controlling the variance, we bench- mark PPO-TD (Algorithm 2 in the appendix). PPO-TD is the same default PPO except that the critic is updated with one-step TD, i.e., the update target for i(S)) is now Re+1 + 7c0(S++1). Although Figure 2 shows that PPO-TD (?c 1) outperforms PPO (%e 1) by large margin, indicating bias-variance may be at play, Figure Suggestt that for PPO-TD as well, % < is still benchmark PPO-TD-Ex (Algorithm the appendix), in which we provide N extra transitions to the critic by sampling multiple actions at any single state and using an averaged bootstrapping target. The update target for i(St) in PPO-TD- Ex is N+1 ΣND, Ri + (e)(S) Here Rº41 and S refer to the original reward and successor state. To get R;41 and S/+1 for i E {1,..., N). we first sample an action A from the sampling policy, then reset the environment to S,, and finally execute A} to get Ri41 and S Importantly, we do not count those extra transitions in the r-axis when plotting. The advantage for the actor update in PPO-TD-E is estimated with R + 6(5% o(.)) regardless of yo to further control the influence of variance. The critie is not trained on the extra successor states (Stt1iis1...
preferable 7c 1. To further study this phenomenon,
4

---

Under review
We karddd
Humonadi
2000
1500
20:00
2003
1E000
15:30
2003
c0000
1000
1000
to su
1030
530
2k
34
2V
Figure 4: PPO-TD-EX (Yc = 0.99).
kerdaa
Numores
2250
2003
1000
2300
1907
750
600
500
530
<5000
rir da
7M
24
70
Figure 5: PPO-TD-Ex (7c = 1).
So the quality of the prediction 1( depends mainly on the generalization of è. Intuitively, if i generalizes well, providing proper amount of transitions this way should improve or maintain the overall performance as they help reduce variance. As shown by Figure PPO-TD-Ex (Te 0.99) roughly follows this intuition. However, surprisingly, providing extra data to PPO-TD-Ex (% 1) leads to a significant performance drop (Figure This drop suggests that the larger variance from the randomness of S1+1 is not the only issue when using To 1 to train the critic. The quality of the estimate é, at least in terms of making prediction on untrained states { {S++1h.....N is lower when Tc 1 is used than Yo < 1. In other words, the generalization of è is poor when Yr 1. The curves for PPO-TD-Ex (% = 0.995) are a mixture of %c- = 0.99 and % = and are provided in Figure [6 in the appendix.
In the undiscounted setting, we should theoretically have R4+1 + ((S441) as the update target for the critic. When "ec < is used instead, the update target becomes Re41 (c0(S,41) and the variance resulting from the randomness of S.+1 becomes less pronounced. So here, "ee trades off bias with variance, similar to that in Monte Carlo return bootstrapping targets in default PPO. We refer to this effect of Te as variance control. However, "/c can also affect the difficulty of learning a good estimate for v':: we refer to this effect of Tc as learnability control (Lehnert et al., 2018 Laroche & van Seijen, 2018: Romoff et al., 2019). Inspired by the poor generalization of when
Yc = 1, we investigate learnability control mainly from the representation learning perspective. By representation learning, we refer to learning the bottom layers (backbone) of neural network. The last layer the neural network then interpreted as a linear function approximator whose features are the output of the backbone. This interpretation of representation learning is widely used in the RL community, see e.g. Jaderberg et al. (016)); Chung et al. (2018); Veeriah et al. (2019).
Bias- rearestpresnnntnoon trade-off: To separate variance control and learnability control, idcally should investigate the update target Rt-1 + (c,16(S,+1), where is trained to approximate v, and 7c,2 < Te.1 = 1. Learning an estimate ê for o 20.2 however, implies to use the update target Ri+1 + ((+4)() the two effects of "c,2 then get mixed again. To solve this dilemma, consider the update target Ri+1 + 81 11 (S where - is trained to approximate e i.e. we use FHTD to train the critic in PPO, which we refer to as PPO-FHTD (Algorithm | in the appendix). PPO-FHTD implements Yc.1 1 directly, and manipulating H changes the horizon of the policy evaluation problem, which is also one of the effects of manipulating Tc.2-
We test two parameterizations for PPO-FHTD to investigate representation learning. In the first parameterization, to learn !M we parameterize {v=] as H different heads over the same representation layer (backbone). as 1024 different heads over the same representation layer, whatever H we are interested in. To approximate H. we then simply use the output of the H-th head. A diagram (Figure 13 in the appendix further illustrates the difference between the two parameterizations.
the second parameterization, we always learn {o} h=...1.....1022
Figure 6 shows that by tuning H for FHTD, PPO-FHTD with the first parameterization matches or exceeds the performance of PPO-TD (Te < 1) in most tasks, and that the best H is always
5

---

Under review
HarChhecch
Hopper
Harraee
Humere
2300
2000
1903
2000
#9383
1000
3000
1000
03322
43333
Prp.1113 20-304
2M
2k
24
Figure 6: PPO-FHTD with the first parameterization. The best H and yo are used for each game.
-250
15:00
130 tecta
50000
500
500
45000
2k
24
3H
20
Figure 7: PPO-FHTD with the second parameterization.
smaller than 1024. Theoretically, as long as we use an H 2 Tiax 1000, we always have o (s) = (s). Figure 6 shows that the performance of PPO-FHTD (H = 1024) is very close to PPO-TD (nc = 1). indicating that learning {v=]j=i,..,,102 1023 is not an additional overhead for the network in terms of learning / =124, ii... increasing Hf does not pose additional challenges in terms of network capacity. However, Figure 7 suggests that for the second parameterization,
1024 is almost always among the best choices of H. Comparing Figures 6 and 7 conclude that in the tested domains, learning ! with different H requires different representations. This suggests that interpret the results in Figure 6 as bias-representation trade-off. Using larger H is less biased but representation learning may become harder due to the longer policy evaluation horizon. Consequently, an intermediate /l achieves the best performance in Figure As reducing Il cannot bring in advantages in representation learning under the second parameterization, the less biased H, i.e., the larger H, usually performs better Figure 7 Overall, Ye optimizes a bias- slas-representation trade-off by changing the policy evaluation horizon H.
We further conjecture that representation learning may be harder for a longer horizon because good representations can become rarer. We provide a simulated example to support this. Consider policy evaluation on the simple Markov Reward Process (MRP) from Fig- ure g We assume the reward for each transition is fixed and is randomly generated in [0. 1]. Let 2, E RKK be the feature vector for a state s; we set its i-th component as r = tanh(E), where & is random variable uniformly distributed in [-2, 22.. We chose this feature setup as we use tanh as the activation function in our PPO. We use X E R x to denote the feature matrix. To create state aliasing (McCallum) 1997). which is common under func- tion approximation, we first randomly split the N states into S and S2 such that |S1| = aN and |82] (1 N,,,, where a the proportion of states be aliased. Then for every s E St, we randomly select an å E S₁ and set F, < Zz. Finally, we add Gaussian noise (O, 0.1²) to each element of X. We use N 100 and K 30 in our simulation and report the normalized represen- tation error (NRE) as function of For a feature matrix X, the NRE is computed analytically as NRE(>) min. [xw-eell2 where v. is the analytically computed true value function of the MRP. |,,
Figure 8: A simple MRP.
We report the results in Figure 9 where each data point is averaged over 10t randomly generated feature matrices (X) and reward functions. In this MRP, the average representation error becomes larger as increases, which suggests that learning good representation under a large y and state aliasing may be harder than with a smaller y. We report the unnormalized representation error in Figure 17 in the appendix, where the trend is much clearer.
Overall, though we do not claim that there a monotonic relationship between the discount factor and the difficulty of representation learning, our empirical study does suggest that representation learning is a key factor at play in the misuse of the discounting in actor-critic algorithms, beyond
*The trend that NRF decreases as increases is merely an artifact from how we generate thy.

---

Under review
shand
537% eiored
Figure 9: Normalized representation error as function of the discount factor. Shaded regions indicate one standard derivation.
the widely recognized bias-variance trade-off. In the appendix, we provide additional experiments involving distributional RL to further support the bias-variance-represennaa
trade-off hypothe- sis, under the assumption that the benefits of distributional RI, comes mainly from the improved
representation learning (Bellemare et al. 2017 Munos 2018 Petroski Such et al. 2019).
4 OPTIMIZING THE DISCOUNTED OBJECTIVE (SCENARIO 2b
HUFCHodoh
Heaser
Prumsaned
Hamare Coterdip
P0
420
11.902
320
20302
100
100
330
xrops
staps
s7aps
1spp
mapp
Figure 10: Comparison between PPO and DisPPO with y = 0.995
When our goal is optimize the discounted objective (x), theoretically we should have the A term in the actor update and use < 1. Practitioners, however, usually ignore this A (i.e,, set 7/a = 1), introducing bias. Figure 10 shows that even if use the discounted return as the performance metric, the biased implementation of PPO still outperforms the theoretically grounded implementation DisPPO in the domains we tested. Here PPO refers to the default PPO implemen- tation where YA = 1. = < 1, and DisPPO (Alg. 6 in the appendix) adds the missing X term PPO by using 7 = 7c = 7 < 1. We propose to interpret the empirical advantages of PPO over DisPPO with Hypothesis 2 For all experiments in this section, we use "'c = <.
An auxiliary task perspective: The biased policy update implementation of (2) ignoring 7 can be decomposed into two parts as A, 7'A, + - >)) where A, = q2 (St, Ar) Vo T Ss.). We propose to interpret the difference term between the biased implementation (At) and the theoreti- cally grounded implementation (n² Ar) i.e., the (1->')g (Se, As) Vo log (A, S.) term, the gra- dient of an auxiliary objective with a dynamic weighting
7 Let Je. (T) En TTals)q] (s, a); we have VaJe,u(T))lu= = Ea-- 107 V log π (a|ss]. This objective changes every time step (through μ)). Inspired by the decomposition, AuxPPO (Algorithm 7 and Figure 13 in the appendix). In AuxPPO, we have two policies v and x' parameterized by 0 and 6' respectively. The two policies are two heads over the same neural network backbone, where ı is used for interaction with the environment and n is the policy for the auxiliary task. AuxPPO optimizes 0 and 0' simultaneously by considering the following joint loss
augment PPO with this auxiliary task, yielding
L(0.0') = -EE 7² min
(T(A. new (A))
Adv 7h Te..
(s,, clip( Tous (A,
ITe (A,
Adv (Sc. }+
(e ~²) 7²) min
(A, (9.) Ke (A.
A
(S,, Ar), clip(
Te (A,
Adv % Tead
Ai).
Ke S..
where St and A, are obtained by executing Pola. We additionally synchronize O' with @ periodically to avoid an off-policy learning issue.
Flipped rewards: Besides AuxPPO, we also design novel environments with flipped rewards to investigate Hypothesis 2 Recall we include the time step in the state, this allows us to simply create environment by defining
reward function r'(s,t) = r(s) r(s)I where I is the indicator function. During an episode, within the first to steps, this new environment is the same the original one. After to steps, the sign of the reward is flipped. We select tn such that
7

---

Under review
0.53
100
100
-20
26
24
20
33:35
1u/Chorchh yeo
naithastnn ya
Pratheetth y-
23
Audtro
200
docepted
-20
-32
-20
caps
cape
1500
12xpe
Figure 11: Curves without any marker are obtained in the original Ant Hal HalfCheetah. Diamond-marked curves are obtained in Ant / #Cheehhe with r'.
7 is sufficiently small, e.g., define to = min+t?² < 0.05). With this criterion for selecting to the later transitions (i.e., transitions after in steps) have little influence on the evaluation objective, the discounted return. Consequently, the later transitions affect the overall learning process mainly through representation learning. DisPPO rarely makes use of the later transitions due to the 7 term in the gradient update. AuxPPO makes use of the later transitions only through representation learning. PPO exploits the later transitions for representation learning and the later transitions also affect the control policy of PPO directly.
When we consider the original environments, Figure I| shows that in 8 out 12 tasks, PPO outperforms DisPPO, even if the performance metric is the discounted episodic return. In all those 8 tasks, by using the difference term as an auxiliary task, AuxPPO is able to improve upon DisPPO. In 6 out of those 8 tasks, AuxPPO able to roughly match the performance of PPO at the end of training. For 7 E {0.93, 0.9} in Ant, the improvement of AuxPPO is not clear and we conjecture that this because the learning of the r-head (the control head) in AuxPPO is much slower than the learning of in PPO due to the rá term. Overall, this suggests that the benefit of PPO over DisPPO comes mainly from representation learning.
Results:
When we consider the environments with flipped rewards, PPO is outperformed by DisPPO and AuxPPO by large margin in 11 out of 12 tasks. The transitions after to steps are not directly rele- vant when the performance metric is the discounted return. However, learning those transitions may still improve representation learning provided that those transitions are similar to the earlier transitions, which is the case in the original environments. PPO and AuxPPO, therefore, outperform DisPPO. However, when those transitions are much different from the earlier transitions, which is the case in the environments with flipped rewards, learning to control on them directly becomes dis- tracting. PPO, therefore, is outperformed by DisPPO. Different from PPO, AuxPPO does not learn to control on later transitions. Provided that the network has enough capacity, the control head my in AuxPPO will not be affected much by the irrelevant transitions. The performance of AuxPPO is, therefore, similar to DisPPO.
To summarize, Figure I1 suggests that using 7A 1 is simply inductive bias that all transitions are equally important. When this inductive bias is helpful for learning, 7A implicitly imple- ments auxiliary tasks thus improving representation learning and the overall performance. When this inductive bias is detrimental, however, 7A = can lead to significant performance drops. AuxPPO appears to be a safe choice that does not depend much the correctness of this inductive bias.
RELATED WORK
The mismatch in actor-critic algorithm implementations has been previously studied. Thomas 2014) focuses on the natural policy gradient setting and shows that the biased implementation ignoring 7 can be interpreted as the gradient of the average reward objective under a strong as- sumption that the state distribution independent of the policy. Nota & Thomas (020) prove that
"See Scction B.1 for more details about task selection.
8

---

Under review
without this strong assumption, the biased implementation is not the gradient of any stationary ob- jective. This does not contradict our auxiliary task perspective as our objective (T) changes at every time step. Nota & Thomas (2020) further provide counterexample showing that following the biased gradient can lead to a policy of poor performance w.r.t. both discounted and undis- counted objectives. Both Thomas (2014) and & Thomas (2020), however, focus theoretical disadvantages of the biased gradient and regard ignoring 7 as the source of the bias. We instead regard the introduction of "ic in the critic as the source of the bias in the undiscounted setting and investigate its empirical advantages, which are more relevant to practitioners. Moreover, our representation learning perspective for investigating this mismatch is to our knowledge novel.
trade-off, we do not claim that is all that affects. The discount factor also has many other effects (e.g.. [Sutton (1995); Jiang et al. (016));
Although we propose the weeevvaannnceepeeeeettne
Laroche et al. (2017); Laroche & van Seijen (0188); [eehner et al. (2018): Fedus et al. (019))
Van Seijen et al. (2019); Amit et al. (2020). which we leave for future work. Scenario 1, using
Tc < helps reduce the variance. Variance reduction in RL itself is an active research area (see, Papini et al. (2018): Xu al. (2019); Yuan et al. (020)). Investigating those variance reduction techniques with Tc 1 is a possibility for future work. Recently. [Bengio et al. (2020) study the effect of the bootstrapping parameter in TD(A) in generalization. Our work studies the effect of the discount factor y in representation learning in the context of the misuse of the discounting in actor-critic algorithms, sharing a similar spirit of [engii et al. (2020).
6 CONCLUSION
In this paper, we investigate the longstanding mismatch between theorists and practitioners in actor- critic algorithms from a representation learning perspective. Although the theoretical understanding of policy gradient algorithms have recently been significantly advanced (Agarwal et al., 2019; Wu cet al. 2020), this mismatch has drawn little attention. We hope our empirical study can help prac- titioners understand actor-eritic algorithms better and therefore design more efficient actor-critic algorithms in the setting of deep RL, where representation learning emerges as a major considera- tion. We hope empirical study can draw more attention to the mismatch, which could enable the community to finally close this longstanding gap.
REFERENCES
Joshua Achiam. Spinning up in deep reinforcement learning. 2018.
Alekh Agarwal, Sham M Kakade, Jason D) Lee, and Gaurav Mahajan.Optimalle Optimality and approximation with policy gradient methods in markov decision processes. arXiv preprint arXiv:1908 00261,, 2019.
Ron Amit, Ron Meir, and Kamil Ciosek. Discount factor as a regularizer in reinforcement learning. arXiv preprint htt::20000.00040 2020.
Marcin Andrychowicz, Anton Raichuk, Piotr Stanczyk, Manu Orsini, Sertan Girgin, Raphael Marinier, Léonard Hussenot, Matthieu Geist, Olivier Pictquin, Marcin Michalski, et al. What matters in on-policy reinforcement lcarning? a large-scale empirical study. arXiv preprint 2okYv:2006.059990 2020.
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling- The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, jun 2013.
Mare G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. arXiv preprint
2017.
Emmanuel Bengio, Joelle Pineau, and Doina Precup. Interference and generalization in temporal difference learning. arXiv preprint arXiv.2003.06350. 2020.
Dimitri P Bertsekas and John N Tsitsiklis. Heuro-Dynamit Programming. Athena Scientific Bel- mont, MA, 1996.

---

Under review
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016.
Itai Caspi, Gal Leibovich, Gal Novik, and Shadi Endrawis. Reinforcement learning coach, Decem- ber 2017. https::::///0. 10. 5281/ zww.oo.............
Wesley Chung, Somjit Nath, Ajin Joseph, and Martha White. Two-timescale networks for nonlinear value function approximation. In International Conference on Learning Representations, 2018.
Kristopher De Asis, Alan Chan, Silviu Pitis, Richard Sutton, and Daniel Graves. Fixed-horizon temporal difference methods for stable reinforcement learning. arXiv preprini arXiv:1909.0399 2019.
Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford,
John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines. https:
//github. com/ saseenai/baselee lins] 2017.
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep rl: A case study on ppo and trpo. In International Conference on Learning Representations, 2019.
William Fedus, Carles Gelada, Yoshua Bengio, Marc G Bellemare, and Hugo Larochelle. Hyper- bolic discounting and learning over multiple horizons. arXiv preprint wwXiv:1902.06665 2019.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. arXiv preprint 1802.007, 2018.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.01290, 2018.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. arXiv preprint (tXXv:/109.005000 2017.
Andrew Ilyas, Logan Engstrom, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. A closer look deep policy gradients. arXiv preprint arXiv:1811.02553 2018.
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z. Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint Sttv:1611.0539)) 2016.
Nan Jiang. Satinder P Singh, and Ambuj Tewari. On structural properties of mdps that bound loss due shallow planning. In IJCAI, 2016.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization, arXiv preprint 1rXiv:1412.6980 2014.
Vijay R Konda. Actor-critic algorithms. PhD thesis, Massachusetts Institute of Technology, 2002.
Ilya Kostrikov. Pytorch implementations of reinforcement learning algorithms. https://
github. httm///koot.ikko//yyorrc.c
ppo-s--pp-acck--gaa 2018.
Romain Laroche and Harm van Seijen. In reinforcement learning, all objective functions are not equal. 2018.
Romain Laroche, Mehdi Fatemi, Joshua Romoff, and Harm van Seijen. Multi-advisor reinforcement learning. arXiv preprint arXiv:1704.0075:: 2017.
Lucas Lehnert, Romain Laroche, and Harm van Seijen. On value function representation of long horizon problems. In AAAI Conference Artificial Intelligence, 2018.
Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg. Joseph Gon- zalez, Michael Jordan, and lon Stoica. Rllib: Abstractions for distributed reinforcement learning. In International Conference on Machine Learning, 2018
10

---

Under review
R McCallum. Reinforcement learning with selective perception and hidden state. PhD thesis, 1997.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. Proceedings of the 33rd International Conference on Machine Learning, 2016.
Rémi Munos.
Distributional reinforcement learning.
Invited talk at European Work-
shop on Reinforcement Learning https::/ewwl ftr.eell..lll....... files. wworcppeess.com/2000 http://ww....ieeee.oorrree...oo/ distributional rl. xdf 2018.
Chris Nota and Philip Thomas. Is the policy gradient a gradient? In Proceedings of the 19th International Conference on Autonomous Agents and Multiageni Systems, 2020.
OpenAL. Openai five.
2018.
Matteo Papini, Damiano Binaghi, Giuseppe Canonaco, Matteo Pirotta, and Marcello Restelli. Stochastic variance-reduced policy gradient. arXiv preprint arXiv:1806.0555 05668, 2018.
Fabio Pardo, Arash Tavakoli, Vitaly Levdik, and Petar Kormushev. Time limits in reinforcement learning. In International Conference Machine Learning, 2018.
Felipe Petroski Such, Vashisht Madhavan, Rosanne Liu, Rui Wang, Pablo Samuel Castro, Yulun Li, Jiale Zhi, Ludwig Schubert, Mare G. Bellemare, Jeff Clune, and et al. An atari model zoo for analyzing, visualizing, and comparing deep reinforcement learning agents. Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, Aug 2019. doi: 10. 24963/jjcai.2019/44
TRLphtp::// /xx doi. www.:oio..og...... 241.:/222433111.200000 2019/4455
Joshua Romoff, Peter Henderson, Ahmed Touati, Emma Brunskill, Joelle Pineau, and Yann Ollivier. Separating value functions across time-scales. arXiv preprint arXiv:1902.01180 2019.
John Schulman, Sergey Levine, Picter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of the 32nd International Conference Machine Learning, 2015a.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Picter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint Tokiv:/506.0243333 2015b.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 2017.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 2016.
Adam Stooke and Pieter Abbeel. ripyt: A research code base for deep reinforcement learning in pytorch. arXiv preprint arXiv: 19Xiv19009.00 5000, 2019.
Richard Sutton. Learning to predict by the methods of temporal differences. Machine Learning, 1988.
Richard Sutton. Td models: Modeling the world at mixture of time scales. Machine Learning Proceedings 1995. Elsevier, 1995.
Richard Sutton and Andrew G Barto. Reinforcement Learning: An Introduction (2nd Edition). MIT press, 2018.
Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Infor- mation Processing Systems, 2000.
Philip Thomas. Bias in natural actor-criti algorithms. Proceedings of the 31st International
Conference Machine Learning, 2014.
11

---

Under review
Hann Van Seijen, Mehdi Fatemi, and Arash Tavakoli. Using logarithmic mapping to enable lower discount factors in reinforcement learning. Advances in Neural Information Processing Systems, 2019.
Vivek Veeriah, Matteo Hessel, Zhongwen Xu, Janarthanan Rajendran, Richard L Lewis, Junhyuk Oh, Hado P van Hasselt, David Silver, and Satinder Singh. Discovery of useful questions as auxiliary tasks. Advances in Neural Information Processing Systems, 2019.
Arthur F Veinott.
Discrete dynamic programming with sensitive discount optimality criteria. The Annals of Mathematical Statistics, 1969.
Martin L Weitzman. Gamma discounting. American Economic Review, 2001.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning, Machine learning, 1992.
Yue Wu, Weitong Zhang, Pan Xu, and Quanquan Gu. A finite time analysis of two time-scale actor critic methods. arXiv preprint arXiv:2005.0/355 2020.
Pan Xu, Felicia Gao, and Quanquan Gu. Sample efficient policy gradient methods with recursive variance reduction. arXiv preprint arXi::1909.086/1 2019.
Huizhuo Yuan, Xiangru Lian, Ji Liu, and Yuren Zhou. Stochastic recursive momentum for policy gradient methods. arXiv preprint arXiv:2003.04302, 2020.
Shangtong Zhang, Modularized implementation of deep rl algorithms in pytorch. ht https:// g github. htspe/////aaaaagooogggg/////ggg/ 2018.
12

---

Under review
A PROOF OF LEMMA 2
Proof. The proof is based on Appendix B in Schulman et al. (0155)) where perturbation theory used to prove the performance improvement bound (Lemma D To simplify notation, vector and a function interchangeably, i.e., we also use and μ①0 to denote the reward vector and the initial distribution vector. (T) and dzr are shorthand for (T) and (s) with y All vectors are column vectors.
Let S+ be the set of states excluding 8² i.e,, S+ = S/(8))) that Pa (s. s') = Ea r(a]s)p(s'|s,d) a). Let G' = Lt= P. According to standard Markov chain theories, G(s, s') is the expected number of times that s' visited before R²0 hit given S. s. Thax < co implies that G is well-defined and we have G = (I P=)_1. Moreover, Tiax < co. also implies Vs, E.r G(s,s') S Tmax, i.e., ||G||> S maax. We have J(x) Mo Gr.
define Ps IR StIxis*1 such
Let G' = (1 (I- Px.) 11 we have
J(x) - J(T)) = (m) G)r.
Let A Pa. - Ps. we have
G'
Left multiply by G' and right multiply by G.
G G² = GGAAGG G' = G G'AG (Expanding G' in RHS recursively)
= GG+GAG GGAGG'GGAAGG
So have
J(T') - ((T) = 1Ā GAG PO GGAGGr
is easy to see G d= and Gr = Un- So
No GAGr = d77 AUT
= Σ (s) Σ Σ त {alls)p(s\ss.a) (s) a) Σ [x(ab))(((ls.a)) vo
Edn ds(s) Σ ( Σ p(s)(s,a)e7(3)
Σ (s) - (()) + Ep(dls,, - Ux vs()))
(z(
- T(ais))f(s) 0 holds for any f that dependes only on s)
Σ (s) Σ
C. (s..) = 0 by Bellman equation)
We now bound 14 G'AGGG.. First,
((Ar))(s)= ΣΣ -'aass)
- =(als) ((lls))
Σ (((sss) -(als) (r(s) [p(d)s - (x)))
Σ (a((s)) [(als)) (s,
< 2 max Drv ( (+19), (-1s)) sa
max (A,v
13

---

Under review
where Dyv is the total variation distance. So
||Gr||b> S max (I ((1s)) max Adv (s,
Moreover, for any vector
((a)(()) & 2 max Drv ( (+ =|lbo
|| max Drv (+1ss,
So
|| max (T
I10 G'AGAGr S |
47 max max (T (- max Adv
- 472 max max DKL (T max (s,
8,0
which completes the proof.
Note this perturbation- proof of Lemma holds only for S - R. S S A we can turn to the coupling-based proof as Schulman al. (2015a), which, however, complicates the presentation and deviates from the main purpose of this paper. We, therefore, leave it for future work.
B EXPERIMENT DETAILS
B. METHODOLOGY
We use HalfCheetais Walker, Hopper, Ant, Humanoid, and HumanoidStanduy as our benchmarks. We exclude other tasks find PPO plateaus quickly there. The tasks con- sider have a hard time limit of 1000. Following Pardo et al. 2018)) we add time step information into the state, i.e., there an additional scalar t/ 1/1000 in the observation vector. Following Achiam (2018). we estimate the KL divergence between the current policy 0 and the sampling policy Hold when optimizing the loss 3 When the estimated KL. divergence greater than a threshold, we stop updating the actor and update only the critic with current data. We use Adam (Kingma & Ba, 2014] as the optimizer and perform grid search for the initial learning rates of Adam optimizers. Let a and ac = B be the learning rates for the actor and critic respectively. For each algo- rithmic configuration (i.e., curve in a figure), we tune a E {0.115, 0.25, 0.5, 1, 2} x 3 10-4 and {1,3} with grid search in Ant with independent runs maximizing the average turn the last 100 training episodes. In particular, a = 3 10 and 3 = is roughly the default learning rates for the PPO implementation in Achiam (2018). We then this al- gorithmic configuration with the best 0 and ac in all tasks. Overall, we find after remov- ing GAE, smaller learning rates are preferred. When we use FHTD, we additionally consider H E {16,32, 64, 128, 256, 512, 1024] in the grid search. When we use C51, we additionally con- sider V Vmax {20, 40, 80, 160, 320, 640, 1280, 2560, 5120, 10240, 81920, 163840, 327680} in the grid search. We use PPO-TD with configuration in Ant transfers
0.99 as example to study how the best hyperparameter other games. As shown Figure T2 the best learning rates of Ant (o. = 3 10-1 and 3 = 3) yields reasonably good performance in all the other games except Humanoid d. In the paper, we do not draw conclusion from single task. So an outler is unlikely to affect the overall conclusion.
In the discounted setting, we consider only Ant, HalfCheetah and their variants. For Walker2d, dd Hopper, and Humanoid, we find the average episode length of all algorithms are smaller than ty. f.e., the flipped reward rarely takes effects. For Humano dS andup, the scale of the reward is too large. To summarize, other four environments not well-suited for the purpose of our empirical study. Moreover, in the discounted setting, we performed the grid search of the learning rates for both Ant and Hal fCheetah.
14

---

Under review
HalfCheetah
Walker2d
Hopper
[0.125.
2000
2500
undiscounted return
3000
(0.125. 3)
2000
1500
[0.25,
2000
[0.5, 1)
1500
[0.5, 3)
1000
1000
1000
500
1)
500
(2.3)
0
2M
2M
steps
steps
steps
Ant
Humanoid
HumanoidStandup
4000
100000
2000
3000
80000
2000
1000
60000
1000
40000
0
2M
0
2M
2M
steps
steps
steps
Figure 12: PPO-TD (7c = 0.99) with different learning rates. curve labeled with (ar, 9) cor- responds to an initial learning rate for the actor and critic of a& x x 3 10-1 and oe Bax respectively. The best learning rates for Ant (a,, 3 10-4 and = 3) yields reasonably good performance in all the other games except Humanoid.
B.2 ALGORITHM DETAILS
The pseudocode of all implemented algorithms are provide in Algorithms f with their architec- tures illustrated in Figure 13 For hyperparameters that are not included in the grid search, we use the same value as Dhariwal al. (2017); Achiam (2018). In particular, for the rollout length, we set K = 2048. For the optimization epochs, we set K opt 320. For the minibatch size, we set B 64. For the maximum KL divergence, we set Ltarget = 0.01. We clip me (a|s)
ss(als)
into [-0.2, 0.2). We
use N. 51 supports for PPO-C51.
We use two-hidden-layer neural networks for function approximation. Each hidden layer has 64 hidden units and tanh activation function. The output layer of the actor network has a tanh acti- vation function and is interpreted as the mean of an isotropic Gaussian distribution, whose standard derivation is global state intepenndennnnnt variable as suggested by Schulman et al. (2015a).
15

---

Under review
Algorithm 1: PPO
Input:
t parameters of न. 0
a,, act Initial learning rates of the Adam optimizers for 0, ab
K, Kopt, B: rollout length, number of optimization epochs, and minibatch size
K gtt maximum KL divergence threshold
So 140
while True do Initialize buffer M
a &- for 0, ....... K do
A; ~ Tome (-S.:)
Execute AG, get Ri+1, S
if S is a terminal state then
m. t 0, S po
else end
m. +
end
GK +- ((Sk)
for = K 1....,01 do
G. + R441 + G
Adv; R4+1 + je (S(+1) (8.)
Store (S,, Aj. Adv,) in M
end
Normalize Adv, in M as Adv; Adv, -mcan (Adv.)) sed(AAdv,)
for o= 1..... Kopt do Sample minibatch {(S,, Ar, Ad, )= from M
L(tb) + 28 EE (es - G2)² / No gradient through G,
L(0) - B
D minf (A(A,
Tootd (A; S Adv, clip( Tegia (A:15,) Adv,
se(A,
Perform one gradient update to v minimizing L(1) with Adam
if EB log To... (A,(S:) log (A,,,S, < K Ltarget then
Perform one gradient update to 0 maximizing L(0) with Adam
end
end end
16

---

Under review
Algorithm 2: PPO-TD
Input:
t parameters of म. 0
a,, act Initial learning rates of the Adam optimizers for 0, bb
K, Kopt, B: rollout length, number of optimization epochs, and minibatch size
K gtt maximum KL divergence threshold
So 140
while True do Initialize buffer M
a &- for 0, ....... K do
A; ~ Tome (-S::)
Execute AG, get Ri+1, S
if S is a terminal state then
m. t 0, S po
else end
m, +
end
for
i K - 1,...,,0 do
Adv; Rit1 ( (S.)
S + S441.1 t R1+1
Store (s,,A,, Mi, ri, Sy Adv;) in M
Ag,
end
Normalize Adv; in M as Adv; + Adv;-mean( (Adv.)
(dd()))v))
for o L..., Kopt do
Sample minibatch {S,, A,, T/, Sy Adv;) from M Ne Ti 0 (
m4
L(tb) + 28 (e (S:) - y:) / No gradient through (s(A,))))
L(0) + B DE minf Toote (A; S;) Adv;, Tegia (A,1S,) Adv,)
(s.A,
Perform one gradient update to v minimizing L(1) with Adam
if EB log To... (A,(S:) log (o(A, < K Ltarget then Perform one gradient update to 0 maximizing L(0) with Adam
end
end end
17

---

Under review
Algorithm 3: PPO-TD-Ex
Input:
0, di parameters of 6
O,A, act Initial learning rates of the Adam optimizers for 0, K. Kopt, B: rollout length, number of optimization epochs, and minibatch size Ltarget:: maximum KL divergence threshold
N: number of extra transitions
r: transition kernel and reward function of the oracle
So ~ while True do Initialize buffer M Oold < for i 0, .... K do
for j = 0,..., N do
A! ~ Taaa (1S:); R2+ + r(S., A3), St ~ ~p(-Ss.,
if 1% is terminal state then
m 0, 8/ Po
else
m -
end
end
S:-1 t 1+1
end for
i K 1,..., do
Adv; + R + rm f (S tgp
for j = O,..., N do
S/ + 8/
end
Store (1S/, A², mi r2. S/9 ) f=O...., N, Adv;) in M
end
Normalize Adv, in M as Adv; + Adv; -mean( (Adv.)) sad({Adv.))
for 1,... Kope do
Sample minibatch {(S)) AL, m r/, S.5 Jo= Adv:))==..... B from A -=
- N+1 Σx= of + tem! (S,))
L(b) - 2B (h (S y:)² /* No gradient through y:
L(0) + ΣE minf Tents (A S.)
(s()
Adv, clip( (A
Ne.sa (AQ1SP) Adv;;
Perform one gradient update to v minimizing L(1) with Adam
if #/ Σ4 log To.. SA)) log (A ( K Learget then
Perform one gradient update to 0 maximizing L(0) with Adam
end
end end
18

---

Under review
Algorithm 4: PPO-FHTD
Input:
do parameters of π, {60 j==l....
O.A, Oct Initial learning rates of the Adam optimizers for 0
K, Kopt, B: rollout length, number of optimization epochs, and minibatch size
K Kttorget maximum KL divergence threshold
So ~ 14 while True do Initialize buffer M
Oold <- f
for O,.... 1 do
A Toaia (-IS.) Execute Ai, get Ri+1, S if S,+1 is a terminal state then m, S1+1 Po
else end
m, 1-
end
for
i
K 1,...,0da do
Adv, t R4+1 + m (S(+1) (S,)
S. + S411,; t R141
Store (Si, Ai, Mi, ri, St Adv;) in M
end
Normalize Adv, in M as Adv; +- Adv.- (Adv, })
std([Adv.)
for o 1..... Kopt do Sample minibatch {(S; Ai, Ai, ri, S% Adv;)) from M for j = 1.... H do
y +- r; + - x (S)) =0
end
L(
2B IB (
/ gradient through of (s(A.15))
L(0) + 1/ Σ min (A(A. Adv;, clip( To (A:15,) Advv; Adv;)
To (4, 8,
Perform one gradient update to & minimizing L(w) with Adam
13 - log To... (As(S:) - log To (.) < K Tstaget then
Perform one gradient update to 0 maximizing L(0) with Adam
end
end end
19

---

Under review
Algorithm 5: PPO-C51
Input:
ti parameters of π, { j.=.,....,,, N, with N₂ being the number of supports and e being the probability of each support
O,A, act Initial learning rates of the Adam optimizers for 8, vb
K. Kop Kopt, B: rollout length, number of optimization epochs, and minibatch size
K Tatarget: maximum KI. divergence threshold
A. = 2Vmxx N. {2j = -max + +
1)A2 : = = N.) // Define the supports
So (4n
while True do
Initialize buffer M
Oold t
for O, ...., K 1
A; Toma (+1S))
Execute As, get Re+1, Si+1 if S1+1 is a terminal state then m. S141 po
else end
m, +1
end
for i = K - 1,...,... do
Adv, Ru+1 + m770 EØ=1 o (S.-1)zy En (S))
N.
S + S441 Ri+1
Store (S,, Ai, mi, ri, Sy Adv;) in M
end
Normalize Adv, in M as Adv;
Adv. Adve-mean( (omnan(((Av))) (Adv;) }
std( (Adva)
for o L...., Kopt do
Sample minibatch {(S, mi, ri, S% for i 1, B do
from M
for j 1.....1.. N₂ do
r, m2
end
end for
j = 1,... N₂ do
V - Nm, k==
[[=]]] Vear -2: 110 /+ [x]]] min(max(x, 1), u)
end
I(1)) < B D N of log (S:) / No gradient ent xo(A.)).)
ent through 5/
[(O) + B Σ min{ xs(A.)
Tontd (A; SJ Adv;, clip( Te (A:1S)) ( Adv;)
Perform one gradient update to v minimizing I((b) with Adam
if ΣB log Ne. (A,)S:) log (A. < K Learget then
Perform one gradient update to 0 maximizing L(0) with Adam
end
end
end
20

---

Under review
Algorithm 6: DisPPO
Input:
di parameters of n. 0
a,, act Initial learning rates of the Adam optimizers for 0, &b
K, Kopt, B: rollout length, number of optimization epochs, and minibatch size
K Ltarget: maximum KL divergence threshold
So Mo,t +-0 0
while True do Initialize buffer M a &- for 0, .,...,. K do
A; ~ To (1S:). +- Execute AG, get Ri+1, S if S is a terminal state then
m, x 0, S No, t +
else
m, t 1,
end
end
GK +- ((Sk)
for == K 1,...,00 do
G. + R441 + G+11
Adv; R4+1 + j (S(+1) (S.)
Store (S,, Aj. G,, Adv,, t;) in M
end
Normalize Adv, in M as Adv; Adv, -mcan (annaa((((dy)))) (Adv.)) std( (Adv,))
for o= 1..... Kopt do Sample minibatch {(S,, Ar, Ad,,, (.)d, from M (S:) - G2)² / No gradient through G, D AA minf (s,
L(tb) + 28 EE
L(0) - B
Toace (4,15)) (5)) Adv,, clip( Te (A,15) Adv,
=s(A,)))
Perform one gradient update to & minimizing L(b) with Adam
if EB log To... (A,(S:) log To (A,)S) < K LLtarget then
Perform one gradient update to 0 maximizing L(0) with Adam
end
end end
21

---

Under review
Algorithm 7: AuxPPO
Input:
0,0', gh: parameters of т. n
Q,, act Initial learning rates of the Adam optimizers for 8, yb
K. K Kopt, B: rollout length, number of optimization epochs, and minibatch size get maximum KI. divergence threshold
So ~ Hn, tn,te +-0 while True do
Initialize a buffer M Gold t 0 A
for i O,..., K do
A; ~ To (-S.)), + Execute A,, get S
S is a terminal state then
m, 1,,,, 0, S.+1 ~ po, t
else
m, 1+1
end
end
GK
0(Sx)
for i K - -1....,de
G. R441 + G
Adv, t RM+1 + 0s (8++1) - (S,) Store (S,, Ag, G., Adv,, t,) in M
end
Normalize Adv; in M as Adv; + Adv, Adcyn mmcan((Avv)) (Adv.)) (dd())
for o L.... opt do
Sample minibatch {(S,, A,, Ad,,,v, to t) from M
L( 2B (6 (S.) - G,)² /+ gradient through G₁
L(0,0) + 1/2 Σ Tc min (s(A,)))
Adv,, clip(
Mon (A.IS.) Adv.
(u(A,)))
no S.J
( Tea St)
min (A, 15))
Adv,, clip(
(4) Adv;
3e (A. St))
Perform one gradient update to & minimizing ((bb with Adam if D log Te... (A,[S,) - log (A,,SS. < Inarget then
Perform one gradient update to 0, 0' maximizing L(O,0' 0') with Adam
end
end
end
22

---

Under review
π
π
1
S
S
S
(a) Architecture of PPO, PPO- (b) The first parameterization of PPO-FHTD
TD, PPO-TD-EX, DisPPO
π
10%
(c) The second parameterization of PPO-FIIT
(d) Architecture of PPO-C51
π
π
S
S
(e) Architecture of AuxPPO
Figure 13: Architectures of the algorithms
C ADDITIONAL EXPERIMENTAL RESULTS
C.I DISTRIBUTIONAL RL
Hypothesis T and the previous empirical study suggest that representation learning may be the main bottleneck of PPO-TD (7e = 1). To further support this, we benchmark PPO-C51 (?c 1) (AI- gorithm 5 in the appendix). where the critic of PPO is trained with C51. C51 is usually considered to improve representation learning by implicitly providing auxiliary tasks (Bellemare et al.) 2017 Munos] 2018, Petroski Such et al., (019). Figure 14 shows that training the critic with C51 deed leads to performance improvement and PPO-C51 (Ye 1) sometimes outperforms PPO-TD (Te < 1) by a large margin. Figure further shows that when Vmaax is optimized for PPO-C51, the benefit for using Vc < in PPO-C5T is less pronounced than that in PPO-TD, indicating the role of Tc < and distributional learning may overlap. Figures & suggest that the overlapping is representation learning.
C.2 OTHER COMPLEMENTARY RESULTS
Figure 16 shows how PPO-TD-E, -x (?e = 0.995) reacts to the increase of N. Figure 17 shows the unnormalized representation error in the MRP experiment. Figure [88 shows the average episode length for the Ant environment in the discounted setting. For Hal fCheetah, it is always 1000.
D LARGER VERSION OF FIGURES
23

---

Under review
HallCheetah
Walker2d
Hopper
PPO-TD ==
undiscounted return
PPO-TD ye 0.99
2000
1500
PPO-C51 Vns. -81920
1500
1000
1000
1000
PPO-TD =1
500
500
PPO-TD y: ===2 1
PPO-TD Y: 0.99
PRO-TD y: = 0.995
PPO-C51 VTTw 1280
PPO-C51 Vma 320
0
2M
0
2M
2M
steps
steps
steps
Ant
Humanoid
HumanoidStandup
4000
2000
PPO-TD =
140000
PPO-TD y: 0.995
1500
3000
PPO-C51 Viaw 163840
120000
100000
PPO-TD Yt- 11
1000
2000
PPO-TD Y:
80000
PPO-C51 163840
500
PPO TD
1000
60000
PPO-TD 0.
PPO-CS1 Vin. 10240
40000
0
2M
0
2M
2M
steps
steps
steps
Figure 14: For PPO-C51, we % 1.
HalfCheetah
Walker2d
Hopper
2000
1500
undiscounted return
3000
1500
1000
2000
1000
1000
500
PPO C51 Y- -1
500
PPO-C51 ye 0.99
PPO-C51 Ye - 0.995
0
2M
0
2M
2M
steps
steps
steps
Ant
Humanoid
HumanoidStandup
4000
2000
3000
150000
1500
1000
2000
100000
500
1000
50000
0
2M
0
2M
2M
steps
steps
steps
Figure 15: For each game, Vmax is the same the Vmax in Figure 14
24

---

Under review
HalfCheetah
Walker2d
Hopper
2000
1500
undiscounted return
2000
1000
1000
1000
PPO-TD Ex N-D
500
PPO-TD-EX N-1
500
PPO-TD-EX N-2
PPO-TD-EX
2M
2M
2M
steps
steps
steps
Ant
Humanoid
HumanoidStandup
2000
3000
50000
1500
47500
2000
1000
45000
42500
500
1000
40000
37500
2M
2M
steps
steps
steps
Figure 16: PPO-TD-Ex (?e- 0.995).
sluad
HOME
20/Ph
Unnormalized representation emror (RE) as function of the discount factor. Shaded regions indicate one standard derivation. RE is computed analytically as RE(X, 7) = min X ny
Figure 17:
25

---

Under review
Anl y 0.995
Ant y 0.99
Anl y 0.97
1000
1000
1000
episode length
800
800
800
600
600
600
400
400
400
PPO
200
DisPPO
200
200
AuxPPO
2M
steps
steps
steps
Ant y 0.95
Ant 0.93
Ant y 0.9
1000
1000
1000
800
800
800
600
600
600
400
400
400
200
200
2M
0
2M
2M
steps
steps
steps
Figure 18: Curves without any marker are obtained the original Ant. Diamond markeed curves are obtained in Ant with r'.
HalfCheetah
Walker2d
Hopper
2000
2500
undiscounted return
1500
2000
2000
1500
1500
1000
PPO
1000
1000
500
PPO 0,995
PPO 0.99
500
500
PPO 0.97
PPO yo 0.95
0
2M
steps
steps
steps
Ant
Humanoid
HumanoidStandup
2000
1500
120000
1250
1500
100000
1000
1000
750
80000
500
500
60000
250
0.
40000
2M
0
2M
2M
steps
steps
steps
Figure 19:
The default PPO implementation with different discount factors. The larger version of
Figure
T
26

---

Under review
HalfCheetah
Walker2d
Hopper
PPO y: =1
1250
undiscounted return
V::
1500
1000
1000
750
500
500
500
250
2M
2M
2M
steps
steps
steps
Ant
Humanoid
HumanoidStandup
1500
2000
1000
1500
50000
1000
45000
500
500
40000
2M
2M
2M
steps
steps
steps
Figure 20: Comparison between PPO and PPO-TD when Yo 1. The larger version of Figure
HalfCheetah
Walker2d
Hopper
undiscounted return
1500
1500
2000
1000
1000
1000
PPO-TD y: =1
PPO-TD y: - 0.995
PPO TD v = 0.99
500
500
PPO-TD v. = 0.97
PPO-TD V: 0.95
2M
2M
2M
steps
steps
steps
Ant
Humanoid
HumanoidStandup
2500
3000
90000
2000
80000
1500
2000
70000
1000
60000
1000
500
50000
40000
steps
steps
steps
Figure 21: PPO-TD with different discount factors. The larger version of Figure 3
27

---

Under review
HalfCheetah
Walker2d
Hopper
3000
undiscounted return
1500
2000
1000
1000
1000
PPO TD Ex N=0
500
500
PPO-TD-EX N-1
PPO-TD-EX N-2
PPO-TD-EX N-4
2M
2M
2M
steps
steps
steps
Ant
Humanoid
HumanoidStandup
3000
70000
2000
1500
2000
60000
1000
50000
1000
500
40000
2M
2M
2M
steps
steps
steps
Figure 22: PPO-TD-Ex (tc 0.99). The larger version of Figure
HalfCheetah
Walker2d
Hopper
1500
1250
undiscounted return
1000
1000
1000
750
750
500
500
500
PPO-TD- N=
PPO TD Ex N=1
250
250
PRO/TD-ER Ex N=2
PPO-TD-EX N-4
2M
2M
2M
steps
steps
steps
Ant
Humanoid
HumanoidStandup
1500
2000
1000
1500
50000
1000
45000
500
500
40000
steps
steps
steps
Figure 23: PPO-TD-EX (te 1). The larger version of Figure
28

---

Under review
HalfCheetah
Walker2d
Hopper
2000
undiscounted return
2000
1500
1000
1000
1000
PPO-TD
PPO-TD y==1
PPO-TD Y: ==
PPO-TD 0.99
500
PPO-TD 0.99
500
PPO-TD V: 0.995
FHTD H-128
PPO FHTD H-512
PPO FHTD H-256
PPO-PHTD H-1024
0
PPO-FHTD H-1024
PPO-FHTD H-1024
2M
steps
steps
steps
Ant
Humanoid
HumanoidStandup
3000
3000
100000
2000
2000
80000
1000
Y:
1000
Y:
60000
PPO-TD Y:
RPO-TD ye
y:
PPO-TD y:
PPO-FHTD H-64
PPO-FHTD
PPO-FHTD H-64
PPO-FHTD H-1024
PPO-FHTD H-1021
40000
PPO-FHTD H-1024
2M
0
2M
2M
steps
steps
steps
Figure 24: PPO-FHTD with the first parameterization. The best H and Yo are used for each game. The larger version of Figure 6
HalfCheetah
Walker2d
Hopper
1250
undiscounted return
1000
1500
1000
PPO-FHTD H-16
500
#PO-FHTD H=32
750
1000
PPO-FHTD H=64
500
PPO-FHTD H-128
PPO-FHTD H=256
500
250
PPO-FHTD H=512
2M
2M
0
2M
steps
steps
steps
Ant
Humanoid
HumanoidStanduu
2500
1500
Mayny
55000
2000
1000
1500
50000
500
1000
45000
500
40000
steps
steps
steps
Figure 25: PPO-FHTD with the second parameterization. The larger version of Figure 7
29

---

Under review
HalfCheetah
Walker2d
Hopper
400
discounted retur
300
DisPPO
400
300
200
300
200
200
100
100
100
0
0
2M
2M
2M
steps
steps
steps
Ant
Humanoid
HumanoidStandup
11000
600
200
10000
400
9000
100
200
8000
0
2M
steps
steps
steps
Figure 26: Comparison between PPO and DisPPO with 7 0.995. The larger version of Figure 10
30

---

Under review
Ant y= 0.995
Ant y 0.99
Ant 0.97
discounted return
PPO
DisPPO
150
60
AuxPPO
200
100
20
50
100
0
-20
-50
0
2M
0
2M
2M
steps
steps
steps
Ant y 0.95
Ant y 0.93
Ant y y= 0.9
40
20
10
20
-10
-10
-20
-20
-20
0
2M
2M
2M
steps
steps
steps
HalfCheetah y= 0.995
HalfCheetah y= 0.99
HalfCheetah 0.97
300
150
40
discounted return
200
100
20
100
50
PPO
0-
DisPPO
0
AuxPPO
-20
0
2M
0
2M
2M
steps
steps
steps HalfCheetah y 0.9
HalfCheetah y= 0.95
HalfCheetah y= 0.93
20
10
10
0
-10
-10
-5
-20-
-10
-20
-30
0
2M
0
2M
steps
steps
steps
Figure 27: Curves without any marker obtained in the original Ant environment. Diamond- marked curves are obtained in Ant with r'. The larger version of Figure 11
31


Reviews:
[{"review_id": "dpFqXilr3b4", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall I like this direction since this is an important, open problem in RL that does not seem to be widely known (I was unaware of it until I looked into the related work) and could lead to improved algorithms. I encourage the authors to continue to pursue this line of research. However, I have a few clarifications and questions regarding the experiments which make it unclear how meaningful the results are. For now, I vote to reject this work but am willing to change my opinion based on the rebuttal.\n\nStrengths:\n- The paper investigates and draws further attention to an important open problem that does not seem to widely known. Based on my reading of Nota and Thomas, it appears most major papers in the field today do not acknowledge the discrepancy of the missing discount factor.\n- The paper includes many experiments especially in the Appendix each with a robust 10 seeds. I do have some issues with the experimental setup that I will detail later but I appreciate the variation in experiments.\n- I also think the representation learning experiments in Scenario 1 using FHTD are an interesting approach to study the effect of learnt representations.\n- The experimental setup and methods used are clearly described and it appears the code will be made available in the final version thereby potentially making the experiments highly reproducible.\n\nIssues/Points of clarification:\n- Most of the study is done in the setting where \\gamma=1 (Scenario 1 in the paper). This corresponds to the undiscounted objective where the current time index must be included in the state for correct estimation of the value function. However the setting that is most widely used in existing literature involves a discount factor<1. For instance, all of the methods cited in the Methodology section: Henderson et al., 2017; Ilyas et al., 2018; Engstrom et al., 2019; Andrychowicz et al., 2020, Fujimoto et al., 2018, Haarnoja et al., 2018 use a discount<1 (Andrychowicz et al. do not include a discount of 1 in their sweep over discount factors either).  This is dubbed Scenario 2 in the main text and includes only one experiment on the Ant task. It is fine to try to draw insights and focus on Scenario 1 as long as it is well motivated. However I do think it is misleading to claim ‘we believe our empirical results are relevant to most practitioners’ when most of the study does not involve a setting that is actually used by said practitioners. \n\n- My second concern is with the method used to choose hyperparameters for the experiments. In particular, the learning rate is chosen based on the ‘Ant’ experiment and then the best performing parameters are fixed and transferred to the others. While I appreciate the motivation behind this approach, I’m not certain how well these transfer to some tasks. In particular, the HumanoidStandup task seems to involve returns which are an order of magnitude greater than the other tasks. I think at least for this one task a small sweep is essential to be confident of the claims.\n\n- There are a few points in the paper where correlation seems to be misinterpreted as causation. For instance Figures 11-13 in the paper indicate that: a) a discounted critic (\\gamma_c<1) performs better on all tasks; b) biased updates using TD instead of empirical returns performs better on some tasks. These two statements alone are insufficient to claim that the advantage of a discounted critic (\\gamma_c=1) is therefore partly due to bias. Looking at Figures 11 and 13, I think a figure similar to Figure 12 comparing TD and empirical returns can be generated for any discount factor (e.g. \\gamma=0.995). Perhaps I am missing something here and if so clarification from the authors would be much appreciated!\n\n- These discrepancies combine in Figure 1 where for \\gamma_c=0.99, different values of extra transition samples (N) are plotted. Ostensibly, increasing N should reduce the variance even further. However quite a few of the curves choosing N=2 or 4 performs significantly worse. Could the authors clarify why they think this happens? Interestingly, the only task where the effect of N seems to not matter is the Ant task for which a hyperparameter sweep was completed. Additionally the task where increasing N impacts performance the most is the HumanoidStandup task where the returns are quite significantly different. To me, this result stresses that there might be more at play here and a more detailed study is required to tease apart the various confounding factors. \n\t\nIn summary, while I think the approach is quite interesting, there are concerns in some of the claims made in the text. I appreciate the effort that went into the current set of results and the experimental setup. With that in mind, I would be willing to accept this submission if my concerns above are clarified and if the conclusions drawn from the results are tempered given the evidence.\n\nFinally there are minor points of clarification that did not affect my overall review but I nonetheless list below:\n\n- In the discounted infinite horizon setup of Scenario 2, the timestep no longer needs to be added to the state. However the text indicates that this is still done even in this case. I think this does affect bootstrapping and thus learning the value target. Specifically it may be easier to learn a consistent value function that in this setting when the time index is not included in the state. Could the authors clarify this point?\n\n- As a minor point for readability, it would be good if the algorithm boxes for PPO-TD and PPO-TD-Ex etc included colours to highlight the changes to PPO (Algorithm 1) since these overlap quite a bit. This is purely from a presentation perspective of course.\n\nReferences:\nPeter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. arXiv preprint arXiv:1709.06560, 2017.\n\nAndrew Ilyas, Logan Engstrom, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. A closer look at deep policy gradients. arXiv preprint arXiv:1811.02553, 2018.\n\nLogan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep rl: A case study on ppo and trpo. In International Conference on Learning Representations, 2019.\n\nMarcin Andrychowicz, Anton Raichuk, Piotr Stanczyk, Manu Orsini, Sertan Girgin, Raphael ´ Marinier, Leonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, et al. What ´ matters in on-policy reinforcement learning? a large-scale empirical study. arXiv preprint arXiv:2006.05990, 2020.\n\nScott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. arXiv preprint arXiv:1802.09477, 2018.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "dXUvyB_O4yd", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "***Summary***\nThe paper proposes an empirical study of the discount factor as a regularization parameter in the actor-critic architectures. Specifically, the paper considers the case in which the actor and the critic employ different values of the discount factor. Two scenarios are considered. First, the paper analyzes the case in which the true objective is undiscounted and a discount factor is employed in the critic (like in TRPO and PPO). Second, the case in which the true objective is actually discounted but the discount factor is ignored in the update of the actor. A quite large suite of experimental results is reported.\n\n***Major issues***\n- (Organization) The paper presents an extensive experimental evaluation that is split between the main paper and the appendix. However, in the main paper, there are a lot of references and discussions related to experimental results that are provided in the appendix only. This happens both in Section 3 and in Section 4. Sometimes these results (presented in the appendix only) seem to be some fundamental claims of the paper, like for Figures 11, 12, and 13. I think this choice makes affects negatively the readability and clarity of the paper. Indeed, the reader has to continuously jump between the main paper and the appendix. Similarly, the pseudocodes are reported in the appendix only, but I think that this is less relevant compared to the plots. I think that the paper would greatly benefit from a reorganization, making it more self-contained.\n- (Bias-Representation Trade-off) One of the main claims of the paper is that using a discount factor < 1 in the critic when the true objective is undiscounted has a regularization effect not only on the variance but also on the learnability of the value function itself. I have to admit that the paper has not convinced me on this point. It is hard to say that the representation of the value function becomes more complex as the discount factor approaches one or, similarly, as the horizon increases. In general, I think that is possible to devise MDPs in which the value function representation becomes simpler as the horizon increases as well as MDPs in which it becomes more complex. I can imagine that for a class of tasks the statement can be true, but the paper does discuss the properties of these tasks. Can the authors elaborate more on this point?\n- (Auxiliary task perspective) The paper proposes a perspective of the critic update without a discount factor for a discounted objective as a sum of two terms. However, I have some concerns about the application of the clipping technique independently for the two terms. Why not perform the clipping just once to the original discounted objective? \n\n***Minor issues***\n- In Section 2, the MDP model is introduced assuming finite state-action spaces. Is this assumption really necessary? The experiments are carried out on Mujoco tasks that are characterized by continuous state-action spaces.\n- The plots are very small, including the ticks and labels on the axis. Moreover, they are not readable when printing the paper in grayscale. I suggest using different linestyles or markers.\n\n***Overall***\nI think that the paper addresses a relevant problem that is surely important to bridge the gap between theory and practice. However, I have some concerns about the organization and about the conclusions (especially regarding the bias-representation trade-off) that the paper draws from the presented results. For these reasons, I think that the paper is currently not ready for publication at ICLR.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Z55VCPn3Ekv", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors focus on the discounting mismatch in the Actor-Critic algorithm. From comprehensive experiments, the authors claim that this mismatch is either a bias-variance representation tradeoff or an auxiliary task for the actor update. Since the discounting mismatch problem is a well-known gap between the theoretical analysis and the application, their work, especially the experiments, might have some impact on how to understand this gap.\n\nHowever, since it does not provide any new analysis technique or practical model to improve the performance of the AC algorithm. I would encourage the authors to do more analysis of the choice of $\\gamma$, like how to choose $\\gamma$ might lead to a good performance (either experimentally or theoretically). And I believe that would have more impact on both the theoretical analysis and practical algorithm design. and Meanwhile, since in the first scenario, the mismatching of $\\gamma$ is considered to reduce the variance, it would be interesting if the authors could compare this kind of variance reduction with the stochastic variance reduction on the policy-gradient algorithms [1] [2] [3].\n\nTherefore, though this paper lacks a theoretical analysis or a ground-breaking experimental performance, this paper has an interesting and comprehensive experimental survey and proposes some new hypothesizes on this problem, I will suggest borderline accept for this paper. I might consider modifying my suggestion after discussion with other reviewers and the author's response.\n\n[1] Papini, Matteo, et al. \"Stochastic variance-reduced policy gradient.\" arXiv preprint arXiv:1806.05618 (2018).\n\n[2] Xu, Pan, Felicia Gao, and Quanquan Gu. \"Sample efficient policy gradient methods with recursive variance reduction.\" arXiv preprint arXiv:1909.08610 (2019).\n\n[3] Yuan, Huizhuo, et al. \"Stochastic Recursive Momentum for Policy Gradient Methods.\" arXiv preprint arXiv:2003.04302 (2020).", "extra_scores": {}, "extra_text": {}}, {"review_id": "Lazi9yFypq", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors examine the commonly used paradigm of not discounting in the policy gradient objective. They propose two hypotheses relating to discounting. (1) discounting the critic improves representation learning. (2) undiscounted policy gradient is similar to discounting + an auxiliary loss. These hypotheses are studied through a series of empirical tests in the MuJoCo domain with PPO. \n\nStrengths:\n- I believe this paper is asking the right type of questions about common setups. There are a lot of choices made in deep RL algorithms which don't align with theory and are otherwise unstudied and empirical studies are an important. \n- Some of the approaches used to answer these questions are quite unique.\n- Overall, there a lot of experiments both in the paper and the appendix, which is detailed. This is a paper which will benefit from the additional page of content as a lot of key figures can be shifted to the main body.\n\nWeaknesses:\n\nGiven the empirical nature of this study, it is really important to have robust experimentation to really answer the hypotheses the paper raises. I think the paper falls short at this aspect and I wasn't convinced by the arguments made for either hypothesis. Furthermore, the conclusions that could be drawn from the results are generally not that surprising. \n- I'm not sure PPO is the best algorithm to analyze many of these questions. For example, Engstrom et al., 2019 showed a lot of very minor implementation level details had a large impact on the performance. Consequently, it may be difficult to disentangle the actual causative factors in performance. This is problematic as many of the claims in the paper are supported by empirical tests where the performance is not strikingly different. For example, Figure 1 is meant to justify that for $\\gamma_c = 0.99$ additional transitions improved performance, but on several environments increasing $N$ to 2 or 4 seems to hurt performance, going against our intuition about variance reduction. Figure 2 shows that for $N \\neq 0$ there is a large performance drop, but all values of $N \\neq 0$ achieve a very similar performance rather than trending downwards as $N$ increases. To me this suggests a very brittle algorithm. \n- For section 3 the bias-variance trade-off is evident from prior work (as referenced by the authors) so the result is of course not novel. I think analyzing it in a deep RL setting is important but because of the problems mentioned prior, I didn't find that these results provide anything solid to add to our understanding. \n- The results for Figure 3 aren't convincing (1) because they are overfit, by selecting the best possible H for each it seems likely to always arrive at a high performing agent. (2) This more suggests that these environments don't require the full horizon to achieve a high performance. Consider a simple cartpole problem which is optimal using greedy actions but has a horizon of 1 million time steps. Since were in an approximate setting with deep networks, it isn't surprising that the agent can achieve a high performance without considering the full horizon. \n- The results from the toy MRP experiment and distributional RL do suggest some kind of connection to representation learning, but isn't considering a longer horizon simply a more difficult learning problem? Is the representation necessarily an important aspect here? I didn't find that the authors answered this question. \n- The conclusion from Section 4 is that $\\gamma_A=1$ is an inductive bias that all transitions are equally important seems entirely self-evident from the mathematical definition given it applies equal weight to all transitions. At the same time the main question of hypothesis 2 seems unanswered. Shouldn't AuxPPO $\\approx$ PPO, rather than DisPPO if this was true? \n- A single environment for Figure 9 is not enough to draw any meaningful conclusions. I did not find the discussion in B.1. convincing that the other environments were not suitable. Simply change $t_0$ for the other environments. From personal experience the horizon of Ant is generally large (near 1000) as the terminal condition is hard to achieve meaning the difference between Ant and the fixed length environments should be small.\n\nAdditional Comments:\n1. I do wonder if this paper is better off as two separate documents where each hypothesis is provided much more significant attention/experimentation. For example, hypothesis 1 isn't actor-critic specific and is also applicable to Q-learning based methods. These experiments could be simplified by looking at algorithms with significantly fewer components and more settings. \n2. For the PPO-TD-Ex experiment I think it's also worth considering extrapolation error (Fujimoto et al., 2019) in TD learning. Since $S^i_{t+1}$ is sampled from a single transition rather than a full trajectory it is not necessarily contained in the batch. As a result, $\\hat v$ is not trained on $S^i_{t+1}$ and produces an erroneous TD target. My first impression was that the performance drop for $\\gamma_c=1$ was not surprising but the performance gain from $N=1$ for $\\gamma_c=0.99$ was, and I think are are unanswered questions here. Another important reference is Bengio et al., 2020 which showed TD(0) generalizes worse than TD($\\lambda$) and there is clearly a related result here. \n3. Given MuJoCo environments are time-limited to 1000 time steps, 1024 heads for PPO-FHTD seems like a mistake/oversight. \n4. Why does PPO-FHTD with H=1024 produce different results for the different parametrizations? \n5. Is Figure 6 surprising since the value function needs to consider a large space of solutions as the horizon increases?\n6. Given distributional RL provides a large performance gain (which to the best of my knowledge, we are still missing a conclusive reason as to why), I'm not sure PPO-C51 > PPO-TD is a significant result. \n7. It would be clearer if DisPPO was described before mentioning Figure 15. \n8. Figure 15 seems like an important conclusion and should be contained in the main body of the paper. However, the y-axis of Figure 15 also conflicts with the description in the main body so I'm not sure what the correct interpretation is.\n9. I wonder if the result from Figure 9 is reproducible if the flipping was done in a different way. In the MuJoCo environments is the agent is rewarded mainly for velocity and the behavior of the agent in these cases would be enlightening. Does the agent run forward and then attempt to terminate? Can it move backwards?\n\nConclusion:\n\nI think the authors present a lot of interesting ideas and experimental approaches to answer their underlying questions. However, I felt that the experimentation was not sufficiently robust to justify their conclusions and I cannot recommend acceptance.\n\nReferences\n- Engstrom, Logan, et al. \"Implementation Matters in Deep RL: A Case Study on PPO and TRPO.\" 2019.\n- Fujimoto, Scott, et al. \"Off-policy deep reinforcement learning without exploration.\" 2019.\n- Bengio, Emmanuel, et al. \"Interference and Generalization in Temporal Difference Learning.\" 2020.\n\n** Edit (Nov 23): I have slightly increased my score due to the improvements made to the paper (mainly reorganization) & some clarifications made by the authors, but I still don't feel like my main concerns were addressed. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the effect of the discount mismatch in actor-critics: the discount used for evaluation (often 1), the discount used for the critic and the discount used for the actor. There’s notably a representation learning argument supported by a series of experiments.
The initial reviews pointed out that this paper addresses very relevant research questions, sometimes in a quite original way, with a large set of experiments. However, they also raised concerns about the organization/clarity of the paper, and possible weaknesses about the experimental studies.
The authors provided a rebuttal and a revision, that clarified some points and triggered additional discussions. However, if the revision improved the initial submission, the shared assessment is that the clarity and experiments themselves are still somewhat lacking. As such, the AC cannot recommend accepting this paper.
Yet, this work does have interesting ideas, and the problem considered is of interest for the community and under studied. The authors are strongly encouraged to submit a revised version to a future venue.

Decision: Reject