Adaptive N-step Bootstrapping with Off-policy Data
Authors: Guan Wang
Dong Yan
Hang Su
Jun Zhu
Venue:  (2021)
Keywords: reinforcement learning
policy evaluation

Abstract: The  definition  of  the  update  target  is  a  crucial  design  choice  in  reinforcement learning.  Due to the low computation cost and empirical high performance,  n-step returns with off-policy data is a widely used update target to bootstrap from scratch. A critical issue of applying n-step returns is to identify the optimal value of n.  In practice, n is often set to a fixed value, which is either determined by an empirical guess or by some hyper-parameter search.  In this work,  we point out that the optimal value of n actually differs on each data point, while the fixed value n is a rough average of them. The estimation error can be decomposed into two sources, off-policy bias and approximation error, and the fixed value of n is a trade-off between them.  Based on that observation, we introduce a new metric, policy age, to quantify the off-policyness of each data point.  We propose the Adaptive N-step Bootstrapping, which calculates the value of n for each data point by its policy age instead of the empirical guess. We conduct experiments on both MuJoCo and Atari games.  The results show that adaptive n-step bootstrap-ping achieves state-of-the-art performance in terms of both final reward and data efficiency.

Under review as a conference paper at ICLR 2021
ADAPTIVE N-STEP BOOTSTRAPPING WITH OFF- POLICY DATA
Anonymous authors Paper under double-blind review
ABSTRACT
The definition of the update target is crucial design choice in reinforcement learning. Due to the low computation cost and empirical high performance, n- step returns with off-policy data is widely used update target to bootstrap from scratch. A critical issue of applying n-step returns identify the optimal value of n. In practice, is often set to fixed value, which is either determined by an empirical guess or by some nyper-parameter search. out that the optimal value of actually differs on each data point, while the fixed value n a rough average of them. The estimation error can be decomposed into sources, off off-policy bias and approximation error, and the fixed value of is a trade-off between them. Based on that observation, we introduce a new met- ric, policy age, quantify the off-policyness of each data point. We propose the Adaptive N-step Bootstrapping. which calculates the value of n for each data point by its policy age instead of the empirical guess. We conduct experiments on both MuloCo and Atari games. The results show that adaptive n-step bootstrap- ping achieves state-of-the-art performance in terms of both final reward and data efficiency.
this work, we point
INTRODUCTION
The goal of reinforcement learning (RL) is to find an optimal policy by interacting with the en- vironment. order to do that, a RL algorithm needs to define a target, e.g., Q function or value function, and update it iteratively to bootstrap from scratch. The challenge of designing efficient update target manifests both on the sample complexity and computation complexity. Ideally, the target should only be updated by the data generated by the corresponding policy to obtain an unbi- ased estimate (Sutton & Barto 2018), while the amount the data needs to reach a certain scale to control the variance (Schulman et al. 2015). These two requirements limit the update frequency and lead to a high sample complexity finally. On the computational part, the consideration is to make a trade-off between the computation cost of each step and the number of total steps. Monte-Cart turns has advantages on generalization (behave well with function approximation) and exploration (a quick propagation of new findings) the cost of computing the whole trajectory each step (Sut-) ton & Barto 2018). Bootstrapping methods apply readily on off-policy data and control the trace length (Sutton Barto 2018). However, they require more update steps to converge comparing with Monte-Carlo returns. Those design concerns are nested together, which makes it hard to achieve good balance.
N-step returns (Sutton & Barto 2018) serves as the basis of various update targets, due to its flexi- bility and simple implementation. Together with off-po learning (Sutton & Barto 2018) and the
replay buffer (Mnih et al. 2015), n-step returns is able update the target frequently while ensures that the variance is in controllable range. However, a systematical study (Fedus et al. 2020)) reveals that the performance of n-step returns highly relies on the exact value of n. Since the underlying working mechanism is unclear, previous research can only give some vague suggestion based on empirical results, that simply increases the value from one larger number, e.g. or
In this paper, we illustrate that the estimation error of n-step returns can be decomposed into off- policy bias of n controls the balance between them. Data stored the replay buffer are generated by previous policies. Thus, adopting them for update introduces the off-policy bias. Since the current policy
part) and approximation error (over-estimation part), and the selection

---

Under review as a conference paper at ICLR 2021
is better than previous policies. the off-policy bias is underestimation. Replay buffer is not the only source of the ofllcc bias, apsilon-greede exploration also introduces the off- issue. On the other hand, n-step returns adopts max operator explicitly (on Q-learning based algorithms) or implicitly (on actor-critic algorithms) on an existing function to approximate the real target. The max operator brings the approximation error, which is overestimation. Sec gives the formal definition of the decomposition and verifies the conclusion by experiments. According to our anal- ysis, the quantity of the off-policy bias and approximation error varies a lot on different data points. Thus, fixed value of n is just a rough average, and there is plenty of room for improvement.
We introduce new metric, policy age, to quantify the off poolccycsssssss of each data point. As the policy age grows, the off- bias increases linearly, while the approximation error decreases exponentially. Based on this observation, we propose a novel algorithm, named adaptive n-step bootstrapping. Given the policy age of each data point, adaptive n-step calculates the optimal by an exponential function. Hyperparameter of the function determined by the tree-structured parzen estimator (Bergstra et al. 2011). We conduct extensive experiments on both MuJoCo and Atari games. Adaptive n-step bootstrapping outperforms all fixed-value n settings with a large mar- gin, in terms of both data efficiency and final reward. For the other update target definitions, select Retrace (Munos et al., 2016) as a representative. Compared with Retrace, our method main- tains the performance advantage under the premise of low computational complexity and simple implementation.
2
RELATED WORKS
2.1
RESEARCH ON N-STEP RETURNS
The recent works on n-step returns focus on finding the optimal value of n. In Ape-X (Horgan al. 2018) and R2D2 (Kapturowski et al. 2019). the value of n is fixed, which is set by manual tuning or pareeer-rreeterreeeee search. Rainbow (Hessel et al. 2018) figures out that the final performance is sensitive to the value of n, and n achieves the best score in most cases on Atari games. [Fedus et al. (2020) verifies that setting n is good choice, and further reveals that the replay buffer must also be large enough to gain performance benefits. Those researches give some heuristic rules of setting the value of but the underlying mechanism of why n 3 performs better than one-step temporal difference is still unclear.
2.2 OTHER UPDATE TARGETS
To improve the performance of the vanilla n-step returns, there are many other update target defini- tions in the literature Heemandez-Garee & Sutton 2019). Importance sampling (IS) (Precup et al.] 2000) provides simple way to correct the off-policy bias. It can be seen weighted average of multiple one-step TD(0) target. However, IS brings large (and possibly infinite) variance, which makes it impractical on large-scale problems.
Retrace Muunss et al. 2016) clips the IS ratio to a maximum value to reduce the large variance of IS targets. It has many applications in recent reinforcement learning agents, like distributed off- policy learning agent Reactor (Gruslys et al. 2017]. The most serious disadvantage of Retrace is its high computation cost. Retrace needs to calculate O(n) times of Q and O(n) times of in each time, compared with only O(1) from n-step returns (n is trace length). In large-scale problems, evaluating Q and æ requires forward pass in the neural network, which is slow and expensive. Reactor (Gruslys et al. 2017) calculates the Retrace target as linear combination of many n-step targets and dispatches those calculation workloads into different nodes for acceleration. Since the computation complexity is still high, reactor can not work under limited resources. Furthermore, even without considering the calculation cost, the application scope of Retrace is not as good as n-step returns, as reported in Beeraande--areiaa & Sutton (2019).
PRELIMINARIES
Reinforcement learning's goal is to find an optimal policy Tn with maximal discounted returns R₂ F, E, rr] given the Markov Decision Process (MDP). To achieve this, agents often
2

---

Under review as a conference paper at ICLR 2021
estimate the state-action value function (s E,, E, (t=less) a, a]. Let (s. denote the estimation of 9 (8, a). In tabular settings, Qx can be represented by a table of all state- action pairs (s, a), while in large-scale settings, Qr is often approximated by deep neural network (DNN) with parameter 0, written as Qr:0.
During the training process, Q. is continuously updated by the update target G., which is calculated from data points (s, a,,, s'). r,,r,, In tabular settings (Watkins & Dayan 1992), the update equation can be written as: (s. + (r a) + [ a) (x,s.a) a)}, where a is the learning rate. In
large-scale settings (Mnih al. 2015 Lillicrap et al. 2016). 2T:0 is updated by mini-batch gradient
ΣN VoL(Qx;0(8) az), G (5t, az)),
descent on the neural network parameter 0 as: + a N
where L is the loss function.
Off-policy learning adopts two policies, the behavior policy μ for generating data points and the target policy for learning from the data points. Replay buffer (Fedus et al., 2020 is often used together with the off- policy to handle the increasing sample complexity brought by large-scale prob- lems. Agents draw data points from the replay buffer update the estimator QT. This enables agents to learn from past experiences, thus yields higher sample efficiency. In the sequel, the term off-ppolccy learning refers adopt off-policy learning together with the replay buffer, as used by most recent off off-policy algorithms.
N-step returns (Sutton Barto 2018) together with off-policy learning have strong empirical per- formance (Hessel et al. 2018). as well as being easy to calculate. It calculates the G as:
(s, a) ro + m + - + 7" ((ss,) an)].
(1)
It can be seen as mix of Monte-Carlo (MC) estimation of 9Tr E. -"- (n - case) and one-step TD(0) estimation To + (=Ee= (s1, (n = case). In off-policy learning, to (80 Gg To: 16).
draw consecutive transitions (st, de, Tr, =,),.0,....... in the same trajectory T as current data point (so, ao. To, s)) from replay buffer. As n-step returns estimates 4r using trajectory generated by behavior policy μu, the discrep- ancy between π and μ makes it biased. We define this off off-policy induced bias as off-policy bias, and difference between π and j as off-policyness.
calculate n-step returns G% (Sp, for data point (No;
4 UNDERLYING WORKING MECHANISM OF N-STEP BOOTSTRAPPING
N-step returns lays in the center of designing the update target. It not only unifies the Monte-Carll returns and one-step temporal difference but also lays the foundation of the eligibility traces (Singh & Sutton, 1996). Together with off off-policy learning, n-step bootstrapping works well because achieves a good balance between the bias (TD) and variance (MC). Considering its importance and wide application, the underlying mechanism of n-step returns has not been studied in detail. this section, we give careful analysis of why n-step bootstrapping works and what property the optimal selection of n should satisfy.
4.1 DECOMPOSITION OF THE ESTIMATION ERROR
To understand how n-step bootstrapping works, we conduct a systematical analysis of the estimation error. We formalize the estimation error of an update target G2 its difference with ground truth 4ur for every (s, a) pair that the agent experienced:
{(G,) = (x, - an (8, ())
(2)
As shown in Eq. I the estimation error consists of two parts the off-policy bias and approxima- tion error. The off off-policy bias is introduced by the difference of л and μ., as we mentioned before, while the approximation error comes as agent's 's estimation of Q2 is not perfect, c.g. Qn # qr Note that, the behavior policy j is an older version of the target policy æ.
We split these two types of error by defining two intermediate targets, &n Gr and 6n Qu:7 to eliminate the other type of error.
3

---

Under review as a conference paper at ICLR 2021
Gm G.ggr adopts the ground truth dn instead of the estimated Qx to eliminate approximation enror:
G (s, rn + yr1 + -m= n- E - 19 (s a,,)].
G Qr:F uses the trajectory (fe; di, Ft; 0,), which is generated by the current policy instead of the old trajectory to remove the off-policy bias:
G G Fo + y + in-1 + "a - ~a.
1Q (5 an)].
Then we can quantify the off-poliey bias Eoflpoliey and the approximation error Espprox indepen- dently as:
Eoftpolicy (G))) E(G Eapprox (G#) E(C
Now, we can decompose the total error E(G into the sum of off-policy bias and approximation error, plus a negligible small residual term Eyesidual (G
Eresidual
(Gai)) (G*) 7 (En 1Q (8p, a) - 9n (sp, a)l Es (8,. - qa (Sp, a)l)-
E(G Eottpolicy (C+) Eapprox (G
The residual term Evesidual (Õ²) is a difference of the discounted error > ""Eawn(s) 1 qa (s, a)] on terminal states Sn and 8n- If n is small, the difference between Qa and qu on 8n will be close the difference between Qx and qa 5n. Otherwise, the discount factor y will shrink exponentially, making the residual term very small.
The experimental results in 4.2 the two main sources in practice. Thus, it can be ignored and we get an approximate decomposition:
4.224.2soo show that the residual term is order of magnitude smaller than
E(CT) R Eottpolicy (Ō+) Eapprox (Cm).
4.2 VERIFICATION BY EXPERIMENTS
In this section, we perform tabular Q-Learning and Soft Actor Critic (Haarnoja et al. 2018a) on the Pendulum-vO task (Brockman et al. 2016) to verify the decomposition quantitatively. For tab- ular Q-learning, we added replay buffer (Miih et al. 2015) as recent off off-policy learning does, and discretization to deal with continuous observation space and action space.
The approximation error is an overestimation, which means Eapprux 2 while the off-policy bias is an underestimation, Eoftpolicy < O. Single-step temporal difference target (n 1) will have a large approximation error, while too many steps (n is large) leads to too much underestimation that cannot be balanced out. N-step returns works because suitable selection of makes the overestimation and underestimation cancel each other.
eFror value (SAC) crror value (labular, ar;
total emrr
offpolicy emror
approx error
residual error
200
S-ttet
-25
200
10
100
32-iem
-0
100
-75
0
5000
10000
5000
10000
0
5000
10000
5000
10000
0
20
200
200
-00
10
100
1000 2000 3000 4000
1000 2000 3000 4000
1080 2000 3000 4000
1000 2000 3000 4000
oof-policyeess
Figure 1: N-step target errors. Top row is tabular settings, bottom is SAC. X-axis is off-policyness, quantified by policy age that is described in the beginning of See
4

---

Under review as a conference paper at ICLR 2021
Overestimation of Eapprox is well-studied issue. The greedy policy in Q-learning targets. Gradient ascent on л in actor-critic architectures also an implicit max operator (Fujimoto et al. 2018). The max operator is main cause of overestimation, reported in (Hasselt 2010). Usage of function approximation TThrun & Schwartz, 1999, Van Hasselt et al,, 2013) in large-scale settings also boosts this issue. Off-poliey learning with replay buffer is the root cause of the underestimation of Coffpolicy- With less learning update iterations, often has worse performance than æ. Estimating Or using y leads underestimating, as we are accumulating μ's reward in G
operator
As shown in Figure I on per-datapoint perspective, off-policy bias Eoffpoliey grows with off- policyness. Older data points take large proportion replay buffer, but data points from current policy π have only few, so Q- has been updated with much more older data points. That makes approximation error Eapprox decrease with off-policyness.
On value of perspective, larger leads to less approximation error, weight y" of estimated Q shrinks exponentially. But larger also leads target O accumulating rewards from μ., thus enlarges off-policy bias.
5 IDENTIFY THE OPTIMAL VALUE OF
The magnitude of off-policy bias is closely related to off- off-pooliysess. To analyze the error of n- step returns, we need a precise measurement of the off-pooicyeess. Evaluating real off- off-policyness directly, e.g. calculating the real difference between and μu is hard, as calculating the difference of two policies is non-trivial and computational heavy, especially for complicated environments. So introduce new metric policy age, which is simple to evaluate and also predicts well of real off- off-policyness. For every data point (5t, Qr, Tt. st). policy age is defined as number of update steps between æ and μ.. As show in Figure 2 policy age predicts accurately of the difference between and μu, e.g. Ey log (a) s) - log p(a ||]] for every (s, a) pair that agent experienced. We will use policy age as indicator off off-policyness in the paper.
5.1 ADAPTIVE N- STP BOOTSTRAPPING
As we pointed out in Sec 4.2, the optimal value of n achieves a balance between overestimation and underestimation. Since the optimal value of varies by policy age, a fixed value is only coarse approximation. For each data point, the selection n should be calculated individually to achieve the best performance.
We propose a novel algorithm, Adaptive N-step Bootstrapping, to select the optimal n which achieves minimal error for every data point. We define error for policy age p as:
(²( = Eup, GG a) (x,
(4)
where Hp refers to behavior policy that is updates away from æ. Then, the optimal n² for every data point is calculated as:
n n²(p) arg min 8d (c
(5)
Pendulum-vO
difference
0
1000
2000 3000
4000
policy age
Figure 2: The axis is the policy age, and the axis is the difference between l and n, which grows approximately linearly as the policy age grows.
5

---

Under review as a conference paper at ICLR 2021
Tabular Pendulum-vo
SAC on Pendulum--v
SAC on Swimmer-v3
140
exporental st
160
140
120
optimal
140
120
100
120
100
Value
80
60
60
40
40
40
20
0
800010000.2000
poll cy age
Figure 3: n² and exponential fitting, Note that we only evaluate error for 128 steps, so the value of n² may clipped a maximum of 128.
As calculating error E (P()T) requires the ground truth 9.. directly solving 5 on every data point is very expensive large-scale environments. We start by solving n² for a small number of data points. shown Figure 3 these data points reveal that n (p) is an exponential form wide variety of settings, both tabular and large-scale.
This exponential pattern is rooted the management of replay buffer and mini-batch gradient de- scent. In replay buffer, the larger the policy age of data point, the more times that it will be sampled into the mini-batch. This difference on sample times inflects both the off-policy bias and the approximation error. The off- off-policy bias part is simple, it increases linearly the policy age grows. The approximation error is little bit more complicated because the sample data update the parameter through gradient descent. Bhandari al. (2018) describes similar condition, and it concludes that the rate of convergence is exponential. In summary, as the policy age grows, the off- off-policy bias increases linearly while the approximation error decreases exponentially, Thus, use an exponential approximation for n² as follows:
Y (p) = arg minle
(6)
round(@mus *e - (oloo(())mmin(1...
where p the policy age for the data point. The maximum factor Thmax and decay rate d are hyper- parameters.
Adaptive n-step returns can also be seen form of off-policy correction, cutting when the difference between и and is large. It is simpler and more stable than IS and Retrace because does not rely on the metric like IS ratio, which may have infinite variance. Algorithm 5. describes how adaptive n-step bootstrapping works.
Algorithm Adaptive N-step Bootstrapping
Hyperparameter: Maximum steps Flmax decay rate d for each bootstrapping iteration do sample batch (s;, 0g, Fi, st) from replay buffer for each data point in batch do calculate policy age p for data point
12 round( * log(nmox) miim..)))
91 G (8i, a4)
end for
update Qn using batch (si, as rı. s/) with target 9: end for
5.2 HYPERPARAMETER SELECTION
As we show in Sec 5.1. solving Eq on limited data points is enough to reveal the exponential form. n" of complicated Swimmer-v's environment is also successfully reconstructed.

---

Under review as a conference paper at ICLR 2021
We also adopt the same setting to solve the optimal value of Rama and d. The exact solution Swimmer-v3 give us mmax 755. On near-onpolicy case, adaptive n-step behaves like MC target, which is unbiased for on-policy case (Sutton & Barto 2018). However, MC target also has large variance, which leads to performance degradation. So we need variance control measurements, like the factor in Retrace (Munos et al. 2016). Vanilla n-step returns sets
small value do variance control (Sutton & Barto 2018). We clip Rmax to reduce the variance in our method.
The notorious bias-variance trade-off makes it non-trivial solve optimal Témax- However, the solved = 122952 is very close to = 100000 that works for all MuJoCo tasks, hinting the best hyperparameter range. We found that the best hyperparameter varies little with different environ- ments. With this assumption, we use the tree-structured parzen estimator (Bergstra et al. (011)) to optimize hyperparameter on one environment and use this single set of hyperparameter for the whole benchmark.
EXPERIMENTAL RESULTS
Q-learning and Actor-Critic methods span the whole space of model-free reinforcement learning algorithms. Q-learning updates the state-action value function Q. and select action with maximum Qx in e-greedy manner. Actor-Critic has explicit representation of n, and update Qn and simultaneously. We conduct experiments on two representatives of both worlds, DQN (Mnih et al., 2013) for Q-learning methods, and SAC (Haarnoja et al. (0188) for actor-critic, to test the generality of our method.
6.1 ACTOR CRITIC METHODS
Soft Actor-Critic (SAC) is the state-of-the-art algorithm in off-policy actor-critic domain. It fo- cuses on stability and data-efficiency, and even can be applied to challenging real-world robot con- trol (Haarnoja et al. 2018b).
We compare adaptive n-step with fixed n-step and Retrace on the SAC algorithm. For each update target, use it in place of single-step TD target in original SAC implementation and evaluate on Gym MuloCo (Brockman et al. 2016) benchmark.
Ant-v3
HallCheetah-v3
10000
5000
average return
4000
8000
3000
6000
2000
1 stop
4000
2-steps
1000
4-steps
2000
Adaptive
Retrace
0
0.0
1.0
2.0
3.0
0.0
1.0
2.0
3.0
HumanoidStandup-v2
Swimmer-v3
275000
140
250000
225000
100
200000
average
175000
80
150000
125000
40
100000
75000
0.0
1.0
2.0
3.0
0.0
1.0
million steps
million steps
Figure Training curve on continuous control benchmark. report top a total 8 runs.

---

Under review as a conference paper at ICLR 2021
The result shows that adaptive n-step returns outperforms all fixed n-step returns consistently across all tasks. Adaptive n-step also has the lowest variance across different runs, being more stable. is worth to note that different environments have different best performing fixed n, while adaptive n-step perform well with only one set of max and d across all environments. This suggests that optimal n varies with different data points, while fixed is only coarse approximation.
Adaptive n-step also outperforms the Retrace method, both in terms of average return and compu- tational cost, The calculation of Retrace target requires 2n Qx and 3n π evaluations per step (n is trace length, n = 32 for our MuJoCo experiment), and each evaluation is a network forward pass. In contrast, adaptive n-step larget inherits O(1) complexity of the vanilla n-step method, only evaluates 1 Qr and π for arbitrary n, thus makes it much faster than Retrace.
6.2 Q-LEARNING
DQN (Hessel et al. 2018) sets the foundation of combining Q-Learning with deep neural networks, we pick as the representation of Q-learning methods. We conduct our experiment on a subset of Atari 2600 games. We compare the adaptive n-step with fixed n-step with = 3, which is the best n on Atari games recommended by Hessel et al. (2018) and Fedus et al. 2020).
BreakouttWoFFrmmskiipvv
Asteroids NiseroiissooFPammskipp.v
BeamRRderkooFFameekkkty
350
300
1400
4000
250
3000
200
150
800
2000
100
3-step adaptive
600
50
1000
400
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
AsterialloFrammssiip.d
1e7
Bankkieisttofrrmmsskivvv
167
CentipedeNof rameipcde v4
rameskip-ve 1e7
5000
7000
1000
4500
AV0 55 return
800
5000
3500
4000
600
3000
400
2500
2000
2000
1000
1000
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2 0.4
0.6
0.8
1.0
1e7
1o7
1e7
+ of training frames
Comparison on Atari 2600 games. For both fixed 3-step and adaptive n-step, report the agent that obtained highest reward during training. The figure is smoothed by moving average with length 10 improve readability.
Figure 5:
As shown in Figure 5 adaptive n-step outperforms the fixed value n-step returns in all games. And in most of them, the performance benefit brings by the adaptive n-step method exceeds 20%.
CONCLUSION
Generally, n-step bootstrapping is simply viewed as a unification between Monte-Carlo returns and one-step temporal difference. However, with the introduction of replay buffer to apply reinforcement learning on large-scale problems, we figure out that n-step bootstrapping actually serves as a control factor to reduce the estimation error. Thus, the selection of should differ on each data point instead of a fixed value.
Based on this observation, propose the adaptive n-step bootstrapping algorithm to select the value of n for each data point individually. Experimental results show that adaptive n-step outperforms all fixed value settings with large margin. Comparing with other update target definitions, e.g. Retrace, adaptive n-step bootstrapping only introduces negligible computation cost and is easy to implement. Those characters make it be easily embedded into other algorithms.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
James S Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper-parameter optimization. In Advances in neural information processing systems, pp. 2546-2554, 2011.
Jalaj Bhandari, Daniel Russo, and Raghav Singal. finite time analysis of temporal difference learning with linear function approximation. arXiv preprini
2018.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540. 2016.
William Fedus, Prajit Ramachandran, Rishabh Agarwal, Yoshua Bengio, Hugo Larochelle, Mark Rowland, and Will Dabney. Revisiting fundamentals of experience replay. arXiv preprint arXiv:2007.06700, 2020.
Scott Fujimoto, Herke Van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. arXiv preprint 1802: 09077,
A. Gruslys, Mohammad Gheshlaghi Azar, Marc G. Bellemare, and R. Munos. The reactor: A sample-efficient actor-critic architecture. ArXiv, abs/1704.04651, 2017.
Haarnoja, Aurick Zhou, Abbeel, and Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In ICML, 2018a.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- cations. arXiv preprint arXiv:1812.05905 2018b.
Hado V. Hasselt. Double q-learning. In J. D. Lafferty, C. K. I. Williams, J. Shawe-Taylor R. S. Zemel, and A. Culotta (eds.), Advances in Neural Information Processing Systems 23, pp. 2613-2621. Curran Associates, Inc., 2010. URL http: ccppseer/ppper
J Fernando Hernandez- Garcia and Richard Sutton. Understanding multi-step deep reinforcement learning: A systematic study of the dqn target. arXiv preprint 1901v01901.055/1 2019.
Matteo Hessel, Joseph Modayil, H. V. Hasselt, T. Schaul, Georg Ostrovski, W. Dabney, Dan Horgan, B. Piot, Mohammad Gheshlaghi Azar, and D. Silver. Rainbow: Combining improvements in deep reinforcement learning. ArXiv, abs/ 1 100002298, 2018.
Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado Van Hasselt, and David Silver. Distributed prioritized experience replay. arXiv preprint arXiv:1803.000%) 2018.
Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney. Recurrent expe- rience replay in distributed reinforcement learning. 05 2019.
Lillicrap, J. Hunt, A. Pritzel, N. Heess, Erez, Y. Tassa, D. Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. CoRR, 0ttp//1990099999 2016.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Marc Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg. and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518:529-33 33, 02 2015. doi: 10.1088/nature142335
R. Munos, Tom Stepleton, A. Harutyunyan, and Marc G. Bellemare. Safe and efficient off-policy reinforcement learning. In NIPS, 2016.
Doina Precup, R. Sutton, and Satinder Singh. Eligibility traces for off-policy policy evaluation. In ICML,, 2000.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv: /506.023384 2015.

---

Under review as a conference paper at ICLR 2021
Satinder P Singh and Richard S Sutton. Reinforcement learning with replacing eligibility traces. Machine learning, 22(1-3):123-158 1996.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
S. Thrun and A. Schwartz. Issues in using function approximation for reinforcement learning. 1999.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. arXiv preprint arXiv:1509.06445 2015.
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3-4):279 -929 1992.
10


Reviews:
[{"review_id": "pAhghkOm-_K", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper deals with an intriguing point in RL -- how to correctly choose an adaptive $n$ for $n$-step bootstrap. From such a paper one might expect theoretical results on the tradeoff between bias and variance in the presence of off-policyness. However, as opposed to the picture portrayed in the first section, no such analysis follows. \n\nI believe the research question at hand can highly benefit from rigorous analysis, but other than the rudimentary Eq. (3) and the few equations that precede it there are no established principles according to which one should derive a formula for $n$. Instead, the argument that is given for choosing it regards the 'age' of the policy. But, this argument is hand-wavy and might as well be less important than other factors such as the level of accuracy of the value function per given state at a given time. Also, such a formula for updating $n$ is highly dependent on parameter choice which would naturally change for different environments. \n\nI'm also particularly disturbed by the fact that it seems, by the age heuristic used in the paper, that there's a hidden assumption that DQN is a monotone algorithm. This is even explicitly mentioned in the first sentence of page 2. But it is, of course, incorrect. Based on stochastic approximation analysis, only asymptotic results on convergence exist or finite-time analysis that doesn't assume monotonicity. \n\nThere is also a comparison to Retrace [Munos et al., 2016] and claims on the method here being more efficient. I find that selling point acceptable but relatively weak; the Retrace paper is backed by detailed analysis and thorough comparison to other alternatives. A similar distinction can also be made, by the way, to the Vtrace paper [Espeholt et al., 2018]. \n\nIn terms of experiments, it seems that the proposed algorithm works well. But, as mentioned, I tend to believe it needs tweaking per environment, which is equivalent to tweaking to a 'good' $n$ value. \n\nTo summarize, as an empirical paper that focuses on a problem that is very natural to be theoretically analyzed, I consider the contribution here to be not significant enough for publication. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "U3QFlQieiEb", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Evaluation\nAlthough the paper presents interesting observations and some promising experimental results, I recommend rejection. The proposed method is not general, because it is purely based on empirical observations obtained with only SAC and Q-learning. As a result, it is unclear if the proposed method can be combined with other algorithms. The proposed method is also a kind of hack that makes use of the overestimation problem, and thus, it won't work if we solve the overestimation problem. Besides, the paper is not well-written, and experimental methods are a bit inappropriate.\n\n### Paper Summary\nUncorrected N-step return is very effective and frequently used in off-policy RL algorithms, such as R2D2 and Ape-X. Its drawback is that \"N\" is determined heuristically or by a hyperparameter search. The paper presents interesting observations that\n\n1. an error between expected uncorrected N-step return and the true Q-value mainly consists of underestimation due to uncorrected off-policy bias and Q-value overestimation due to function approximation error, \n2. balance between the underestimation and overestimation can be controlled by \"N\", and that\n3. an optimal $N$, which minimizes the error, obeys $A e^{-Bp}$ with appropriate $A, B \\in (0, \\infty)$, where $p$ is _policy age_.\n\nBased on these observations, the paper proposes to change and determine $N$ for each $G_t$ by $round(N_{max}^{1 - \\min(1, p/d)})$, where $N_{max}$ and $d$ are hyperparameters. Some experiments show performance gain compared to algorithms with fixed N.\n\n### Strong Points\n1. Interesting observations on uncorrected N-step return (described in the summary above)\n2. Some performance gain in MuJoCo.\n\n### Weak Points\n1. The paper is not well-written. It contains typos, unclear sentences, and ambiguous notations.\n2. The proposed method for choosing N does not seem to be general. (I feel it is specialized in SAC and Q-learning. Besides, it is a kind of hack that does not work if we solve Q-value overestimation.) I wonder if this method works for other algorithms, especially algorithms using trust region methods.\n3. The abstract says a critical issue of uncorrected N-step return is that one needs to choose $N$. The proposed method allows us to avoid choosing $N$, but now we need to choose $N_{max}$ and $d$, which also seems to be difficult.\n4. SAC's performance in the paper is substantially lower than that of a publicly available implementation (e.g., [SpinningUp](https://spinningup.openai.com/en/latest/spinningup/bench.html)). This made me a bit skeptical of the SAC implementation of the paper.\n5. Figure 4 shows average learning curves of top 4 agents. Figure 5 shows learning curves of best agents. I feel mean, median, and std across all agents should be provided.\n\n### Comments to the Authors\n\n1. The paper is not well-written. Please revise it again. Besides, please revise the references section. It refers to arxiv versions of papers that were accepted to conferences.\n2. Is it possible to show that the proposed method can be combined with other algorithms? In particular, I am wondering if the proposed method can be combined with algorithms using trust region methods. (For example, PPO?)\n3. SAC's performance in Figure 4 seems to be substantially lower than that of a publicly available implementation (e.g., [SpinningUp](https://spinningup.openai.com/en/latest/spinningup/bench.html)). Would you share your code so that I can check the implementation?\n4. Or would you run SAC experiments with SpinningUp using N-step return?\n5. Why do you show learning curves of best agents (or top 4 agents) rather than mean or median?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "5csEMD9ujb", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an approach to adapting the n parameter in n-step returns according to the off-policyness of the sampled transition. The key novelty to approximate this off-policyness using the age of the policy that generated the data, and demonstrating that there is a fairly reliable relationship between the two in simple experiments.\n\nOverall, I enjoy the approach taken here, and I think it deserves a place in the landscape of deep RL research. However, I'm not sure that the paper is of sufficiently quality to be accepted as-is. The paper is quite roughly presented, with a number of imprecisions and technical inaccuracies. The empirical results are interesting, but would benefit from being substantiated.\n\nSection 4.1. There are a number of issues here which I'm curious to hear your comments on. \n1) Equation (2): What is the expectation on mu over? Just a?\n2) In this case, where are the r tilde coming from? There seems to be some kind of other sampling\nprocedure generating this.\n3) E_approx(.) is defined in terms of an expectation over mu (c.f. Eqn 2), but I think you really mean pi.\n4) As a consequence, the derivation of E_residual is not quite right. I think you end up with a q_pi - q_mu term. Please confirm, and explain what that means for your argument.\n5) In this context, what does that mean for Eqn 3? Clearly if gamma is large, the first two terms are small. But what about the q_pi - q_mu term?\n6) Actually, the derivation of E_residual seems to imply that the expectation in Eqn 2 is over a full trajectory, otherwise you cannot swap out the sample partial returns with Q-values. But in this case there may be other terms, e.g. the distributions of s_n and s tilde n come from the visitation distribution induced by mu, which is different from that induced by pi.\n\nSection 4.2. \"The approximation error is an overestimation ...\" I don't understand this paragraph, or why it is necessary from the previous section's derivation. Can you give a formal argument for this point? Clearly Fig. 1 shows that the effect is happening, but I don't think your math supports it.\n\nYour definition of n* (Equation 5) only seems to make sense in deterministic environments. Please comment.\n\nAt a higher level, there are a number of presentation gaps between stated results and the theory given. It would be useful to emphasize which of the conclusions are given by theory, versus experiments, versus conjecture.\n\nFig. 2 suggests an almost linear relationship between p and the log-differences. Is this because you are applying SGD with a step size of 0.001, and hence finding that a first-order Taylor expansion to mu, centered at pi, fits well here?\n\nFig 3. Why are all the x axes different? Can you explain these graphs -- would they look the same if the problems were stochastic?\n\n\"In summary, as the policy age grows, the off-policy bias increases linearly ...\" where do you show this? Is this an anecdotal observation from your experiments?\n\nYou optimize hyperparameters on one environment... this matters a lot here. Can you give some details?\n\nFig 4. What hyperparameters are you using in the end?\n\nFig 5. The figure titles are quite rough. Better would be to use the actual game name, not the Gym environment string identifier. Also, I believe v4 is without sticky actions, which woudl make the domain deterministic. Can you confirm?\n\nIn general, the results of fig. 5 don't support the claim that the method is having a significant impact. The claim that the adaptive method outperforms on all games also isn't true -- it's clearly worse on centipede.\n\nWhere you cite Thrun and Schwartz, 1999 -- consider citing Gordon, 1995 instead.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "244u53IS73s", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\nThis paper provides a novel algorithm to estimate the optimal value of $n$ for $n$-step temporal difference methods. The paper derives an optimal value of $n$ based on minimizing a bias term, then utilizes intuition to derive an online approximation algorithm. The paper compares its adaptive $n$-step algorithm against fixed $n$-step values with both DQN and SAC empirically on several domains.\n\n## Review\n\n### Summary\nI am recommending a reject for this paper. The paper is built from the assumption that \"the underlying mechanism of why $n=3$ performs better than one-step temporal difference is still unclear.\" However, the paper misses a discussion on the well-understood bias-variance tradeoff present with $n$-step methods. I do not find that the paper provides any clarity or novelty in the _understanding_ of $n$-step methods. Although the proposed algorithm is novel, I remain unconvinced by the empirical demonstration and the lack of theoretical support for the algorithm.\n\nTo increase my rating of the paper, I would have liked to have seen at least some of:\n * A discussion and analysis of the bias/variance properties of the proposed algorithm.\n * A proof that the proposed algorithm has better bias/variance properties than a naive fixed $n$ approach.\n * A statistically sound empirical investigation of the proposed algorithm (even if on smaller domains, like tabular gridworlds).\n * Fewer meta-parameters for the proposed approach.\n * A sensitivity analysis of the meta-parameters for the proposed approach.\n\n### Theory\nThe theoretical components of the paper were largely poorly written, involving typos, inconsistencies, poorly motivated definitions, several forward references, etc. which made reading and reviewing quite challenging.\n\n* The terms \"off-policyness\" and \"policy age\" are used throughout the paper; however, they are only defined intuitively (i.e. not precisely) until section 5 where they are given an unsatisfactory definition. The paper defines \"off-policyness\" as the number of steps between when the behavior policy was recorded and the current timestep. This, however, does not take into account any aspect of the dynamical system defining the change in policies. For instance, imagine Q-learning with a stepsize of 0. The policy would never change, yet the \"off-policyness\" by definition would grow linearly according to this definition. The lack of solid definition here requires an unfortunate environment/algorithm specific meta-parameter later in the analysis to account for this fact (more discussion on this later in the review).\n\n* Throughout the paper, the term \"error\" is used but the formula given is $\\mathbb{E}\\left[ \\hat{G} - v_\\pi \\right]$ which is the definition of bias. While a minor nitpick in terminology, this leads towards a lot of confusion throughout where some quantities are defined as \"errors\" while other as \"biases\", but what is ultimately meant is \"bias\". The term \"approximation error\" means something specific generally, and is not used correctly in the paper (for instance). Because \"bias\" is the only portion of the error that is investigated throughout the paper, then the most important portion of the discussion around $n$-step methods is ignored (namely, the variance). Even simplifying the setting investigated by this paper to being fixed-policy (e.g. policy evaluation) with target and behavior policies the same (i.e. on-policy), then still there is a trade-off between $n=1$ and $n=\\infty$. This trade-off results from balancing between bias and variance of the estimated returns. While adding shifting policies and considering the off-policy setting does yield an interesting conversation on bias, the variance simply cannot be ignored.\n\n* The error decomposition is incorrect. It does not account for all of the error in the estimate, resulting in a \"residual\" unaccounted for error. We do not know what causes this error, we do not know if it implies that the defined accounted for errors are incorrect due to the residual error, or if there is simply some additional term not being accounted for, but we do know that the magnitude of the residual error is non-negligible due to the empirical investigation in Figure 1. Further, because we are actually looking at biases not errors, we cannot make use of the non-negativeness of these quantities to assert their independence. The residual error appears positive. What if the residual error results from not accounting for all of the error due to \"off-policyness\" (which is always negative) and thus cancels with the off-policy error?\n\n* Where did Equation 7 come from? This appears to be entirely intuited from a couple of empirical plots? Does the equation even make sense? I see $n^* = \\text{round}\\left( \\frac{n_\\text{max}}{n} e^{\\min(1, \\frac{p}{d})} \\right)$ which results from an $\\text{argmin}_n$. How do we solve this equation with an $n$ on both sides wrapped in a non-linear function? Is this based on Bhandari et al. (2018) where this paper incorrectly reads that gradient descent has an **exponential** convergence rate? Where does gradient descent come into play (TD is not the gradient of any function)? Why limit the exponent to be at least 1? Doesn't this lead to an algorithm where $n=2$ is the smallest n-step method we can consider?\n\n### Empirical Study\nI highly appreciated section 4.2. I liked the investigation of the quantities defined in the theoretical section and asserting that assumptions hold. I however do not agree that the empirical results here imply the conclusion drawn. In what way do we see that \"N-step returns works because a suitable selection of $n$ makes the overestimation and underestimation cancel each other\"? This conclusion is not clear from the results. It also lacks the appropriate nuance given the highly limited study and lack of statistical evidence to support the claim.\n\nHow did you choose the meta-parameter value of $d = 122952$? This seems....arbitrary at best. What does the sensitivity of $d$ look like? $d$ appears to result from the fact that you did not account for the rate that the policy is changing, only the number of timesteps that have passed. This makes the fraction $\\frac{p}{d}$ become an approximation for policy change rate, where the onus is entirely on the algorithm user to estimate and tune this quantity for their given algorithm, other meta-parameters, and environment.\n\nThe rest of the plots in section 6 onwards have a fundamentally broken empirical methodology that renders the results largely uninterpretable. Comparing the mean and standard error (I believe this is what the shaded region shows?) of the top 4 out of 8 runs will not yield statistically valid results. The differences between the algorithms will be lost to variance and noise, making it impossible to know which algorithm is actually best. More runs would be needed and likely a different validation metric as well. Note that using the 50th percentile disproportionately favors high-variance methods as it cuts off their failing runs. This implies to me that the proposed algorithm has much higher variance (as I would guess from the algorithm itself) and only works a small percentage of the time. Perhaps this is still a desirable trait, but this should be stated and investigated explicitly and not hidden behind strange empirical decisions.\n\nI will refer to Henderson et al. 2015 for a discussion on why showing the mean and standard error of the top 4 out of 8 runs is insufficient for a statistically valid methodology. I will also attach a short python script which demonstrates a simplified version of Figure 5 from Henderson et al. 2015, demonstrating very clearly that no conclusions can be drawn from this methodology. Even ignoring my complaint at the unusability of the empirical results in section 6, still the difference in performance of the proposed algorithm and a 2-step baseline appear negligible; especially once one considers the introduction of two new meta-parameters to replace the original one meta-parameter, and that these two meta-parameters appear difficult to tune.\n\n#### A demonstration of methodology\n```python\nimport numpy as np\nnp.random.seed(0)\n\nN = 8\nTOP_N = 4\n\n# hidden underlying process\n# let's see which of these two \"algorithms\" is better\n# Note, however, that they are in fact the same\nalg1_performance = np.random.normal(0, 0.1, size=N)\nalg2_performance = np.random.normal(0, 0.1, size=N)\n\n# only look at top n points\nalg1_performance.sort()\nalg2_performance.sort()\n\nalg1_top = alg1_performance[-TOP_N:]\nalg2_top = alg2_performance[-TOP_N:]\n\n# which is better? (note both should be approximately 0)\nprint(\"mean 1:\", alg1_top.mean()) # => 0.17\nprint(\"mean 2:\", alg2_top.mean()) # => 0.08\n\n# can we estimate variance? (Nope)\nprint(\"std 1:\", alg1_top.std()) # => 0.046\nprint(\"std 2:\", alg2_top.std()) # => 0.042\n\n# what about confidence intervals? (Nope)\nalg1_stderr = alg1_top.std() / np.sqrt(TOP_N)\nalg2_stderr = alg2_top.std() / np.sqrt(TOP_N)\n\nprint(\"CI 1:\", (alg1_top.mean() - alg1_stderr, alg1_top.mean() + alg1_stderr))\nprint(\"CI 2:\", (alg2_top.mean() - alg2_stderr, alg2_top.mean() + alg2_stderr))\n# yields CI 1: (0.15, 0.19)\n# yields CI 2: (0.06, 0.10)\n# these confidence intervals don't overlap, therefore\n# conclusion: they must be different algorithms\n# and alg1 has better average performance, so alg1 must be better\n```\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies n-step returns in off-policy RL and introduces a novel algorithm which adapts the return’s horizon n in function of a notion of policy’s age.
Overall, the reviewers found that the paper presents interesting observations and promising experimental results. However, they also raised concerns in their initial reviews, regarding the clarity of the paper, its theoretical foundations and its positioning (notably regarding the bias/variance tradeoff of uncorrected n-step returns) and parts of the experimental results. 
In the absence of rebuttal or revised manuscript from the authors, not much discussion was triggered. Based on the initial reviews, the AC cannot recommend accepting this paper, but the authors are encouraged to pursue this interesting research direction.

Decision: Reject