Stabilizing Off-Policy Reinforcement Learning with Conservative Policy Gradients
Authors: Chen Tessler
Nadav Merlis
Shie Mannor
Venue:  (2020)
Keywords: Deep Reinforcement Learning
Variance Reduction
Policy Gradient

Abstract: In recent years, advances in deep learning have enabled the application of reinforcement learning algorithms in complex domains. However, they lack the theoretical guarantees which are present in the tabular setting and suffer from many stability and reproducibility problems \citep{henderson2018deep}. In this work, we suggest a simple approach for improving stability and providing probabilistic performance guarantees in off-policy actor-critic deep reinforcement learning regimes. Experiments on continuous action spaces, in the MuJoCo control suite, show that our proposed method reduces the variance of the process and improves the overall performance.

Under review as a conference paper at ICLR 2020
STABILIZING OFF-POLICY REINFORCEMENT LEARNING WITH CONSERVATIVE POLICY GRADIENTS
Anonymous authors Paper under double-blind review
ABSTRACT
In recent years, advances in deep learning have enabled the application of re- inforcement learning algorithms in complex domains. However, they lack the theoretical guarantees which are present in the tabular setting and suffer from many stability and reproducibility problems [10]. In this work, we suggest sim- ple approach for improving stability in off off-policy actor-critic deep reinforcement learning regimes. Experiments on continuous action spaces, in the MuJoCo con- trol suite, show that our proposed method reduces the variance of the process and improves the overall performance across most domains.
INTRODUCTION
Reinforcement Learning (RL) is a dynamical learning paradigm, in which the algorithm (also known as the 'agent') learns through sequential interaction with the environment. On each round, the agent performs an action, which transitions it into a new state, and is provided with state-dependent reward. The goal of the agent is to maximize the cumulative reward observes throughout these interactions with the environment. When considering tabular RL, in which there is finite number of states and actions, there exist efficient algorithms with theoretical convergence and stability guar- 5 4] However, this is not the case when considering deep RL approaches when a neural network is used to cope with large state spaces, such as images 19 and/or large action spaces, for example in continuous control [7]
antees [29: 13 14 34
The issue with deep RL twofold: (i) the optimization process is finite, and as such does not follow many of the requirements for the process to converge, such as decaying learning rates (2), and (ii) the non-linearity of the function approximators (e.g., neural networks) results in highly non-convex
optimization landscape. Due to this, as can be seen in all empirical works [22 26 17 6 9. the learning process exhibits high variance; during short time of training, a near-optimal policy may become arbitrarily bad. As a result, in recent years, the stability of RL algorithms has become major concern of the research community [10]
Instead of tackling the stability problems in the optimization process, many previous works focused on different aspects of the loss function 66 30). Then, they usually apply standard variant of gradient descent algorithm on the modified loss function. While these approaches are capable of finding relatively good policies, they strictly inferior when compared to their tabular counterparts the tabular approaches ensure convergence and stability, whereas the practical variants exhibit instability and high variance.
In this work, we suggest novel approach that improves the stability of actor-critic methods. Specif- ically, we draw the connection between deep RL approaches and their tabular counterparts and high- light the issues inherent in deep RL algorithms. Through this connection, we propose a solution, in the form of a hierarchical training procedure. As opposed to the tabular case, approach is not ensured to converge, but rather has stability guarantees, i.e., the stationary distribution of the policies is shown to be stable, meaning that with high probability the performance of the policy improves and does not suffer high degradation. We show, empirically across range of continuous control tasks in MuJoCo, that our approach indeed results improved behavior both in terms of stability and overall performance.
BACKGROUND AND NOTATION
We consider an infinite-horizon discounted Markoy Decision Process [23] MDP]. An MDP defined the 5-tuple (S,A, P,r,)), where S is the state space, A the action space,

---

Under review as a conference paper at ICLR 2020
Algorithm Policy Iteration
1: Initialize To
2: k = 0
3: while not converged do
4: while not converged do
> Policy Evaluation
5: for E S do
6:
0aa (8) = r(s, (s)s + Ex'es (x)s)
7:
for E S do
Policy Improvement
8:
(s) E arg aa a(s.a) + +rEse a('
9: k= 1=
return policy Th
Algorithm 2 DDPG, TD3 and our Conservative update
1: Input: algorithm, learning rate a, averaging parameter 7, steps '" and exploration M.
2: Initialize policy 0, critics @² o6² and replay buffer R
3: Set 0__ = 0, o- = =6", =0
4: while 1 do
5: Collect data and append to replay buffer
6: t=t+ ++++1
7:
(s,, a,, 8141) } R.sample()
Training phase
8:
if DDPG then
9:
Φ Ø a Vollr + 7Q (si+1, no_ (8141)) Qo(s;,
else
TD3
11:
Φ = Φ1 a V 10/1/1 + min, Qsi ($141, To_ (s:+1)) - Q (s,, a
0 Φ aV 02 001rr Ir + ming Qos ($141, To_ (s Q (s,, a
== VoQ (s,, (s:))
14:
6' T)- Td
Update target networks
if Conservative Update then / % Eireg = 0 then evaluate Te and me_ for Emum episodes each P(J _ )
16:
17:
18:
then
19:
0_ =
20:
else
0_ 0= (1-1)--
Colored lines represent similarities between the classic and modern approach.
P:
is a transition kernel, r: : x > Tmin [max] is a reward function and E (0, is the discount factor.
The goal of RL agent is to learn a policy (ss a) that maps states into a distribution over actions.
The quality of policy is measured by its value, vd (s) = E "' 7'r(s,, a.) So =s], which the average cumulative reward when starting from state s and acting according important quantity the Q-function, Q" a) = E Ei "'r(St, az) Su = s, as = a). which is strongly connected to the value through the relation (= = Ja Q a) a)d We denote an optimal policy that maximizes the value simultaneously across all of the states by X¹ arg max, 8 and the optimal value function by v² " When the Q-function and the policy æ represented by parametric function, we denote them by Qs and πT. where & and 0 are the parameters of the function (e.g. the parameters of deep neural network).
Another
While goal is maximize v², in practice measure empirical estimation of Es e (s), where p the initial state distribution. To accommodate for this fact, we define the cumulative reward of sampled trajectory by J This is random variable, as randomness may occur due to (i) the initial state distribution, (ii) the transition kernel and reward function and (iii) the policy.
2

---

Under review as a conference paper at ICLR 2020
2.1 CLASSIC REINFORCEMENT LEARNING ALGORIHHMS
Policy Iteration 12 PIJ is algorithm for solving tabular MDPs with convergence guarantees. PI (Algorithm D iterates between two steps: (i) policy evaluation in which the current policy Ty is evaluated, producing the value v and (ii) policy improvement in which the policy T is updated greedily w.r.t f.a This iterative scheme is proven to converge to the optimal policy.
Pl schemes have also been extended to the approximate case. Scherrer (25] provides an overview of such approaches, which are called Approximate Policy Iteration (API). In API we consider the scenario in which || y** I1>0 S e. This approximation error can be caused due to two factors, the first being the inability to estimate the value and the second being the inability to find the greedy policy, i.e. the arg max at each state. For instance, such errors may occur due to the functional class being used, e.g., neural networks. Most importantly, the sub-optimality
these approaches
bounded by ||o v" 1| (1->)² (25), i.e., the sub-optimality of the final policy is proportional the approximation error c.
2.2 OFF-POLICY DEEP REINFORCEMENT LEARNING FOR CONTINUOUS CONTROL
DDPG [17 and its latest improvement TD3 66 use neural networks as function approximators in order find the optimal policy. Both approaches utilize a target network, which is crucial for the success of the methods. While in their work, this was merely considered as a design parameter, we argue that the importance of these target networks lies in the connection to Policy Iteration.
Deep Deterministic Policy Gradients I7 DDPG) (Algorithm 2 aims to directly learn determin- istic policy in a continuous action space. DDPG has both policy evaluation and policy improvement steps. The Q-function, also known the critic, is trained to estimate the utility of the target policy To_, whereas the online policy (the actor) trained find the 1-step greedy update w.r.t. this esti- mation. We argue that DDPG and PI are connected through the target network update procedure. In PI, the previous policy ग is evaluated by ,* followed by improvement step in which πk+11 be- comes the one-step greedy policy w.r.t. this estimate. DDPG follows a similar scheme, in which the critic Qa estimates the utility of the policy me_, followed by the online policy та which is updated w.r.t. this estimate. In this work, compare TD3 6 an improved version of DDPG.
Twin Delayed DDPG (6 TD3] improves the DDPG algorithm by tackling the problem of overes- timation. Similarly to how Double-DQN (32) overcame the overestimation problem by introducing an additional Q function, in TD3 they introduce an additional critic. They show. both theoretically and empirically, that by bootstrapping the minimal value between both estimators min, Qi π (s)) they are able to overcome the problem of overestimation.
INSTABILITY IN POLICY-BASED DEEP REINFORCEMENT LEARNING
In the previous section, we provided an overview of Pl, which has convergence guarantees, and two off-policy DRL variants. However, since in practical applications the policy is represented using non-linear neural network, small change in the parameters 0 may result in very large change in the output. We illustrate this issue Figure where state of the art off off-policy approach learns to control the Hopper and InvertedPendulum robots.
In these figures, we show the individual learning curves over several seeds, without performing temporal smoothing"] whereas the bold line represents the mean performance across runs. At each evaluation period, we evaluate the performance of the agent across 100 episodes. These results paint a clear picture- - while the agent is capable of learning a policy which attains a score of approximately 3500 in Hopper and the optimal score of 1000 in InvertedPendulum, it is highly unstable, i.c., policy which was near-optimal at time might become arbitrarily bad at time & +
'Temprrl smoothing common practice in empirical studies, in which each data-point in the graph represents moving average over the past N evaluations. While this makes the graphs easier read, hides lot of important information such as the variance of each seed during training.
3

---

Under review as a conference paper at ICLR 2020
Hopper-v2
InvertedPendulum-22
1000
500
2500
2000
200
200
400
600
200
600
Figure 1: Performance plots of the TD3 [6] algorithm.
CONSERVATIVE POLICY GRADIENTS
While there exists a connection between the tabular approach and that which is used with non-linear function approximators, due to the non-convex optimization landscape, small gradient step can lead to an arbitrarily bad solution. As such, current policy-based methods with non-linear function approximation (neural networks) behave poorly when compared to their tabular counterparts.
We propose simple, conservative, yet theoretically grounded approach for tackling these issues. Taking inspiration from the classic approaches, we observe that the gradient-based algorithms lack the improvement guarantees. To overcome this issue, we propose a hierarchical learning procedure: while the online network is trained as before (as seen in Algorithm 2), the target network is updated periodically if, with high probability, the online policy is better.
In simple terms, our proposal is as follows: every T time-steps, evaluate both the target network" and the online network. with confidence of 1 the online network is hetter than the target, perform the switch 0_ = 0. In addition, between updates, the target network is kept fixed, as opposed to the standard approach (17: 6 which performs Polyak averaging [1]. We provide additional discussion
in Appendix D
The following proposition motivates this approach. We discretize the range [Umin Umax] into bins of size A and formulate the learning process random- walk, in which with probability 1 6, the policy improves (1 step to the right) and decreases otherwise. The random-wall walk bounded both ends by Umin and Umax. such that the value is projected back into this range.
Proposition 1. Let of be random walk admitting values in the set V {Umin, Umin A ...., Umax A, Umax]. Also, assume that w.p. d: o = min(vī + A, Umax and 1 max A, Umial otherwise. Then, the stationary distribution of of equals P(u 2)
aaa for any E V, where r T-8-
Proof Sketch. We can write the recurrence relations by:
P(u = Umin) = (1 - 8)P(c = Umin) + 5PP(v = Umin + A)
P x) (1--5)P(c A) 5P(r x A)
PP( Ummx) (1 5)1P(v Ummax A) + 5P( Umax)
the stationary distribution follows.
illustration of this process is provided Appendix A When the probability of improvement 50%, i.e., we are unable to determine which policy is better, the stationary distribution over the value may be arbitrarily bad. However, the confidence in the improvement of the policy increases, the distribution shifts towards delta function at the optimal value function.
Remark 1. In practice, the optimal attainable policy a function of the initial starting parameters. Due to the non-convexity of the optimization landscape, not all initializations are ensured to be capable of attaining an optimal policy.
"The target network needs he re-evaluated in order ensure an unbiased estimate of its performance.
4

---

Under review as a conference paper at ICLR 2020
No
Evallate online policy over episodes
Train anline pollcy for N steps
Set target policy's perameters to be to online policy's parameters
Yes
Evaluate target polley over K episodes
online policy better the target probability at least 1-6 ?
Figure 2: Conservative Policy Gradients.
EXPERIMENTS
Implementing our approach in an off-policy actor-critic scheme is straightforward. While current approaches I1: 6 hold policies, online and target policy, such that the target policy is updated each step using a Polyak averaging procedure [21] - our approach, illustrated in Fig. 2 requires the target policy to be updated only when the online policy is superior, with high probability.
Since in framework, the online policy is aimed at finding the 1-step greedy policy, and due the continuous nature of the action space, we add 'trust region' element to the loss X/ (s] Fe (s) such that the online policy is kept close to the target. In addition, we found that collecting data interchangeably from the online and the target network leads to further improvement in stability.
For all the experiments, we plot different random seeds for each configuration, discarding failure [] seeds] At each evaluation step, each seed is evaluated for 100 episodes, and the average perfor- mance is presented. We plot both the individual seeds (light curves) and the average results across seeds (dark curves). As opposed to previous works, we do not smooth the graphs using a moving average procedure, which enables better analysis of the experiments' variance and stability.
5.1 EVALUATING THE POLICIES
Our approach requires determining whether or not an online policy is better, with probability 1 6. Assuming J the performance of sampled trajectory for any deterministic policy æ, is normally distributed, we may consider the t-test for evaluating improvement. We formulate the null hypothe- sis as HTT, J < not improved over the target policy, thus we use the Welch's one-tailed t-test [33]. The f-statistic
ET MTa_ In other words, we hypothesize that the online policy has
is then defined as = (MTo- Pmo_.) // ).)r)/((
(Voor))(V///)) o + 03 /n where Paz is the empirical
estimate of E, G₂ is its standard deviation and ny the number of sampled trajectories. The t- statistic enables us to find the p-valuc, which represents the confidence in the null hypothesis. Under Gaussian assumption, the p-value cumulative distribution function of a standard normal random variable. Hence, when < 1 - å we have a high enough confidence that the null hypothesis is incorrect thus the new policy is better, in which case we update the target policy.
be directly calculated by p
(({), where +(x) is the
5.2 PARAMETER SELECTION
For this approach, there are three parameters which require tuning. (i) how often we evaluate the online policy, (ii) how many episodes are used for evaluation and (iii) the minimal confidence re- quired for policy improvement. Notice that there is a connection between the confidence and the number of evaluation episodes; as the number of evaluations grows, the confidence itself increases. Hence, when there a relatively high variance in the evaluations, low number of samples will sult in a low confidence, even when the empirical mean is higher. On the other hand, the evaluation frequency controls how fast the process converges. We evaluate the effect of these parameters and provide comprehensive results Appendix B
'e observed that some seeds in deep RL result in utter failure, for instance, score of 40 in Hopper. This issue is relatively rare, and also happens the baselines, but hinders the proper evaluation of the policy.
5

---

Under review as a conference paper at ICLR 2020
Confidence Value ó: The confidence level is 5, i.e. P(E natural upproach would be to attempt a high level of confidence, our empirical results show other- wise. Demanding a confidence which is too high, makes it much harder to find an improving policy, resulting in stable yet strictly sub- sub-optimal policy. On the other hand, when very low, the approach becomes unstable. We found that ò [0.1, 0.2] work well in practice.
8. While
Evaluation Episodes K: The number of evaluations is correlated to the confidence. A higher number of evaluations increases the confidence in the empirical mean, enabling the algorithm to determine that policy is better evaluating the policy takes time and a multitude of evaluations may drastically lengthen training.
when the difference is small. However, this comes cost
Evaluation Frequency N: When comparing evaluation frequency, we see that when the evalua- tion is performed every 10, 000 steps, it requires high probability of improvement to successfully converge. However, when the evaluation is performed more often, the process behaves well even for a lower improvement probability (fewer evaluation episodes).
we opted to run with a minimal confidence for swapping 90%, evaluation every 1, 000 steps over 10 episodes (sampled trajectories).
Based on these insights and the results in Appendix B
5.3
RESULTS
TD3
C-TD3
Hopper-v2
Inverted@eendllumeee
Hopper-v2
InvertedPendulummv2
1130
45
1.20
30
30
2001
1001
n/verteddzoubbeeeendduuu um--y2
Wolker/2d-22
Inverteelzoublleeenntu umm-y2
Walkerzd-y2
2M0
4.24
12
HallCheetahb.u
Ant-v2
HollChhootah--2
Antrv2
12352
0.50
-1.9
-100
Figure 3: Evaluation without (TD3) and with the Conservative (C-TD3) update rule. X-axis denotes the number of gradient steps taken.
We compared our approach to the baseline TD3 without our policy selection scheme. The results are presented in Figure These results show high resemblance the theoretical model, showing that, across all domains, the conservative approach (Conservative- TD3) reduces the variance of individual seeds, also seen in Table 1 and results in increased stability across most domains.
InvertedPendulum and InvertedDoubleeenduue of finding an optimal policy- e.g., the maximal score subsequent policies may be arbitrarily bad. By adding the hierarchical selection scheme, our approach is capable of reducing this behavior. That is, once optimal policy found, it is not likely to be replaced with an inferior
serve perfect examples. While TD3 is capable
interesting result can be seen the Walker2d domain. While some seeds attain near-optimal per- formance, others converge to sub-optimal policy. As our approach searches for a high-confidence improvement, it may sometimes get stuck. However, the inner-process variance, i.e., the variance of individual seeds during training, is much lower across all domains.
Finally, we observe sub-optimal behavior in the Ant domain, both in terms of individual seeds and average performance across seeds. The parameters (confidence, number of evaluation episodes and

---

Under review as a conference paper at ICLR 2020
Online
Target
X02
00
2514
40x
801
105
14
(a)
(b)
(c)
(d)
Figure 4: Failure analysis of the Ant-v2 domain.
evaluation frequency) were selected based on the Hopper domain, however it is important to note that these parameters not optimal for each domain.
Figure presents the standard deviation (STD) of both the online (Figure 4a and target (Figure 4b) networks, Ks, and the difference between the empirical means 1 j& (Figure (cc). This clearly shows that in Ant, there is very large variance, compared the difference between the means, which results in a low confidence. To overcome this issue, we provide an additional experiment, for Ant domain based on these insights (Figure 4d), which considered a minimal confidence of 20% (90% in Figure 3) These results show that while the parameters transferable across domains, optimal parameters found on one domain are not necessarily optimal for another.
5.3.1
STABILITY ANALYSIS
Hopper
InvertedPendulum
InvertedDoublePendtlumm
Walker2d
HalfCheetah
Ant
C-TD3
36
0
99
81
57
TD3
799
139
2409
468
350
902
Table 1: Process STD calculated over the last 100 training episodes.
To analyze the stability of each algorithm (TD3 vs C-TD3), we consider the variance of the policy during the last 100 episodes of training. In Table present the maximum of the standard deviation in different seeds, where per-seed STD calculated over the 100 last episodes training.
Our empirical results match the theoretical behavior. Indeed, the inner-seed variance is dramatically reduced when using the conservative update rule, as opposed to the non-conservative TD3.
5.4 POTENTIAL EXTENSIONS
In this work, presented approach for providing probabilistic improvement guarantees. The performance of a trajectory a random variable, as the initial state, the transition kernel and some- times the policy - may all be stochastic. To determine which policy x_ or (the target or online) better, performed N evaluations, assumed that the samples formed a f-test. Below we present several intuitive extensions to our work:
normally distributed and per-
While the Gaussian assumption worked well in practice, it may not always be correct. In such scenario, maximal possible values are known, where one may use the Empirical Bernstein [18), or (ii) when the range is unknown, one may consider a bootstrap- poostrap-bbsed approach for estimating the confidence intervals around the mean [I]
Unknown distribution:
two possible routes: (i) known range, in which the minimal and
Upper/Lowee Confidence Bounds: Another approach is consider improvement by a minimal
margin. For instance, we may set the update rule to P(v" 2 n ) > - 8. so that the statistical test ensures that with high confidence, the online policy has improved by at least over the target.
Adaptive Sampling: In our approach, we considered rather small number of evaluations, and thus resorted to an equal number of evaluations of each policy up to 100 evaluations for each policy. However, when provided with larger budget, it is of great interest combine adaptive sampling procedures. Gabillon al. 7 propose such an approach, which takes into account the empirical mean and variance of each policy evaluation in order to determine which policy to evaluate. At the end of this process, the policy with the highest empirical mean is returned.

---

Under review as a conference paper at ICLR 2020
In our initial experiments, we tested the UCB/LCB approach. However, we found that is domain- specific. While in some domains the attained rewards are relatively high, and the policies improve greatly between iterations, in others the improvement is rather small. This led to an early conver- gence to sub-optimal policies. Nevertheless, this approach may prove beneficial in certain settings and should considered, Additionally, we tested policy swap based on the empirical estimate of the mean, without ensuring a proper confidence. While this approach worked relatively well in practice, as it lacks proper confidence bounds, we decided to focus on the t-test based approach.
6 RELATED WORK
Trust Region Optimization:
Recent works in on-policy RL have also taken inspiration from Con- servative Policy lteration [15] CPIJ. Schulman et al. [26] formulated learning through a trust region optimization scheme, which was later optimized in Schulman et al. (27]. These approaches bound the KL-divergence between the policy before and after the update. While TRPO greatly improved the stability compared to previous policy gradient approaches, such as A2C 20 it lacks the theo- retical guarantees.
Safe Policy Improvement: Given a baseline policy, safe policy improvement (SPI) algorithms aim to find a new policy that is better than the baseline with high probability [31 8 [6]. Although this may seem similar to our update stage, there are two major differences: (i) most SPI algorithms directly calculate the improving policy, which is much less efficient than continuously updating the policy and performing the switch when improvement is achieved, and (ii) due to safety constraints, SPI algorithms usually perform of--policy evaluation for the suggested policy, which is still an open problem in high dimensional problems. In contrast, we evaluate it directly and thus enjoy a better sample complexity.
Evolution Strategies: ES 24 28 3 is gradient-free approach for solving reinforcement learn- ing tasks. At each iteration, given the previous policy To., a set of augmented policies is created (Te,+e. ); and evaluated, where & is some random noise. The ES update rule can be seen form of numerical gradient estimation (i.e., finite differences) [35]. There are some similarities between our approach and ES. While ES is a gradient-free method, uses empirical evaluations of the poli- cies to determine the parameter update direction. Our approach uses these evaluations in order to determine whether or not the policy has improved, and in contrast to ES, will only change the policy there is a high enough confidence for improvement.
CONCLUSIONS
In this work, we tackled the stability problems of policy- based reinforcement learning approaches. Even though the empirical gradient (SGD) is a noisy estimate of the ideal descent direction, and due to the structure of the optimization landscape and the large step sizes used in practice, the policy's performance varies greatly during training.
Drawing from the connection between actor-critic approaches and approximate policy iteration schemes, we introduced a hierarchical training scheme, which we analyzed by comparing this ap- proach to a bounded random-walk. While this scheme is not exact representation of the optimiza- tion process, it provides an intuition on the behavior of such a probabilistic improvement scheme, i.c., as the probability of improvement increases, the stationary distribution over the performance of the policy shifts towards the optimal solution.
Finally, we proposed the Conservative-TD:: which performs a periodic evaluation of both the online and target policies such that it updates the target network only when there is a higher enough confi- dence that the online network has surpassed its performance. We validate this on several continuous control tasks in MuJoCo; empirical evidence shows that indeed this approach behaves similarly to the theoretical analysis. When compared to the baseline, we observe a dramatic reduction in vari- ance, increased stability and an overall improvement in the performance itself.
REFERENCES
[1] Vineet Abhishek and Shie Mannor. A nonparametric sequential test for online randomized experiments. In Proceedings of the 26th International Conference on World Wide Web Com- panion, pp. 610-616. International World Wide Web Conferences Steering Committee, 2017.
8

---

Under review as a conference paper at ICLR 2020
[2] Vivek S Borkar et al. Stochastic approximation. Cambridge Books, 2008
[3] Edoardo Conti, Vashisht Madhavan, Felipe Petroski Such, Joel Lehman, Kenneth Stanley, and Jeff Clune. Improving exploration in evolution strategies for deep reinforcement learning via population of movelty-seeking agents. In Advances Neural Information Processing Systems, pp. 5027-5038, 2018.
[4] Christoph Dann, Lihong Li, Wei Wei, and Emma Brunskill. Policy certificates: Towards ac- countable reinforcement learning. arXiv preprint arXiv: 1.030066 2018.
[5] Yonathan Efroni, Nadav Merlis, Mohammad Ghavamzadeh, and Shie Mannor. Tight re- gret bounds for model-based reinforcement learning with greedy policies. arXiv preprint 1rXiv:1905.11522 2019.
[6] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error actor-critic methods. arXiv preprint arXiv: 180.00090 2018.
[7] Victor Gabillon, Mohammad Ghavamzadeh, Alessandro Lazaric, and Sébastien Bubeck. Multi-bandit best arm identification. In Advances in Neural Information Processing Systems, pp. 22222222222222 2220, 2011.
[8] Mohammad Ghavamzadeh, Marek Petrik, and Yinlam Chow. Safe policy improvement by minimizing robust baseline regret. In Advances in Neural Information Processing Systems, pp- 2298-2306, 2016.
[9] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, pp. 1856-1865, 2018.
[10] Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
[11] Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dab- ney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining im- provements in deep reinforcement learning. arXiv preprint arXiv: 1710.00200 2017.
[12] Ronald A Howard. Dynamic programming and markov processes. John Wiley, 1960.
[13] Thomas Jaksch, Ronald Ortner, and Peter Auer. Near-optimal regret bounds for reinforcement learning. Journal of Machine Learning Research, 11(AAp):1563- 1600, 2010.
[14] Chi Jin, Zeyuan Allen-Zhu, Sebastien Bubeck, and Michael I Jordan. Is q-learning provably efficient? In Advances in Neural Information Processing Systems, pp. 4863-4873, 2018.
[15] Sham Kakade and John Langford. Approximately optimal approximate reinforcement learn- ing. In ICML, volume 2, pp. 267-274, 2002.
[16] Romain Laroche and Paul Trichelair. Safe policy improvement with baseline bootstrapping. arXiv preprini arXiv: 1722.06924 2017.
[17] Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: 550900297, 2015.
[18] Andreas Maurer and Massimiliano Pontil. Empirical bernstein bounds and sample variance penalization. arXiv preprint vxXiv:0907.374p0 2009.
[19] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
[20] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu, Asynchronous methods for deep reinforce- ment learning. In International Conference Machine Learning, pp. 1928-1937, 2016.
[21] Boris T Polyak. New stochastic approximation type procedures. Automat. Telemekh, 7(98- 107):2, 1990.

---

Under review as a conference paper at ICLR 2020
[22] LA Prashanth. Policy gradients for evar-constrained mdps. In International Conference Algorithmic Learning Theory, pp. 155-169. Springer, 2014.
[23] Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 1994.
[24] Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor. and Ilya Sutskever. Evolution strategies as scalable alternative to reinforcement learning. arXiv preprint arXiv: Ttt:::/733.0300 773.00364,, 2017.
[25] Bruno Scherrer. Approximate policy iteration schemes: comparison. In International Con-
ference Machine Learning, pp. 1314-1322, 2014.
[26] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, pp. 1889-1897, 2015.
[27] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
(28) Felipe Petroski Such, Vashisht Madhavan, Edoardo Conti, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Deep neuroevolution: Genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning. arXiv preprint 2017.
[29] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
[30] Chen Tessler, Guy Tennenholtz, and Shie Mannor. Distributional policy optimization: An alternative approach for continuous control. arXiv preprint arXiv:1905.09855, 2019.
[31] Philip Thomas, Georgios Theocharous, and Mohammad Ghavamzadeh. High confidence pol-
icy improvement. In International Conference on Machine Learning, pp. 2380-2388, 2015.
[32] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learming. In Thirtieth AAAI conference artificial intelligence, 2016.
[33] Bernard L Welch. The generalization ofstudent's' problem when several different population variances are involved. Biometrika, 34(1/2):28-33 83,,, 1947.
[34] Andrea Zanette and Emma Brunskill. Tighter problem-dependent regret bounds rein- forcement learning without domain knowledge using value function bounds. arXiv preprint 1rXiv:1901 10, 2019.
[35] Xingwen Zhang, Jeff Clune, and Kenneth O Stanley, On the relationship between the openai evolution strategy and stochastic gradient descent. arXiv preprint arXiv:1712.06566 2017.
10

---

Under review as a conference paper at ICLR 2020
A RANDOM WALK
Random walk value occupancy comparison
p-0.5
Pm0.6
08
Ocoupancy
02
60
100
Value
Figure 5: Stationary distribution of a bounded random walk process. As the probability of improving (moving to the right) increases, distribution shifts towards a delta function around the optimal value.
We expect our approach to approximately behave random walk on bounded region. As the probability of taking step towards the right increases, the stationary distribution shifts towards the optimal value. On the other hand, as the uncertainty grows, i.e., the probability of improvement becomes closer y. the stationary distribution is uniform.

---

Under review as a conference paper at ICLR 2020
B PARAMETER ABLATION TESTS
In this section, we present the simulation results for all possible parameter combinations, in the Hopper domain. Specifically, tested minimal required improvement probability of [0.5, 0.8, 0.9, 0.95J. number of evaluations per evaluation steps of [5, 10, 100) and evaluation fre- quency of [1000, 10000).
Eval Freq: 1000, Eval Num:5, Conf: 0.95
Eval Freq: 1000, Eval Num:5, Conf: 0.9
1500
3000
2500
2000
1500
500
Eval Freq: 1000, Eval Num:5, Conf: 0.8
Eval Freq: 1000, Eval Num:5, Conf: 0.5
4000
3500
2500
3000
3000
2000
1500
1500
1000
1000
200
400
600
510
200
400
600
300
Figure 6: C-TD3, parameter ablation. Comparison is performed every 1,000 steps over episodes with swap confidence 8 = 95, 90, 80 and 50.
Eval Freq: 1000, Eval Num:10, Conf: 0.95
Eval Freq: 1000, Eval Num:10, Conf: 0.9
4000
3500
3000
2500
2000
2000
1500
500
500
200k
400
600
800
202
400
600
800 K
M.
Eval Freq: 1000, Eval Num:10, Conf: 0.8 Eval Freq: 1000, Eval 10, Conf: 0.5
4020
3000
3020
2500
2500
1000
1000
1000
520
200
400
020
800
200
400
000
800
Figure 7: C-TD3, parameter ablation. Comparison is performed every 1,000 steps over 10 episodes with swap confidence of = 95, 90, 80 and 50.
12

---

Under review as a conference paper at ICLR 2020
Eval Freq: 1000, Eval Num:100, Conf: 0.95
Eval Freq: 1000, Eval Num:100, Conf: 0.9
6000
3500
3050
3000
2530
2000
1300
1200
503
500
2003
4001
100
500
1M
200
430
600
900
Eva Freq: 1000, Evall Num:100, Conf: 0.8
Eval Freq: 1000, Eval Num:100, Conf: 0.5
4050
35.00
3500
2500
2550
2000
2000
1000
1200
Figure 8: C-TD3, parameter ablation. Comparison is performed every 1,000 steps over 100 episodes with swap confidence of - 8 95, 90, 80 and 50.
Eval Freq: 10000, Eval Num:5, Conf: 0.95
Eval Freq: 10000, Eval Num:5, Conf: 0.9
4000
3500
3000
2500
2000
2000
1500
500
500
200
400
600
800
202
400
600
800 K
M
Eval Freq: 10000, Eval Num:5, Conf: 0.8 Eval Freq: 10000, Eval Num:5, Conf: 0.5
3030
3020
2500
2500
1000
1000
1000
520
200
400
620
800
200
400
000
800
Figure 9: C-TD3, parameter ablation. Comparison is performed every 10,000 steps over 5 episodes with swap confidence of 8 = 95, 90, 80 and 50.
13

---

Under review as a conference paper at ICLR 2020
Eval Freq: 10000, Eval Num:10, Conf: 0.95
Eval Freq: 10000, Eval Num:10, Conf: 0.9
6000
3500
3550
3000
2530
2000
1300
1200
503
500
200
4001
100
500
200
430
500
900
Eva Freq: 10000, Eval Num:10, Conf: 0.8 Eval Freq: 10000, Eval Num:10, Conf: 0.5
4050
35.00
3500
2500
2550
2000
2000
1000
1200
Figure 10: C-TD3, parameter ablation. Comparison performed every 10,000 steps over 10 episodes with swap confidence of 1 - 6 95, 90, 80 and 50.
Eval Freq: 10000, Evall Num:100, Conf: 0.95
Eval Freq: 10000, Eval Num:100, Conf: 0.9
1200
400
600k
Eval Freq: 10000, Eval Num:100, Conf: 0.8 Eval Freq: 10000, Eval Num:100, Conf: 0.5
4002
3000
2000
2000
2000
1000
1000
200
100
600
1.1
400
620k
Figure 11: C-TD3, parameter ablation. Comparison performed every 10,000 steps over 100 = 95, 90, 80 and 50.
episodes with swap confidence
The graphs above show the effect of tuning the evaluation frequency, the number of evaluations and the minimal required confidence. As the evaluation becomes more frequent, the agent learss faster. As our approach requires the critic to evaluate the target policy æ- the online policy aims to find the 1-step greedy policy. This may happen rather quickly, thus a more frequent evaluation scheme will improve faster. On the other hand, the number of evaluation episodes is correlated to improvement, i.e., the confidence we have that indeed "better" than т_
the confidence see that as the number of evaluations rises, the agent is capable of finding good solutions even when higher confidence is required. However, lower confidence results in noisier training regime and a small number evaluations leads to a low confidence hence high confidence requirements with a low number of evaluations often leads to the inability to converge to good solution.
14

---

Under review as a conference paper at ICLR 2020
C
ONLINE Vs. TARGET NETWORKS
TD3 Actor
TD3 Target
Hoppor-v2
Hopppe-v2 v2
2550
2300
200
1933
-.5
4008
830%
2001
430%
2004
14
InvertedPendulum-v2
InvertedPendulummv2
800
532
300
400
InverteeDouubleenndduuuzzz
invertedDoubleeendlluuy
6000
600
222
401
6.0
203
020
Walker2d v22
Walker2d v22
0.10
4200
2.3
401
$20
082
203
4.0%
062
1001
HalfCheetah-v2
HalfCheetah-v2
22300
300
0030
2000
220
02.6
0.0
801%
1.4
19.1
4001
11.
Anl-v2
Anl-v2
4000
6204
200
420
225
1.00
500
3.50
Figure 12: On the left is the evaluation of the online network, whereas on the right we present the performance curves of the target network. It is clear across all domains that the target network cxhibits a dramatic reduction in variance.
The common approach is present the performance of the online network. In these graphs, show that, as opposed to common belief, the target network suffers less from instability and offers improved performance. These results are intuitive once the connection to policy iteration schemes is made.
15

---

Under review as a conference paper at ICLR 2020
Parameter
Value
Comment
0.1
The trust-region weight.
leam_start
1,000
Number of steps during which actions sampled uniformly.
swap start
10,000
Initial steps during which training is performed using Polyak Averaging.
max timesteps
le6
Total number of steps during training.
expi_noise
0.1
Exploration noise variance.
policy_nor nnose
0.2
Smoothing parameter for the critic training.
noise_clip
0.5
Clip the noise [-0.5, 0.5].
policy_free freq
2
Policy is trained once every two critic training phases.
reset. coceetpllyyy...o noise
0.01
Noise applied to policy when stuck for too long.
Table 2: Hyper Parameters.
D EMPIRICAL DETAILS AND DISCUSSION
We follow the same hyper-parameter scheme as is used for the TD3 algorithm [6] The first 10, 000 steps are played using randomly sampled actions, during which the target policy is updated in Polyak averaging technique. After which we initialize the conservative periodic policy updates.
In addition, we noticed that while the target policy has a high probability stability assurance, this does not hold for the online policy. As such, we found that often the online policy converges to strictly sub-optimal solution. We overcome this issue using two heuristics, (i) using a trust-region update rule in which we penalize (an additional loss term) the distance between the action predicted by the online and the target policies. As the critic estimates the performance of the target policy, the estimation error grows as the online policy diverges from the target. Finally, (ii) if the policy has not been updated for over 20, 000 steps, we assume that it has converged to a sub-optimal region event,
of the loss landscape. At such (O, 0.1) this is akin to an ES search procedure in which noise added to the parameters in order to overcome local extrema. This approach helps, but does not ensure, preventing the approach from "flatlining" in which the online policy is no longer updated.
set the online parameters as 0 =
where
While the online policy benefits from the periodic updating procedure, we observed that when the critic is also updated periodically, the performance becomes strictly sub-optimal. We believe that this is since this introduces additional time-scale, where three timescale algorithms are known to be tricky to tune and properly operate.
Finally, we observed that in certain initializations both TD3 and C-TD3 were incapable of learning. This resulted in a flat-line throughout the entire training regime, e.g., results such score of 40 in Hopper. Although rare, this is indeed an important problem in DRL, which needs be addressed properly, and is not apparent in supervised learning regimes. However, as this is not the focus of this paper, we discard such seeds enable the focus on the proper training regimes.
16


Reviews:
[{"review_id": "S1xsOUhhYH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed to use target network policy as a conservative policy for performance evaluation. Instead of performing Polyak averaging on the target network, the authors proposed to utilize statistical hypothesis testing to check whether the performance of the online policy is better than the target network policy and update the target network policy according to the results of the hypothesis testing.\n\nThe paper is clear written and easy to follow the core idea. As for the experiments, the authors evaluate the proposed method on a variant of TD3 (Conservative-TD3) and the experimental results indicate the proposed method indeed reduces the variance of the expected return. Ablation studies has been provided in the appendix to show the effectiveness of the proposed method.\n\nBesides the promising results, I believe there are several concerns that should be clarified before we can conclude that the proposed method can improve the stability. \n\n- Stability Measurement: While the experimental results show that the proposed method can reduce the variance of expected return, it is not a direct measurement of the stability or the robustness of the learned policy. It is better to show whether the proposed method can satisfy stability or robustness definition of RL algorithms. (For example, whether the proposed method can improve the robustness: Given a policy, by adding a \\epsilon perturbation to the input, the output is still \\epsilon-robustness) Otherwise, the author should add some discussion to clarify the difference.\n\n- Conservative Updates on Q functions. In Actor critic frameworks such as DDPG or TD3, the performance of the actor is usually determined by the critic. The proposed method is more likely to “select” stable actors rather than directly improving the stable of the training process or the stability of the policy. I wonder whether it is possible to improve the stability of Q updates such that the “selection” process of policy can be easier, which may accelerate the current training process (or making the hypothesis easier to satisfy).\n\n- More related work should be compared. The authors only compare the original version of TD3 and the modified proposed method. Other recent proposed methods to improve stability should be compared in the experiments, such as Constrained Policy Optimization (Achiam et. al 2017), Lypunov-based Safe Policy methods (such as Chow et. al 2019). \n\n- Noisy Environment. The authors demonstrate the stability of the proposed method in the ordinary mujoco benchmarks. How does the proposed method perform in the noisy MDP settings? Since the original Mujoco implementation is deterministic, the experiments that the author conducted are not enough to show the proposed method can generalize to more realistic settings such as noisy MDPs. It would be more convincing if the method can still perform well in such settings to support the claim.\n\nOverall I think the authors proposed a simple but yet effective method to improve the stability of policy, while the current paper requires more comparison with other methods and more challenging settings to show the effectiveness of the proposed method. \n\n--------------------------------\nI will update my score if the author clarify above questions.\n\n\n-------------------------------\nThe author clarified one of my main concern, but the other reviewers point out that the comparison is not fair (using only 5 seeds and discarding the failure seeds).\n\nRelated Papers:\nAchiam, Joshua, et al. \"Constrained policy optimization.\" Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017.\n\nChow, Yinlam, et al. \"Lyapunov-based Safe Policy Optimization for Continuous Control.\" arXiv preprint arXiv:1901.10031 (2019).", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1xKNGqaFH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n[Summary]\nThis paper proposes an approach called conservative policy gradients to stabilize the training of deep policy gradient methods. At fixed intervals, the current policy and a separate target policy are evaluated with a number of rollouts. The target policy is then updated to match the current policy only if the current policy is better than the target policy. Experiments show that the proposed method, when applied to TD3, reduces the variance in performance through the training.\n\n[Decision]\nI am not convinced that the proposed method is sound and indeed useful and I vote for rejecting this paper. Experiments show stable performance. However, this stability comes at the cost of extra computation and interaction with the environment (to evaluate the policies). Claims about the method's stability guarantees and overall performance are not supported by theory and experiments. The submitted paper also needs major improvements in presentation.\n\n[Explanation]\nWhile Proposition 1 provides insights on how the policy evolves, it is too limited to serve as a guarantee. First, performance does not improve or degrade by a constant number. Second, the time it takes for the policy to improve is not captured by the theory. In reality, this time can depend on the hyperparameters or the policy's current performance and might even be unbounded. \n\nI do not understand why a characterization of the performance in the limit of time in Proposition 1 is called a stability guarantee while in the rest of the paper stability refers to consistent improvements in the interim performance. Does stability in this proposition mean that the performance will reach a stationary distribution with bounded support? This property is merely a result of the assumptions that the performance evolves by a constant number at bounded times and that it does not exceed [v_min,v_max].\n\nThe theory studies the stationary distribution of the target policy's performance but the algorithm uses the online policy to interact with the environment. In Algorithm 5, line 8 (Section D in the Appendix) the target policy is only used for bootstrapping. How can a stable target policy result in more stable performance if it is not used to take actions?\n\nThe paper claims that the proposed method results in improvements in stability and overall performance. In Figure 3, the proposed method is more stable than the baseline but the overall performance is not better.\n\nThe proposed method requires more computation and interaction with the environment than the baseline. The experiments do not seem to compare these two methods with the same number of samples or with the same amount of computation. Perhaps the extra computation and samples are better spent on training TD3 for a longer time.\n\nI find Section 2 hard to follow. This section describes Value Iteration, Policy Iteration, DQN, and DDPG in detail (with pseudocode) along with their convergence rates. The message that deep RL algorithms generally lack theoretical guarantees can be conveyed by just describing the linear and deep variants of one method. In fact, the algorithm whose stability is analyzed in the next sections, TD3, is not described in Section 2 or anywhere else in the paper.\n\nLater in Section 3, DDPG and DQN are described as off-policy Deep RL variants of Value Iteration and Policy Iteration. DQN and DDPG are actually built on Q-learning and Deterministic Policy Gradient (DPG).\n\n\n[Minor comments]\nIn the learning curves in Figure 1, what is the measure of performance and how is it estimated? A description of the plotted measure is necessary to show that the drops in the estimated performance are indeed due to policy degradation rather than poor estimation.\n\n--------------------\nAfter rebuttal: I have read the authors' response, the other reviews, and the revision. The revised version has improved presentation, but the proposed method is still introduced as a method with stability guarantees while the proposition in the paper cannot serve as a stability guarantee, and can only provide intuition on the asymptotic performance.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "B1ey8nZ1cB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a simple method for stabilizing the off-policy deep reinforcement learning algorithm, which updates the target network only when the online network performs better than the target network in order to ensure the stability guarantees. More specifically, at every T time steps, they execute both the online network and the target network so as to evaluate the performance of each policy. Then, they update the target network only when the online network outperforms the target network with high probability. The experimental results show that the proposed Conservative-TD3 (C-TD3) is less prone to performance degradation during training.\n\nWhile the stabilizing off-policy reinforcement learning algorithms would be a significant problem, I have some concerns regarding the presentation and the limitation of the proposed method.\n- In p2, 'Value Iteration and Policy Iteration are algorithms for solving tabular RL tasks with convergence guarantee': Basically, exact value iteration and policy iteration are MDP 'planning' algorithms, NOT reinforcement learning algorithms. VI and PI assume 'known' transition and reward dynamics thus there is no need to learn anything, while RL basically assumes the 'unknown' environment thus the agent should learn by doing. \n- Algorithm 1, 2, 3 and 4 are not directly related to the proposed method, thus they can be omitted from the paper. Instead, it would be great to devote more space to the proposed method such as a more detailed theoretical analysis or the pseudo-code of the proposed algorithm.\n- It seems that the performance of the online and target networks is evaluated by Monte-Carlo return which is obtained by executing each policy in the real environment. This requires additional direct interaction with the environment, which can severely hurt the sample complexity of the algorithm. In Figure 4, does the x-axis of C-TD3 reflect these additional samples for evaluating the performance of two policies?\n- The abstract says 'our proposed method reduces the variance of the process and improves the overall performance'. This claim is too strong. If we see Figure 4, in Walker2d, the mean performance of TD3 reaches 4000 at 400k steps, while the performance of C-TD3 is even less than 3000. Similarly in HalfCheetah and Ant, the asymptotic performance of TD3 is higher than C-TD3. It would be great to show the learning curves of TD3 and C-TD3 overlapped.\n- In the experiments, 'discarding failure seeds' cannot be a proper treatment. Instead of discarding the bad results, the reliable algorithm had to be proposed.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new algorithmic approach to reduced variance in off-policy, policy gradient updates.

Multiple reviewers were concerned with both the soundness of the proposed approach, and the cost of using rollouts. In particular, the interaction between the target policy and the behavior policy, and how they are swapped was unclear, where the algorithms in the paper do not match the code provided.

The results show apparent reduction in variance across runs compared with TD3: clear improvements in two domains, minor improvements , and/or an increase in variance in others. In some domains there was decrease in mean performance. The reviewers wanted comparisons with other baseline methods (even in terms of variance across runs).

It is difficult to evaluate the results in this paper, as the performance is averaged over only 5 runs, and runs which result in "failure" are discarded from analysis. The authors explain this was done in the original TD3 code, and one can sympathise in following common practices in the literature. However, the consensus of the reviewers and the AC was that this choice was not well defended, obscures a key difficulty of the learning problem, and makes algorithms look considerably stronger then they actually are. This is particularly confounding in a paper about improving the robustness of learning algorithms. This is not acceptable empirical practice and we strongly encourage the authors to discontinue this.

The reviewers gave nice suggestions including changing the pitch of the paper, and including results in noisy tasks. To reduce the burden of doing more scientific experiments, we suggest the authors start with small or even designed problems to carefully study robustness of learning and the potential improvements due to their algorithm. After this is done in a statistically significant way, it would be natural to move to more demonstration style scaled up results.
Decision: Reject