Title: STABILIZING OFF-POLICY REINFORCEMENT LEARNING WITH CONSERVATIVE POLICY GRADIENTS Anonymous

Abstract: In recent years, advances in deep learning have enabled the application of reinforcement learning algorithms in complex domains. However, they lack the theoretical guarantees which are present in the tabular setting and suffer from many stability and reproducibility problems [10] . In this work, we suggest a simple approach for improving stability in off-policy actor-critic deep reinforcement learning regimes. Experiments on continuous action spaces, in the MuJoCo control suite, show that our proposed method reduces the variance of the process and improves the overall performance across most domains.

1. INTRODUCTION:

Reinforcement Learning (RL) is a dynamical learning paradigm, in which the algorithm (also known as the 'agent') learns through sequential interaction with the environment. On each round, the agent performs an action, which transitions it into a new state, and is provided with a state-dependent reward. The goal of the agent is to maximize the cumulative reward it observes throughout these interactions with the environment. When considering tabular RL, in which there is a finite number of states and actions, there exist efficient algorithms with theoretical convergence and stability guarantees [29; 13; 14; 34; 5; 4] . However, this is not the case when considering deep RL approacheswhen a neural network is used to cope with large state spaces, such as images [19; 11] , and/or large action spaces, for example in continuous control [17; 27] .

The issue with deep RL is twofold: (i) the optimization process is finite, and as such does not follow many of the requirements for the process to converge, such as decaying learning rates [2] , and (ii) the non-linearity of the function approximators (e.g., neural networks) results in a highly non-convex optimization landscape. Due to this, as can be seen in all empirical works [22; 26; 17; 6; 9] , the learning process exhibits high variance; during a short time of training, a near-optimal policy may become arbitrarily bad. As a result, in recent years, the stability of RL algorithms has become a major concern of the research community [10] .

Instead of tackling the stability problems in the optimization process, many previous works focused on different aspects of the loss function [6; 9; 30] . Then, they usually apply a standard variant of gradient descent algorithm on the modified loss function. While these approaches are capable of finding relatively good policies, they are strictly inferior when compared to their tabular counterparts -the tabular approaches ensure convergence and stability, whereas the practical variants exhibit instability and high variance.

In this work, we suggest a novel approach that improves the stability of actor-critic methods. Specifically, we draw the connection between deep RL approaches and their tabular counterparts and highlight the issues inherent in deep RL algorithms. Through this connection, we propose a solution, in the form of a hierarchical training procedure. As opposed to the tabular case, our approach is not ensured to converge, but rather has stability guarantees, i.e., the stationary distribution of the policies is shown to be stable, meaning that with high probability the performance of the policy improves and does not suffer high degradation. We show, empirically across a range of continuous control tasks in MuJoCo, that our approach indeed results in improved behavior both in terms of stability and overall performance.

2. BACKGROUND AND NOTATION:

We consider an infinite-horizon discounted Markov Decision Process [23, MDP] . An MDP is defined as the 5-tuple (S, A, P, r, γ), where S is the state space, A the action space, Algorithm 1 Policy Iteration for s ∈ S do 6:

v π k (s) = r(s, π k (s)) + γ s ∈S P (s, π k (s), s )v π k (s ) 7:

for s ∈ S do Policy Improvement 

Set θ -= θ, φ i -= φ i , t = 0 4: while t < T do 5:

Collect data and append to replay buffer R 6:

t = t + 1 7: {s i , a i , r i , s i+1 } = R.sample()

Training phase 8:

if DDPG then

9: φ = φ -α∇ φ ||r i + γQ φ-(s i+1 , π θ-(s i+1 )) -Q φ (s i , a i )|| 2 2 10:

else TD3

11:

EQUATION

12:

φ 2 = φ 2 -α∇ φ 2 ||r i + γ min i Q φ i -(s i+1 , π θ-(s i+1 )) -Q φ 2 (s i , a i )|| 2 2 13: θ = θ + α∇ θ Q φ 1 -(s i , π θ (s i ))

14:

φ i -= (1 -τ )φ i -+ τ φ i

Update target networks if P (J π θ > J πθ -) ≥ 1 -p then 

θ -= (1 -τ )θ -+ τ θ

Colored lines represent similarities between the classic and modern approach.

P : S × S × A → [0, 1] is a transition kernel, r : S × A → [r min , r max ] is a reward function and γ ∈ (0, 1) is the discount factor.

The goal of an RL agent is to learn a policy π(s, a) that maps states into a distribution over actions. The quality of a policy is measured by its value,

v π (s) = E π [ ∞ t=0 γ t r(s t , a t ) | s 0 = s]

, which is the average cumulative reward when starting from state s and acting according to π. Another important quantity is the Q-function,

Q π (s, a) = E π [ ∞ t=0 γ t r(s t , a t ) | s 0 = s, a 0 = a],

which is strongly connected to the value through the relation v π (s) = a Q π (s, a)π(s, a)d a. We denote an optimal policy that maximizes the value simultaneously across all of the states by π * ∈ arg max π v π and the optimal value function by v * = v π * . When the Q-function and the policy π are represented by a parametric function, we denote them by Q φ and π θ , where φ and θ are the parameters of the function (e.g. the parameters of a deep neural network).

While our goal is to maximize v π , in practice we measure an empirical estimation of E s∼ρ v π (s), where ρ is the initial state distribution. To accommodate for this fact, we define the cumulative reward of a sampled trajectory by J π . This is a random variable, as randomness may occur due to (i) the initial state distribution, (ii) the transition kernel and reward function and (iii) the policy.

2.1. CLASSIC REINFORCEMENT LEARNING ALGORITHMS:

Policy Iteration [12, PI] is an algorithm for solving tabular MDPs with convergence guarantees. PI (Algorithm 1) iterates between two steps: (i) policy evaluation in which the current policy π k is evaluated, producing the value v π k and (ii) policy improvement in which the policy π k+1 is updated greedily w.r.t. v π k . This iterative scheme is proven to converge to the optimal policy. PI schemes have also been extended to the approximate case. Scherrer [25] provides an overview of such approaches, which are called Approximate Policy Iteration (API). In API we consider the scenario in which ||v πk -v π k || ∞ ≤ . This approximation error can be caused due to two factors, the first being the inability to estimate the value and the second being the inability to find the greedy policy, i.e., the arg max at each state. For instance, such errors may occur due to the functional class being used, e.g., neural networks. Most importantly, the sub-optimality of these approaches is bounded by [25] , i.e., the sub-optimality of the final policy is proportional to the approximation error .

||v π∞ -v * || ∞ ≤ (1-γ) 2

2.2. OFF-POLICY DEEP REINFORCEMENT LEARNING FOR CONTINUOUS CONTROL:

DDPG [17] and its latest improvement TD3 [6] use neural networks as function approximators in order to find the optimal policy. Both approaches utilize a target network, which is crucial for the success of the methods. While in their work, this was merely considered as a design parameter, we argue that the importance of these target networks lies in the connection to Policy Iteration. [17, DDPG] (Algorithm 2) aims to directly learn a deterministic policy in a continuous action space. DDPG has both policy evaluation and policy improvement steps. The Q-function, also known as the critic, is trained to estimate the utility of the target policy π θ-, whereas the online policy (the actor) is trained to find the 1-step greedy update w.r.t. this estimation. We argue that DDPG and PI are connected through the target network update procedure. In PI, the previous policy π k is evaluated by v π k , followed by an improvement step in which π k+1 becomes the one-step greedy policy w.r.t. this estimate. DDPG follows a similar scheme, in which the critic Q φ estimates the utility of the policy π θ-, followed by the online policy π θ which is updated w.r.t. this estimate. In this work, we compare to TD3 [6] , an improved version of DDPG.

Deep Deterministic Policy Gradients:

Twin Delayed DDPG [6, TD3] improves the DDPG algorithm by tackling the problem of overestimation. Similarly to how Double-DQN [32] overcame the overestimation problem by introducing an additional Q function, in TD3 they introduce an additional critic. They show, both theoretically and empirically, that by bootstrapping the minimal value between both estimators min i Q i (s, π(s)) they are able to overcome the problem of overestimation.

3. INSTABILITY IN POLICY-BASED DEEP REINFORCEMENT LEARNING:

In the previous section, we provided an overview of PI, which has convergence guarantees, and two off-policy DRL variants. However, since in practical applications the policy is represented using a non-linear neural network, a small change in the parameters θ may result in a very large change in the output. We illustrate this issue in Figure 1 , where a state of the art off-policy approach learns to control the Hopper and InvertedPendulum robots.

In these figures, we show the individual learning curves over several seeds, without performing temporal smoothing1 , whereas the bold line represents the mean performance across runs. At each evaluation period, we evaluate the performance of the agent across 100 episodes. These results paint a clear picture -while the agent is capable of learning a policy which attains a score of approximately 3500 in Hopper and the optimal score of 1000 in InvertedPendulum, it is highly unstable, i.e., a policy which was near-optimal at time t might become arbitrarily bad at time t + 1. 

4. CONSERVATIVE POLICY GRADIENTS:

While there exists a connection between the tabular approach and that which is used with non-linear function approximators, due to the non-convex optimization landscape, a small gradient step can lead to an arbitrarily bad solution. As such, current policy-based methods with non-linear function approximation (neural networks) behave poorly when compared to their tabular counterparts.

We propose a simple, conservative, yet theoretically grounded approach for tackling these issues. Taking inspiration from the classic approaches, we observe that the gradient-based algorithms lack the improvement guarantees. To overcome this issue, we propose a hierarchical learning procedure: while the online network is trained as before (as seen in Algorithm 2), the target network is updated periodically if, with high probability, the online policy is better.

In simple terms, our proposal is as follows: every T time-steps, evaluate both the target network2 and the online network. If with confidence of 1 -δ the online network is better than the target, perform the switch θ -= θ. In addition, between updates, the target network is kept fixed, as opposed to the standard approach [17; 6] which performs Polyak averaging [21] . We provide additional discussion in Appendix D.

The following proposition motivates this approach. We discretize the range [v min , v max ] into bins of size ∆ and formulate the learning process as a random-walk, in which with probability 1 -δ, the policy improves (1 step to the right) and decreases otherwise. The random-walk is bounded at both ends by v min and v max , such that the value is projected back into this range. Proposition 1. Let v π t be a random walk admitting values in the set V = {v min , v min + ∆, . . . , v max -∆, v max }. Also, assume that w.p. δ: v π t+1 = min{v π t + ∆, v max } and v π t+1 = max{v π t -∆, v min } otherwise. Then, the stationary distribution v of v π t equals

P(v = x) = r x 1-r 1-r (vmax -v min )/∆+1 for any x ∈ V , where r = δ 1-δ .

Proof Sketch. We can write the recurrence relations by:

P(v = v min ) = (1 -δ)P(v = v min ) + δP(v = v min + ∆) P(v = x) = (1 -δ)P(v = x -∆) + δP(v = x + ∆) P(v = v max ) = (1 -δ)P(v = v max -∆) + δP(v = v max )

the stationary distribution follows.

An illustration of this process is provided in Appendix A. When the probability of improvement is 50%, i.e., we are unable to determine which policy is better, the stationary distribution over the value may be arbitrarily bad. However, as the confidence in the improvement of the policy increases, the distribution shifts towards a delta function at the optimal value function.

Remark 1. In practice, the optimal attainable policy is a function of the initial starting parameters. Due to the non-convexity of the optimization landscape, not all initializations are ensured to be capable of attaining an optimal policy. 

5. EXPERIMENTS:

Implementing our approach in an off-policy actor-critic scheme is straightforward. While current approaches [17; 6] hold two policies, an online and a target policy, such that the target policy is updated each step using a Polyak averaging procedure [21] -our approach, illustrated in Fig. 2 , requires the target policy to be updated only when the online policy is superior, with high probability.

Since in our framework, the online policy is aimed at finding the 1-step greedy policy, and due to the continuous nature of the action space, we add a 'trust region' element to the loss λ||π θ (s

)-π θ-(s)|| 2 2

such that the online policy is kept close to the target. In addition, we found that collecting data interchangeably from the online and the target network leads to a further improvement in stability.

For all the experiments, we plot 5 different random seeds for each configuration, discarding failure seeds 3 . At each evaluation step, each seed is evaluated for 100 episodes, and the average performance is presented. We plot both the individual seeds (light curves) and the average results across seeds (dark curves). As opposed to previous works, we do not smooth the graphs using a moving average procedure, which enables a better analysis of the experiments' variance and stability.

5.1. EVALUATING THE POLICIES:

Our approach requires determining whether or not an online policy is better, with probability 1 -δ.

Assuming J π , the performance of a sampled trajectory for any deterministic policy π, is normally distributed, we may consider the t-test for evaluating improvement. We formulate the null hypothesis as

µ π θ = E ρ J π θ ≤ E ρ J π θ -= µ π θ-.

In other words, we hypothesize that the online policy has not improved over the target policy, thus we use the Welch's one-tailed t-test [33] . The t-statistic is then defined as t = μπ θ -μπ θ -/ σ2 µ θ /n π θ + σ2 µ θ - /n π θ -, where μπ is the empirical estimate of E ρ J π , σπ is its standard deviation and n π the number of sampled trajectories. The tstatistic enables us to find the p-value, which represents the confidence in the null hypothesis. Under Gaussian assumption, the p-value can be directly calculated by p = 1 -Φ(t), where Φ(x) is the cumulative distribution function of a standard normal random variable. Hence, when p < 1 -δ we have a high enough confidence that the null hypothesis is incorrect -thus the new policy is better, in which case we update the target policy.

5.2. PARAMETER SELECTION:

For this approach, there are three parameters which require tuning. (i) how often we evaluate the online policy, (ii) how many episodes are used for evaluation and (iii) the minimal confidence required for policy improvement. Notice that there is a connection between the confidence and the number of evaluation episodes; as the number of evaluations grows, the confidence itself increases. Hence, when there is a relatively high variance in the evaluations, a low number of samples will result in a low confidence, even when the empirical mean is higher. On the other hand, the evaluation frequency controls how fast the process converges. We evaluate the effect of these parameters and provide comprehensive results in Appendix B.

Confidence Value δ::

The confidence level is 1 -δ, i.e., P(E ρ J π θ > E ρ J π θ -) ≥ 1 -δ. While a natural approach would be to attempt a high level of confidence, our empirical results show otherwise. Demanding a confidence which is too high, makes it much harder to find an improving policy, resulting in a stable yet strictly sub-optimal policy. On the other hand, when very low, the approach becomes unstable. We found that δ ∈ [0.1, 0.2] work well in practice.

Evaluation Episodes K: The number of evaluations is correlated to the confidence. A higher number of evaluations increases the confidence in the empirical mean, enabling the algorithm to determine that a policy is better even when the difference is small. However, this comes at a costevaluating the policy takes time and a multitude of evaluations may drastically lengthen training.

Evaluation Frequency N : When comparing evaluation frequency, we see that when the evaluation is performed every 10, 000 steps, it requires a high probability of improvement to successfully converge. However, when the evaluation is performed more often, the process behaves well even for a lower improvement probability (fewer evaluation episodes).

Based on these insights and the results in Appendix B, we opted to run with a minimal confidence for swapping of 90%, evaluation every 1, 000 steps over 10 episodes (sampled trajectories).

5.3. RESULTS:



TD3 C-TD3:

Figure 3 : Evaluation without (TD3) and with the Conservative (C-TD3) update rule. X-axis denotes the number of gradient steps taken.

We compared our approach to the baseline -TD3 without our policy selection scheme. The results are presented in Figure 3 . These results show a high resemblance to the theoretical model, showing that, across all domains, the conservative approach (Conservative-TD3) reduces the variance of individual seeds, also seen in Table 1 , and results in increased stability across most domains.

InvertedPendulum and InvertedDoublePendulum serve as perfect examples. While TD3 is capable of finding an optimal policy -e.g., the maximal score -subsequent policies may be arbitrarily bad. By adding the hierarchical selection scheme, our approach is capable of reducing this behavior. That is, once an optimal policy is found, it is not likely to be replaced with an inferior one.

An interesting result can be seen in the Walker2d domain. While some seeds attain near-optimal performance, others converge to a sub-optimal policy. As our approach searches for a high-confidence improvement, it may sometimes get stuck. However, the inner-process variance, i.e., the variance of individual seeds during training, is much lower across all domains.

Finally, we observe sub-optimal behavior in the Ant domain, both in terms of individual seeds and average performance across seeds. The parameters (confidence, number of evaluation episodes and evaluation frequency) were selected based on the Hopper domain, however it is important to note that these parameters are not optimal for each domain.

Figure 4 presents the standard deviation (STD) of both the online (Figure 4a ) and target (Figure 4b ) networks, and the difference between the empirical means |μ π -μπ-| (Figure 4c ). This clearly shows that in Ant, there is a very large variance, compared the difference between the means, which results in a low confidence. To overcome this issue, we provide an additional experiment, for the Ant domain based on these insights (Figure 4d ), which considered a minimal confidence of 20% (90% in Figure 3 ). These results show that while the parameters are transferable across domains, the optimal parameters found on one domain are not necessarily optimal for another. To analyze the stability of each algorithm (TD3 vs C-TD3), we consider the variance of the policy during the last 100 episodes of training. In Table 1 we present the maximum of the standard deviation in different seeds, where per-seed STD is calculated over the 100 last episodes of training.

5.3.1. STABILITY ANALYSIS:

Our empirical results match the theoretical behavior. Indeed, the inner-seed variance is dramatically reduced when using the conservative update rule, as opposed to the non-conservative TD3.

5.4. POTENTIAL EXTENSIONS:

In this work, we presented an approach for providing probabilistic improvement guarantees. The performance of a trajectory is a random variable, as the initial state, the transition kernel and sometimes the policy -may all be stochastic. To determine which policy π -or π (the target or online) is better, we performed N evaluations, assumed that the samples are normally distributed and performed a t-test. Below we present several intuitive extensions to our work:

Unknown distribution: While the Gaussian assumption worked well in practice, it may not always be correct. In such a scenario, we see two possible routes: (i) known range, in which the minimal and maximal possible values are known, where one may use the Empirical Bernstein [18] , or (ii) when the range is unknown, one may consider a bootstrap-based approach for estimating the confidence intervals around the mean [1] .

Upper/Lower Confidence Bounds: Another approach is to consider improvement by a minimal margin. For instance, we may set the update rule to P(v π -≥ v π-) ≥ 1 -δ, so that the statistical test ensures that with high confidence, the online policy has improved by at least over the target.

Adaptive Sampling: In our approach, we considered a rather small number of evaluations, and thus resorted to an equal number of evaluations of each policy -up to 100 evaluations for each policy. However, when provided with a larger budget, it is of great interest to combine adaptive sampling procedures. Gabillon et al. [7] propose such an approach, which takes into account the empirical mean and variance of each policy evaluation in order to determine which policy to evaluate. At the end of this process, the policy with the highest empirical mean is returned.

In our initial experiments, we tested the UCB/LCB approach. However, we found that is domainspecific. While in some domains the attained rewards are relatively high, and the policies improve greatly between iterations, in others the improvement is rather small. This led to an early convergence to sub-optimal policies. Nevertheless, this approach may prove beneficial in certain settings and should be considered. Additionally, we tested policy swap based on the empirical estimate of the mean, without ensuring a proper confidence. While this approach worked relatively well in practice, as it lacks proper confidence bounds, we decided to focus on the t-test based approach.

6. RELATED WORK:

Trust Region Optimization: Recent works in on-policy RL have also taken inspiration from Conservative Policy Iteration [15, CPI] . Schulman et al. [26] formulated learning through a trust region optimization scheme, which was later optimized in Schulman et al. [27] . These approaches bound the KL-divergence between the policy before and after the update. While TRPO greatly improved the stability compared to previous policy gradient approaches, such as A2C [20] , it lacks the theoretical guarantees.

Safe Policy Improvement: Given a baseline policy, safe policy improvement (SPI) algorithms aim to find a new policy that is better than the baseline with high probability [31; 8; 16] . Although this may seem similar to our update stage, there are two major differences: (i) most SPI algorithms directly calculate the improving policy, which is much less efficient than continuously updating the policy and performing the switch when improvement is achieved, and (ii) due to safety constraints, SPI algorithms usually perform off-policy evaluation for the suggested policy, which is still an open problem in high dimensional problems. In contrast, we evaluate it directly and thus enjoy a better sample complexity.

Evolution Strategies: ES [24; 28; 3] is a gradient-free approach for solving reinforcement learning tasks. At each iteration, given the previous policy π θi , a set of augmented policies is created {π θi+ξi } i and evaluated, where ξ i is some random noise. The ES update rule can be seen as a form of numerical gradient estimation (i.e., finite differences) [35] . There are some similarities between our approach and ES. While ES is a gradient-free method, it uses empirical evaluations of the policies to determine the parameter update direction. Our approach uses these evaluations in order to determine whether or not the policy has improved, and in contrast to ES, will only change the policy if there is a high enough confidence for improvement.

7. CONCLUSIONS:

In this work, we tackled the stability problems of policy-based reinforcement learning approaches. Even though the empirical gradient (SGD) is a noisy estimate of the ideal descent direction, and due to the structure of the optimization landscape and the large step sizes used in practice, the policy's performance varies greatly during training.

Drawing from the connection between actor-critic approaches and approximate policy iteration schemes, we introduced a hierarchical training scheme, which we analyzed by comparing this approach to a bounded random-walk. While this scheme is not an exact representation of the optimization process, it provides an intuition on the behavior of such a probabilistic improvement scheme, i.e., as the probability of improvement increases, the stationary distribution over the performance of the policy shifts towards the optimal solution.

Finally, we proposed the Conservative-TD3, which performs a periodic evaluation of both the online and target policies such that it updates the target network only when there is a higher enough confidence that the online network has surpassed its performance. We validate this on several continuous control tasks in MuJoCo; empirical evidence shows that indeed this approach behaves similarly to the theoretical analysis. When compared to the baseline, we observe a dramatic reduction in variance, increased stability and an overall improvement in the performance itself.

B PARAMETER ABLATION TESTS:

In this section, we present the simulation results for all possible parameter combinations, in the Hopper domain. Specifically, we tested minimal required improvement probability of [0.5, 0.8, 0.9, 0.95], number of evaluations per evaluation steps of [5, 10, 100] and evaluation frequency of [1000, 10000]. The graphs above show the effect of tuning the evaluation frequency, the number of evaluations and the minimal required confidence. As the evaluation becomes more frequent, the agent learns faster. As our approach requires the critic to evaluate the target policy π -, the online policy π aims to find the 1-step greedy policy. This may happen rather quickly, thus a more frequent evaluation scheme will improve faster. On the other hand, the number of evaluation episodes is correlated to the confidence in improvement, i.e., the confidence we have that π is indeed 'better' than π -. We see that as the number of evaluations rises, the agent is capable of finding good solutions even when a higher confidence is required. However, a lower confidence results in a noisier training regime and a small number of evaluations leads to a low confidence -hence high confidence requirements with a low number of evaluations often leads to the inability to converge to a good solution.

C ONLINE VS. TARGET NETWORKS TD3 Actor TD3 Target Figure 12 : On the left is the evaluation of the online network, whereas on the right we present the performance curves of the target network. It is clear across all domains that the target network exhibits a dramatic reduction in variance.

The common approach is to present the performance of the online network. In these graphs, we show that, as opposed to the common belief, the target network suffers less from instability and offers improved performance. These results are intuitive once the connection to policy iteration schemes is made.

:

Temporal smoothing is a common practice in empirical studies, in which each data-point in the graph represents a moving average over the past N evaluations. While this makes the graphs easier to read, it hides a lot of important information such as the variance of each seed during training.

The target network needs to be re-evaluated in order to ensure an unbiased estimate of its performance.

We observed that some seeds in deep RL result in utter failure, for instance, a score of

in Hopper. This issue is relatively rare, and also happens in the baselines, but hinders the proper evaluation of the policy.

Figures and Tables:

Figure 1: Performance plots of the TD3 [6] algorithm.

Figure 2: Conservative Policy Gradients.

Figure 4: Failure analysis of the Ant-v2 domain.

Figure 6: C-TD3, parameter ablation. Comparison is performed every 1,000 steps over 5 episodes with swap confidence of 1 -δ = 95, 90, 80 and 50.

Figure 7: C-TD3, parameter ablation. Comparison is performed every 1,000 steps over 10 episodes with swap confidence of 1 -δ = 95, 90, 80 and 50.

Figure 8: C-TD3, parameter ablation. Comparison is performed every 1,000 steps over 100 episodes with swap confidence of 1 -δ = 95, 90, 80 and 50.

Figure 9: C-TD3, parameter ablation. Comparison is performed every 10,000 steps over 5 episodes with swap confidence of 1 -δ = 95, 90, 80 and 50.

Figure 10: C-TD3, parameter ablation. Comparison is performed every 10,000 steps over 10 episodes with swap confidence of 1 -δ = 95, 90, 80 and 50.

Figure 11: C-TD3, parameter ablation. Comparison is performed every 10,000 steps over 100 episodes with swap confidence of 1 -δ = 95, 90, 80 and 50.

Algorithm 2 DDPG, TD3 and our Conservative update 1: Input: algorithm, learning rate α, averaging parameter τ , steps T and exploration N . 2: Initialize policy θ, critics φ 1 , φ 2 and replay buffer R 3:

Table content: <table><tr><td>9:</td><td>k = k + 1</td></tr><tr><td colspan="2">10: return policy π k</td></tr></table>

Process STD calculated over the last 100 training episodes.

Table content: <table><tr><td/><td colspan="6">Hopper InvertedPendulum InvertedDoublePendulum Walker2d HalfCheetah Ant</td></tr><tr><td>C-TD3</td><td>36</td><td>0</td><td>6</td><td>99</td><td>81</td><td>57</td></tr><tr><td>TD3</td><td>799</td><td>139</td><td>2409</td><td>468</td><td>350</td><td>902</td></tr></table>

Peer review (iclr2020):
[Summary] This paper proposes an approach called conservative policy gradients to stabilize the training of deep policy gradient methods. At fixed intervals, the current policy and a separate target policy are evaluated with a number of rollouts. The target policy is then updated to match the current policy only if the current policy is better than the target policy. Experiments show that the proposed method, when applied to TD3, reduces the variance in performance through the training. [Decision] I am not convinced that the proposed method is sound and indeed useful and I vote for rejecting this paper. Experiments show stable performance. However, this stability comes at the cost of extra computation and interaction with the environment (to evaluate the policies). Claims about the method*s stability guarantees and overall performance are not supported by theory and experiments. The submitted paper also needs major improvements in presentation. [Explanation] While Proposition 1 provides insights on how the policy evolves, it is too limited to serve as a guarantee. First, performance does not improve or degrade by a constant number. Second, the time it takes for the policy to improve is not captured by the theory. In reality, this time can depend on the hyperparameters or the policy*s current performance and might even be unbounded. I do not understand why a characterization of the performance in the limit of time in Proposition 1 is called a stability guarantee while in the rest of the paper stability refers to consistent improvements in the interim performance. Does stability in this proposition mean that the performance will reach a stationary distribution with bounded support? This property is merely a result of the assumptions that the performance evolves by a constant number at bounded times and that it does not exceed [v_min,v_max]. The theory studies the stationary distribution of the target policy*s performance but the algorithm uses the online policy to interact with the environment. In Algorithm 5, line 8 (Section D in the Appendix) the target policy is only used for bootstrapping. How can a stable target policy result in more stable performance if it is not used to take actions? The paper claims that the proposed method results in improvements in stability and overall performance. In Figure 3, the proposed method is more stable than the baseline but the overall performance is not better. The proposed method requires more computation and interaction with the environment than the baseline. The experiments do not seem to compare these two methods with the same number of samples or with the same amount of computation. Perhaps the extra computation and samples are better spent on training TD3 for a longer time. I find Section 2 hard to follow. This section describes Value Iteration, Policy Iteration, DQN, and DDPG in detail (with pseudocode) along with their convergence rates. The message that deep RL algorithms generally lack theoretical guarantees can be conveyed by just describing the linear and deep variants of one method. In fact, the algorithm whose stability is analyzed in the next sections, TD3, is not described in Section 2 or anywhere else in the paper. Later in Section 3, DDPG and DQN are described as off-policy Deep RL variants of Value Iteration and Policy Iteration. DQN and DDPG are actually built on Q-learning and Deterministic Policy Gradient (DPG). [Minor comments] In the learning curves in Figure 1, what is the measure of performance and how is it estimated? A description of the plotted measure is necessary to show that the drops in the estimated performance are indeed due to policy degradation rather than poor estimation. -------------------- After rebuttal: I have read the authors* response, the other reviews, and the revision. The revised version has improved presentation, but the proposed method is still introduced as a method with stability guarantees while the proposition in the paper cannot serve as a stability guarantee, and can only provide intuition on the asymptotic performance.