Title: STABILIZING OFF-POLICY REINFORCEMENT LEARNING WITH CONSERVATIVE POLICY GRADIENTS Anonymous

Abstract: In recent years, advances in deep learning have enabled the application of reinforcement learning algorithms in complex domains. However, they lack the theoretical guarantees which are present in the tabular setting and suffer from many stability and reproducibility problems [10] . In this work, we suggest a simple approach for improving stability in off-policy actor-critic deep reinforcement learning regimes. Experiments on continuous action spaces, in the MuJoCo control suite, show that our proposed method reduces the variance of the process and improves the overall performance across most domains.

1. INTRODUCTION:

Reinforcement Learning (RL) is a dynamical learning paradigm, in which the algorithm (also known as the 'agent') learns through sequential interaction with the environment. On each round, the agent performs an action, which transitions it into a new state, and is provided with a state-dependent reward. The goal of the agent is to maximize the cumulative reward it observes throughout these interactions with the environment. When considering tabular RL, in which there is a finite number of states and actions, there exist efficient algorithms with theoretical convergence and stability guarantees [29; 13; 14; 34; 5; 4] . However, this is not the case when considering deep RL approacheswhen a neural network is used to cope with large state spaces, such as images [19; 11] , and/or large action spaces, for example in continuous control [17; 27] .

The issue with deep RL is twofold: (i) the optimization process is finite, and as such does not follow many of the requirements for the process to converge, such as decaying learning rates [2] , and (ii) the non-linearity of the function approximators (e.g., neural networks) results in a highly non-convex optimization landscape. Due to this, as can be seen in all empirical works [22; 26; 17; 6; 9] , the learning process exhibits high variance; during a short time of training, a near-optimal policy may become arbitrarily bad. As a result, in recent years, the stability of RL algorithms has become a major concern of the research community [10] .

Instead of tackling the stability problems in the optimization process, many previous works focused on different aspects of the loss function [6; 9; 30] . Then, they usually apply a standard variant of gradient descent algorithm on the modified loss function. While these approaches are capable of finding relatively good policies, they are strictly inferior when compared to their tabular counterparts -the tabular approaches ensure convergence and stability, whereas the practical variants exhibit instability and high variance.

In this work, we suggest a novel approach that improves the stability of actor-critic methods. Specifically, we draw the connection between deep RL approaches and their tabular counterparts and highlight the issues inherent in deep RL algorithms. Through this connection, we propose a solution, in the form of a hierarchical training procedure. As opposed to the tabular case, our approach is not ensured to converge, but rather has stability guarantees, i.e., the stationary distribution of the policies is shown to be stable, meaning that with high probability the performance of the policy improves and does not suffer high degradation. We show, empirically across a range of continuous control tasks in MuJoCo, that our approach indeed results in improved behavior both in terms of stability and overall performance.

2. BACKGROUND AND NOTATION:

We consider an infinite-horizon discounted Markov Decision Process [23, MDP] . An MDP is defined as the 5-tuple (S, A, P, r, γ), where S is the state space, A the action space, Algorithm 1 Policy Iteration for s ∈ S do 6:

v π k (s) = r(s, π k (s)) + γ s ∈S P (s, π k (s), s )v π k (s ) 7:

for s ∈ S do Policy Improvement 

Set θ -= θ, φ i -= φ i , t = 0 4: while t < T do 5:

Collect data and append to replay buffer R 6:

t = t + 1 7: {s i , a i , r i , s i+1 } = R.sample()

Training phase 8:

if DDPG then

9: φ = φ -α∇ φ ||r i + γQ φ-(s i+1 , π θ-(s i+1 )) -Q φ (s i , a i )|| 2 2 10:

else TD3

11:

EQUATION

12:

φ 2 = φ 2 -α∇ φ 2 ||r i + γ min i Q φ i -(s i+1 , π θ-(s i+1 )) -Q φ 2 (s i , a i )|| 2 2 13: θ = θ + α∇ θ Q φ 1 -(s i , π θ (s i ))

14:

φ i -= (1 -τ )φ i -+ τ φ i

Update target networks if P (J π θ > J πθ -) ≥ 1 -p then 

θ -= (1 -τ )θ -+ τ θ

Colored lines represent similarities between the classic and modern approach.

P : S × S × A → [0, 1] is a transition kernel, r : S × A → [r min , r max ] is a reward function and γ ∈ (0, 1) is the discount factor.

The goal of an RL agent is to learn a policy π(s, a) that maps states into a distribution over actions. The quality of a policy is measured by its value,

v π (s) = E π [ ∞ t=0 γ t r(s t , a t ) | s 0 = s]

, which is the average cumulative reward when starting from state s and acting according to π. Another important quantity is the Q-function,

Q π (s, a) = E π [ ∞ t=0 γ t r(s t , a t ) | s 0 = s, a 0 = a],

which is strongly connected to the value through the relation v π (s) = a Q π (s, a)π(s, a)d a. We denote an optimal policy that maximizes the value simultaneously across all of the states by π * ∈ arg max π v π and the optimal value function by v * = v π * . When the Q-function and the policy π are represented by a parametric function, we denote them by Q φ and π θ , where φ and θ are the parameters of the function (e.g. the parameters of a deep neural network).

While our goal is to maximize v π , in practice we measure an empirical estimation of E s∼ρ v π (s), where ρ is the initial state distribution. To accommodate for this fact, we define the cumulative reward of a sampled trajectory by J π . This is a random variable, as randomness may occur due to (i) the initial state distribution, (ii) the transition kernel and reward function and (iii) the policy.

2.1. CLASSIC REINFORCEMENT LEARNING ALGORITHMS:

Policy Iteration [12, PI] is an algorithm for solving tabular MDPs with convergence guarantees. PI (Algorithm 1) iterates between two steps: (i) policy evaluation in which the current policy π k is evaluated, producing the value v π k and (ii) policy improvement in which the policy π k+1 is updated greedily w.r.t. v π k . This iterative scheme is proven to converge to the optimal policy. PI schemes have also been extended to the approximate case. Scherrer [25] provides an overview of such approaches, which are called Approximate Policy Iteration (API). In API we consider the scenario in which ||v πk -v π k || ∞ ≤ . This approximation error can be caused due to two factors, the first being the inability to estimate the value and the second being the inability to find the greedy policy, i.e., the arg max at each state. For instance, such errors may occur due to the functional class being used, e.g., neural networks. Most importantly, the sub-optimality of these approaches is bounded by [25] , i.e., the sub-optimality of the final policy is proportional to the approximation error .

||v π∞ -v * || ∞ ≤ (1-γ) 2

2.2. OFF-POLICY DEEP REINFORCEMENT LEARNING FOR CONTINUOUS CONTROL:

DDPG [17] and its latest improvement TD3 [6] use neural networks as function approximators in order to find the optimal policy. Both approaches utilize a target network, which is crucial for the success of the methods. While in their work, this was merely considered as a design parameter, we argue that the importance of these target networks lies in the connection to Policy Iteration. [17, DDPG] (Algorithm 2) aims to directly learn a deterministic policy in a continuous action space. DDPG has both policy evaluation and policy improvement steps. The Q-function, also known as the critic, is trained to estimate the utility of the target policy π θ-, whereas the online policy (the actor) is trained to find the 1-step greedy update w.r.t. this estimation. We argue that DDPG and PI are connected through the target network update procedure. In PI, the previous policy π k is evaluated by v π k , followed by an improvement step in which π k+1 becomes the one-step greedy policy w.r.t. this estimate. DDPG follows a similar scheme, in which the critic Q φ estimates the utility of the policy π θ-, followed by the online policy π θ which is updated w.r.t. this estimate. In this work, we compare to TD3 [6] , an improved version of DDPG.

Deep Deterministic Policy Gradients:

Twin Delayed DDPG [6, TD3] improves the DDPG algorithm by tackling the problem of overestimation. Similarly to how Double-DQN [32] overcame the overestimation problem by introducing an additional Q function, in TD3 they introduce an additional critic. They show, both theoretically and empirically, that by bootstrapping the minimal value between both estimators min i Q i (s, π(s)) they are able to overcome the problem of overestimation.

3. INSTABILITY IN POLICY-BASED DEEP REINFORCEMENT LEARNING:

In the previous section, we provided an overview of PI, which has convergence guarantees, and two off-policy DRL variants. However, since in practical applications the policy is represented using a non-linear neural network, a small change in the parameters θ may result in a very large change in the output. We illustrate this issue in Figure 1 , where a state of the art off-policy approach learns to control the Hopper and InvertedPendulum robots.

In these figures, we show the individual learning curves over several seeds, without performing temporal smoothing1 , whereas the bold line represents the mean performance across runs. At each evaluation period, we evaluate the performance of the agent across 100 episodes. These results paint a clear picture -while the agent is capable of learning a policy which attains a score of approximately 3500 in Hopper and the optimal score of 1000 in InvertedPendulum, it is highly unstable, i.e., a policy which was near-optimal at time t might become arbitrarily bad at time t + 1. 

4. CONSERVATIVE POLICY GRADIENTS:

While there exists a connection between the tabular approach and that which is used with non-linear function approximators, due to the non-convex optimization landscape, a small gradient step can lead to an arbitrarily bad solution. As such, current policy-based methods with non-linear function approximation (neural networks) behave poorly when compared to their tabular counterparts.

We propose a simple, conservative, yet theoretically grounded approach for tackling these issues. Taking inspiration from the classic approaches, we observe that the gradient-based algorithms lack the improvement guarantees. To overcome this issue, we propose a hierarchical learning procedure: while the online network is trained as before (as seen in Algorithm 2), the target network is updated periodically if, with high probability, the online policy is better.

In simple terms, our proposal is as follows: every T time-steps, evaluate both the target network2 and the online network. If with confidence of 1 -δ the online network is better than the target, perform the switch θ -= θ. In addition, between updates, the target network is kept fixed, as opposed to the standard approach [17; 6] which performs Polyak averaging [21] . We provide additional discussion in Appendix D.

The following proposition motivates this approach. We discretize the range [v min , v max ] into bins of size ∆ and formulate the learning process as a random-walk, in which with probability 1 -δ, the policy improves (1 step to the right) and decreases otherwise. The random-walk is bounded at both ends by v min and v max , such that the value is projected back into this range. Proposition 1. Let v π t be a random walk admitting values in the set V = {v min , v min + ∆, . . . , v max -∆, v max }. Also, assume that w.p. δ: v π t+1 = min{v π t + ∆, v max } and v π t+1 = max{v π t -∆, v min } otherwise. Then, the stationary distribution v of v π t equals

P(v = x) = r x 1-r 1-r (vmax -v min )/∆+1 for any x ∈ V , where r = δ 1-δ .

Proof Sketch. We can write the recurrence relations by:

P(v = v min ) = (1 -δ)P(v = v min ) + δP(v = v min + ∆) P(v = x) = (1 -δ)P(v = x -∆) + δP(v = x + ∆) P(v = v max ) = (1 -δ)P(v = v max -∆) + δP(v = v max )

the stationary distribution follows.

An illustration of this process is provided in Appendix A. When the probability of improvement is 50%, i.e., we are unable to determine which policy is better, the stationary distribution over the value may be arbitrarily bad. However, as the confidence in the improvement of the policy increases, the distribution shifts towards a delta function at the optimal value function.

Remark 1. In practice, the optimal attainable policy is a function of the initial starting parameters. Due to the non-convexity of the optimization landscape, not all initializations are ensured to be capable of attaining an optimal policy. 

5. EXPERIMENTS:

Implementing our approach in an off-policy actor-critic scheme is straightforward. While current approaches [17; 6] hold two policies, an online and a target policy, such that the target policy is updated each step using a Polyak averaging procedure [21] -our approach, illustrated in Fig. 2 , requires the target policy to be updated only when the online policy is superior, with high probability.

Since in our framework, the online policy is aimed at finding the 1-step greedy policy, and due to the continuous nature of the action space, we add a 'trust region' element to the loss λ||π θ (s

)-π θ-(s)|| 2 2

such that the online policy is kept close to the target. In addition, we found that collecting data interchangeably from the online and the target network leads to a further improvement in stability.

For all the experiments, we plot 5 different random seeds for each configuration, discarding failure seeds 3 . At each evaluation step, each seed is evaluated for 100 episodes, and the average performance is presented. We plot both the individual seeds (light curves) and the average results across seeds (dark curves). As opposed to previous works, we do not smooth the graphs using a moving average procedure, which enables a better analysis of the experiments' variance and stability.

5.1. EVALUATING THE POLICIES:

Our approach requires determining whether or not an online policy is better, with probability 1 -δ.

Assuming J π , the performance of a sampled trajectory for any deterministic policy π, is normally distributed, we may consider the t-test for evaluating improvement. We formulate the null hypothesis as

µ π θ = E ρ J π θ ≤ E ρ J π θ -= µ π θ-.

In other words, we hypothesize that the online policy has not improved over the target policy, thus we use the Welch's one-tailed t-test [33] . The t-statistic is then defined as t = μπ θ -μπ θ -/ σ2 µ θ /n π θ + σ2 µ θ - /n π θ -, where μπ is the empirical estimate of E ρ J π , σπ is its standard deviation and n π the number of sampled trajectories. The tstatistic enables us to find the p-value, which represents the confidence in the null hypothesis. Under Gaussian assumption, the p-value can be directly calculated by p = 1 -Φ(t), where Φ(x) is the cumulative distribution function of a standard normal random variable. Hence, when p < 1 -δ we have a high enough confidence that the null hypothesis is incorrect -thus the new policy is better, in which case we update the target policy.

5.2. PARAMETER SELECTION:

For this approach, there are three parameters which require tuning. (i) how often we evaluate the online policy, (ii) how many episodes are used for evaluation and (iii) the minimal confidence required for policy improvement. Notice that there is a connection between the confidence and the number of evaluation episodes; as the number of evaluations grows, the confidence itself increases. Hence, when there is a relatively high variance in the evaluations, a low number of samples will result in a low confidence, even when the empirical mean is higher. On the other hand, the evaluation frequency controls how fast the process converges. We evaluate the effect of these parameters and provide comprehensive results in Appendix B.

Confidence Value δ::

The confidence level is 1 -δ, i.e., P(E ρ J π θ > E ρ J π θ -) ≥ 1 -δ. While a natural approach would be to attempt a high level of confidence, our empirical results show otherwise. Demanding a confidence which is too high, makes it much harder to find an improving policy, resulting in a stable yet strictly sub-optimal policy. On the other hand, when very low, the approach becomes unstable. We found that δ ∈ [0.1, 0.2] work well in practice.

Evaluation Episodes K: The number of evaluations is correlated to the confidence. A higher number of evaluations increases the confidence in the empirical mean, enabling the algorithm to determine that a policy is better even when the difference is small. However, this comes at a costevaluating the policy takes time and a multitude of evaluations may drastically lengthen training.

Evaluation Frequency N : When comparing evaluation frequency, we see that when the evaluation is performed every 10, 000 steps, it requires a high probability of improvement to successfully converge. However, when the evaluation is performed more often, the process behaves well even for a lower improvement probability (fewer evaluation episodes).

Based on these insights and the results in Appendix B, we opted to run with a minimal confidence for swapping of 90%, evaluation every 1, 000 steps over 10 episodes (sampled trajectories).

5.3. RESULTS:



TD3 C-TD3:

Figure 3 : Evaluation without (TD3) and with the Conservative (C-TD3) update rule. X-axis denotes the number of gradient steps taken.

We compared our approach to the baseline -TD3 without our policy selection scheme. The results are presented in Figure 3 . These results show a high resemblance to the theoretical model, showing that, across all domains, the conservative approach (Conservative-TD3) reduces the variance of individual seeds, also seen in Table 1 , and results in increased stability across most domains.

InvertedPendulum and InvertedDoublePendulum serve as perfect examples. While TD3 is capable of finding an optimal policy -e.g., the maximal score -subsequent policies may be arbitrarily bad. By adding the hierarchical selection scheme, our approach is capable of reducing this behavior. That is, once an optimal policy is found, it is not likely to be replaced with an inferior one.

An interesting result can be seen in the Walker2d domain. While some seeds attain near-optimal performance, others converge to a sub-optimal policy. As our approach searches for a high-confidence improvement, it may sometimes get stuck. However, the inner-process variance, i.e., the variance of individual seeds during training, is much lower across all domains.

Finally, we observe sub-optimal behavior in the Ant domain, both in terms of individual seeds and average performance across seeds. The parameters (confidence, number of evaluation episodes and evaluation frequency) were selected based on the Hopper domain, however it is important to note that these parameters are not optimal for each domain.

Figure 4 presents the standard deviation (STD) of both the online (Figure 4a ) and target (Figure 4b ) networks, and the difference between the empirical means |μ π -μπ-| (Figure 4c ). This clearly shows that in Ant, there is a very large variance, compared the difference between the means, which results in a low confidence. To overcome this issue, we provide an additional experiment, for the Ant domain based on these insights (Figure 4d ), which considered a minimal confidence of 20% (90% in Figure 3 ). These results show that while the parameters are transferable across domains, the optimal parameters found on one domain are not necessarily optimal for another. To analyze the stability of each algorithm (TD3 vs C-TD3), we consider the variance of the policy during the last 100 episodes of training. In Table 1 we present the maximum of the standard deviation in different seeds, where per-seed STD is calculated over the 100 last episodes of training.

5.3.1. STABILITY ANALYSIS:

Our empirical results match the theoretical behavior. Indeed, the inner-seed variance is dramatically reduced when using the conservative update rule, as opposed to the non-conservative TD3.

5.4. POTENTIAL EXTENSIONS:

In this work, we presented an approach for providing probabilistic improvement guarantees. The performance of a trajectory is a random variable, as the initial state, the transition kernel and sometimes the policy -may all be stochastic. To determine which policy π -or π (the target or online) is better, we performed N evaluations, assumed that the samples are normally distributed and performed a t-test. Below we present several intuitive extensions to our work:

Unknown distribution: While the Gaussian assumption worked well in practice, it may not always be correct. In such a scenario, we see two possible routes: (i) known range, in which the minimal and maximal possible values are known, where one may use the Empirical Bernstein [18] , or (ii) when the range is unknown, one may consider a bootstrap-based approach for estimating the confidence intervals around the mean [1] .

Upper/Lower Confidence Bounds: Another approach is to consider improvement by a minimal margin. For instance, we may set the update rule to P(v π -≥ v π-) ≥ 1 -δ, so that the statistical test ensures that with high confidence, the online policy has improved by at least over the target.

Adaptive Sampling: In our approach, we considered a rather small number of evaluations, and thus resorted to an equal number of evaluations of each policy -up to 100 evaluations for each policy. However, when provided with a larger budget, it is of great interest to combine adaptive sampling procedures. Gabillon et al. [7] propose such an approach, which takes into account the empirical mean and variance of each policy evaluation in order to determine which policy to evaluate. At the end of this process, the policy with the highest empirical mean is returned.

In our initial experiments, we tested the UCB/LCB approach. However, we found that is domainspecific. While in some domains the attained rewards are relatively high, and the policies improve greatly between iterations, in others the improvement is rather small. This led to an early convergence to sub-optimal policies. Nevertheless, this approach may prove beneficial in certain settings and should be considered. Additionally, we tested policy swap based on the empirical estimate of the mean, without ensuring a proper confidence. While this approach worked relatively well in practice, as it lacks proper confidence bounds, we decided to focus on the t-test based approach.

6. RELATED WORK:

Trust Region Optimization: Recent works in on-policy RL have also taken inspiration from Conservative Policy Iteration [15, CPI] . Schulman et al. [26] formulated learning through a trust region optimization scheme, which was later optimized in Schulman et al. [27] . These approaches bound the KL-divergence between the policy before and after the update. While TRPO greatly improved the stability compared to previous policy gradient approaches, such as A2C [20] , it lacks the theoretical guarantees.

Safe Policy Improvement: Given a baseline policy, safe policy improvement (SPI) algorithms aim to find a new policy that is better than the baseline with high probability [31; 8; 16] . Although this may seem similar to our update stage, there are two major differences: (i) most SPI algorithms directly calculate the improving policy, which is much less efficient than continuously updating the policy and performing the switch when improvement is achieved, and (ii) due to safety constraints, SPI algorithms usually perform off-policy evaluation for the suggested policy, which is still an open problem in high dimensional problems. In contrast, we evaluate it directly and thus enjoy a better sample complexity.

Evolution Strategies: ES [24; 28; 3] is a gradient-free approach for solving reinforcement learning tasks. At each iteration, given the previous policy π θi , a set of augmented policies is created {π θi+ξi } i and evaluated, where ξ i is some random noise. The ES update rule can be seen as a form of numerical gradient estimation (i.e., finite differences) [35] . There are some similarities between our approach and ES. While ES is a gradient-free method, it uses empirical evaluations of the policies to determine the parameter update direction. Our approach uses these evaluations in order to determine whether or not the policy has improved, and in contrast to ES, will only change the policy if there is a high enough confidence for improvement.

7. CONCLUSIONS:

In this work, we tackled the stability problems of policy-based reinforcement learning approaches. Even though the empirical gradient (SGD) is a noisy estimate of the ideal descent direction, and due to the structure of the optimization landscape and the large step sizes used in practice, the policy's performance varies greatly during training.

Drawing from the connection between actor-critic approaches and approximate policy iteration schemes, we introduced a hierarchical training scheme, which we analyzed by comparing this approach to a bounded random-walk. While this scheme is not an exact representation of the optimization process, it provides an intuition on the behavior of such a probabilistic improvement scheme, i.e., as the probability of improvement increases, the stationary distribution over the performance of the policy shifts towards the optimal solution.

Finally, we proposed the Conservative-TD3, which performs a periodic evaluation of both the online and target policies such that it updates the target network only when there is a higher enough confidence that the online network has surpassed its performance. We validate this on several continuous control tasks in MuJoCo; empirical evidence shows that indeed this approach behaves similarly to the theoretical analysis. When compared to the baseline, we observe a dramatic reduction in variance, increased stability and an overall improvement in the performance itself.

B PARAMETER ABLATION TESTS:

In this section, we present the simulation results for all possible parameter combinations, in the Hopper domain. Specifically, we tested minimal required improvement probability of [0.5, 0.8, 0.9, 0.95], number of evaluations per evaluation steps of [5, 10, 100] and evaluation frequency of [1000, 10000]. The graphs above show the effect of tuning the evaluation frequency, the number of evaluations and the minimal required confidence. As the evaluation becomes more frequent, the agent learns faster. As our approach requires the critic to evaluate the target policy π -, the online policy π aims to find the 1-step greedy policy. This may happen rather quickly, thus a more frequent evaluation scheme will improve faster. On the other hand, the number of evaluation episodes is correlated to the confidence in improvement, i.e., the confidence we have that π is indeed 'better' than π -. We see that as the number of evaluations rises, the agent is capable of finding good solutions even when a higher confidence is required. However, a lower confidence results in a noisier training regime and a small number of evaluations leads to a low confidence -hence high confidence requirements with a low number of evaluations often leads to the inability to converge to a good solution.

C ONLINE VS. TARGET NETWORKS TD3 Actor TD3 Target Figure 12 : On the left is the evaluation of the online network, whereas on the right we present the performance curves of the target network. It is clear across all domains that the target network exhibits a dramatic reduction in variance.

The common approach is to present the performance of the online network. In these graphs, we show that, as opposed to the common belief, the target network suffers less from instability and offers improved performance. These results are intuitive once the connection to policy iteration schemes is made.

:

Temporal smoothing is a common practice in empirical studies, in which each data-point in the graph represents a moving average over the past N evaluations. While this makes the graphs easier to read, it hides a lot of important information such as the variance of each seed during training.

The target network needs to be re-evaluated in order to ensure an unbiased estimate of its performance.

We observed that some seeds in deep RL result in utter failure, for instance, a score of

in Hopper. This issue is relatively rare, and also happens in the baselines, but hinders the proper evaluation of the policy.

Figures and Tables:

Figure 1: Performance plots of the TD3 [6] algorithm.

Figure 2: Conservative Policy Gradients.

Figure 4: Failure analysis of the Ant-v2 domain.

Figure 6: C-TD3, parameter ablation. Comparison is performed every 1,000 steps over 5 episodes with swap confidence of 1 -δ = 95, 90, 80 and 50.

Figure 7: C-TD3, parameter ablation. Comparison is performed every 1,000 steps over 10 episodes with swap confidence of 1 -δ = 95, 90, 80 and 50.

Figure 8: C-TD3, parameter ablation. Comparison is performed every 1,000 steps over 100 episodes with swap confidence of 1 -δ = 95, 90, 80 and 50.

Figure 9: C-TD3, parameter ablation. Comparison is performed every 10,000 steps over 5 episodes with swap confidence of 1 -δ = 95, 90, 80 and 50.

Figure 10: C-TD3, parameter ablation. Comparison is performed every 10,000 steps over 10 episodes with swap confidence of 1 -δ = 95, 90, 80 and 50.

Figure 11: C-TD3, parameter ablation. Comparison is performed every 10,000 steps over 100 episodes with swap confidence of 1 -δ = 95, 90, 80 and 50.

Algorithm 2 DDPG, TD3 and our Conservative update 1: Input: algorithm, learning rate α, averaging parameter τ , steps T and exploration N . 2: Initialize policy θ, critics φ 1 , φ 2 and replay buffer R 3:

Table content: <table><tr><td>9:</td><td>k = k + 1</td></tr><tr><td colspan="2">10: return policy π k</td></tr></table>

Process STD calculated over the last 100 training episodes.

Table content: <table><tr><td/><td colspan="6">Hopper InvertedPendulum InvertedDoublePendulum Walker2d HalfCheetah Ant</td></tr><tr><td>C-TD3</td><td>36</td><td>0</td><td>6</td><td>99</td><td>81</td><td>57</td></tr><tr><td>TD3</td><td>799</td><td>139</td><td>2409</td><td>468</td><td>350</td><td>902</td></tr></table>

Peer review (iclr2020):
This paper proposed to use target network policy as a conservative policy for performance evaluation. Instead of performing Polyak averaging on the target network, the authors proposed to utilize statistical hypothesis testing to check whether the performance of the online policy is better than the target network policy and update the target network policy according to the results of the hypothesis testing. The paper is clear written and easy to follow the core idea. As for the experiments, the authors evaluate the proposed method on a variant of TD3 (Conservative-TD3) and the experimental results indicate the proposed method indeed reduces the variance of the expected return. Ablation studies has been provided in the appendix to show the effectiveness of the proposed method. Besides the promising results, I believe there are several concerns that should be clarified before we can conclude that the proposed method can improve the stability. - Stability Measurement: While the experimental results show that the proposed method can reduce the variance of expected return, it is not a direct measurement of the stability or the robustness of the learned policy. It is better to show whether the proposed method can satisfy stability or robustness definition of RL algorithms. (For example, whether the proposed method can improve the robustness: Given a policy, by adding a epsilon perturbation to the input, the output is still epsilon-robustness) Otherwise, the author should add some discussion to clarify the difference. - Conservative Updates on Q functions. In Actor critic frameworks such as DDPG or TD3, the performance of the actor is usually determined by the critic. The proposed method is more likely to “select” stable actors rather than directly improving the stable of the training process or the stability of the policy. I wonder whether it is possible to improve the stability of Q updates such that the “selection” process of policy can be easier, which may accelerate the current training process (or making the hypothesis easier to satisfy). - More related work should be compared. The authors only compare the original version of TD3 and the modified proposed method. Other recent proposed methods to improve stability should be compared in the experiments, such as Constrained Policy Optimization (Achiam et. al 2017), Lypunov-based Safe Policy methods (such as Chow et. al 2019). - Noisy Environment. The authors demonstrate the stability of the proposed method in the ordinary mujoco benchmarks. How does the proposed method perform in the noisy MDP settings? Since the original Mujoco implementation is deterministic, the experiments that the author conducted are not enough to show the proposed method can generalize to more realistic settings such as noisy MDPs. It would be more convincing if the method can still perform well in such settings to support the claim. Overall I think the authors proposed a simple but yet effective method to improve the stability of policy, while the current paper requires more comparison with other methods and more challenging settings to show the effectiveness of the proposed method. -------------------------------- I will update my score if the author clarify above questions. ------------------------------- The author clarified one of my main concern, but the other reviewers point out that the comparison is not fair (using only 5 seeds and discarding the failure seeds). Related Papers: Achiam, Joshua, et al. *Constrained policy optimization.* Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017. Chow, Yinlam, et al. *Lyapunov-based Safe Policy Optimization for Continuous Control.* arXiv preprint arXiv:1901.10031 (2019).